To answer the question in your answer:
Though I don't know why this works, or what it means.
SecurityBindingElement.CreateCertificateOverTransportBindingElement()
initializes the MessageSecurityVersion of the binding to its defaults, which is:
WS-Security 1.1, WS-Trust of February 2005, WS-SecureConversation of February 2005 and WS-SecurityPolicy 1.1.
With the overload you're now calling, you're specifying this:
Basic Security Profile 1.0 based on WS-Security 1.0, WS-Trust of February 2005, WS-SecureConversation of February 2005 and WS-SecurityPolicy 1.1 security specifications.
To determine what EncodingType WCF actually emits, you'll have to either put an HTTP monitor in between (e.g. Fiddler) or let .NET output trace information to log the message being sent. You can also access or request the server logs to see why the server thinks the message is invalid.
I suspect however, given certain web searches on the actual error message, that the Java server complains about your WCF client omitting the EncodingType=...#Base64Binary
on the wsse:BinarySecurityToken
. According to the spec, that is the only allowed value (or a custom one if both parties agree on it) and it's not marked as optional.
After changing the MessageSecurityVersion to WS-sec 1.0 as also explained here (which is easy to find - once you know what you're looking for), I guess WCF explicitly outputs the EncodingType
attribute, causing the service to accept the message.