The PKIX error means that the client didn't trust the server certificate, where the server in this case was the Registry.
To clarify, you need two private keys and two keystores to hold them in, one each. You then need to create certificates in each keystore, export them, and import them into the peer's truststore. The server's truststore must trust the client's keystore, and vice versa.
Your code looks mostly OK. The result of createRegistry() should be stored in a static variable, to prevent it being GC'd. You don't need a serialVersionUID in the server class, whatever your IDE may tell you. It doesn't get serialized, at least not by RMI.
EDIT The problem is here:
System.setProperty("javax.net.ssl.keyStore", "C:\\ssl\\keystore-server.jks");
System.setProperty("javax.net.ssl.trustStore", "C:\\ssl\\truststore-client.jks");
which should be:
System.setProperty("javax.net.ssl.keyStore", "C:\\ssl\\keystore-server.jks");
System.setProperty("javax.net.ssl.trustStore", "C:\\ssl\\truststore-server.jks");
and here:
System.setProperty("javax.net.ssl.keyStore", "C:\\ssl\\keystore-client.jks");
System.setProperty("javax.net.ssl.trustStore", "C:\\ssl\\truststore-server.jks"
which should be:
System.setProperty("javax.net.ssl.keyStore", "C:\\ssl\\keystore-client.jks");
System.setProperty("javax.net.ssl.trustStore", "C:\\ssl\\truststore-client.jks"
EDIT 2 The underlying problem is that the trust store you need when binding to the Registry is the client truststore, but the truststore you need when running the server is the server truststore.
There are at least three possible solutions, in increasing order of merit:
Set up a subclass of
SslRMIClientSocketFactory
with its ownSSLContext
with its ownTrustManager
loaded from the client truststore, and overridecreateSocket()
. Ouch.Import the server's certificate into the server's truststore as well.
Use the return value of
createRegistry()
to do thebind()
instead of callinggetRegistry()
in the server at all, and avoid the whole problem.