no such algorithm: SunTls12RsaPremasterSecret for provider SunPKCS11-NSSFIPS

Question

I have run into an issue after updating my JRE to 7u51. Prior to this, things were working fine.

I have a web application that runs on Tomcat and it uses the mozilla NSS libraries to achieve FIPS 140-2 compliance when using SSL/TLS. To do this, I had to change the default SunJSSE provider to my custom SunPKCS11-NSSFIPS provider.

Everything starts up fine. The server shows it is ready, but when I try to hit it from a web browser, I get a "The connection was interrupted" error.

Looking at the logs on the server, I see this:

Feb 09, 2014 3:00:16 AM org.apache.tomcat.util.net.NioEndpoint$SocketProcessor run
SEVERE: 
java.lang.RuntimeException: Could not generate dummy secret
    at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1287)
    at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:513)
    at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:790)
    at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:758)
    at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)
    at org.apache.tomcat.util.net.SecureNioChannel.handshakeUnwrap(SecureNioChannel.java:335)
    at org.apache.tomcat.util.net.SecureNioChannel.handshake(SecureNioChannel.java:193)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1642)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.RuntimeException: Could not generate dummy secret
    at sun.security.ssl.RSAClientKeyExchange.generatePreMasterSecret(RSAClientKeyExchange.java:281)
    at sun.security.ssl.RSAClientKeyExchange.polishPreMasterSecretKey(RSAClientKeyExchange.java:245)
    at sun.security.ssl.RSAClientKeyExchange.<init>(RSAClientKeyExchange.java:167)
    at sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:190)
    at sun.security.ssl.Handshaker.processLoop(Handshaker.java:868)
    at sun.security.ssl.Handshaker$1.run(Handshaker.java:808)
    at sun.security.ssl.Handshaker$1.run(Handshaker.java:806)
    at java.security.AccessController.doPrivileged(Native Method)
    at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1227)
    at org.apache.tomcat.util.net.SecureNioChannel.tasks(SecureNioChannel.java:285)
    at org.apache.tomcat.util.net.SecureNioChannel.handshakeUnwrap(SecureNioChannel.java:343)
    ... 5 more
Caused by: java.security.NoSuchAlgorithmException: no such algorithm: SunTls12RsaPremasterSecret for provider SunPKCS11-NSSFIPS
    at sun.security.jca.GetInstance.getService(GetInstance.java:100)
    at javax.crypto.JceSecurity.getInstance(JceSecurity.java:109)
    at javax.crypto.KeyGenerator.getInstance(KeyGenerator.java:287)
    at sun.security.ssl.JsseJce.getKeyGenerator(JsseJce.java:269)
    at sun.security.ssl.RSAClientKeyExchange.generatePreMasterSecret(RSAClientKeyExchange.java:270)
    ... 15 more

I belive this is happening because the browser is trying to perform the handshake with TLSv1.2 but my security provider can't handle it. Is there a way to resolve this while still using my custom provider?

Immediately after that stack trace, there is another in the log file:

Feb 09, 2014 3:00:16 AM org.apache.tomcat.util.net.NioEndpoint$SocketProcessor run
SEVERE: 
java.lang.RuntimeException: java.security.InvalidAlgorithmParameterException: init() failed
    at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1287)
    at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:513)
    at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:790)
    at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:758)
    at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)
    at org.apache.tomcat.util.net.SecureNioChannel.handshakeUnwrap(SecureNioChannel.java:335)
    at org.apache.tomcat.util.net.SecureNioChannel.handshake(SecureNioChannel.java:193)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1642)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)
Caused by: java.security.ProviderException: java.security.InvalidAlgorithmParameterException: init() failed
    at sun.security.ssl.Handshaker.calculateMasterSecret(Handshaker.java:1064)
    at sun.security.ssl.Handshaker.calculateKeys(Handshaker.java:999)
    at sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:234)
    at sun.security.ssl.Handshaker.processLoop(Handshaker.java:868)
    at sun.security.ssl.Handshaker$1.run(Handshaker.java:808)
    at sun.security.ssl.Handshaker$1.run(Handshaker.java:806)
    at java.security.AccessController.doPrivileged(Native Method)
    at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1227)
    at org.apache.tomcat.util.net.SecureNioChannel.tasks(SecureNioChannel.java:285)
    at org.apache.tomcat.util.net.SecureNioChannel.handshakeUnwrap(SecureNioChannel.java:343)
    ... 5 more
Caused by: java.security.InvalidAlgorithmParameterException: init() failed
    at sun.security.pkcs11.P11TlsMasterSecretGenerator.engineInit(P11TlsMasterSecretGenerator.java:89)
    at javax.crypto.KeyGenerator.init(KeyGenerator.java:431)
    at javax.crypto.KeyGenerator.init(KeyGenerator.java:414)
    at sun.security.ssl.Handshaker.calculateMasterSecret(Handshaker.java:1052)
    ... 14 more
Caused by: java.security.InvalidKeyException: Could not create key
    at sun.security.pkcs11.P11SecretKeyFactory.createKey(P11SecretKeyFactory.java:270)
    at sun.security.pkcs11.P11SecretKeyFactory.convertKey(P11SecretKeyFactory.java:175)
    at sun.security.pkcs11.P11SecretKeyFactory.convertKey(P11SecretKeyFactory.java:111)
    at sun.security.pkcs11.P11TlsMasterSecretGenerator.engineInit(P11TlsMasterSecretGenerator.java:87)
    ... 17 more
Caused by: sun.security.pkcs11.wrapper.PKCS11Exception: CKR_ATTRIBUTE_VALUE_INVALID
    at sun.security.pkcs11.wrapper.PKCS11.C_CreateObject(Native Method)
    at sun.security.pkcs11.P11SecretKeyFactory.createKey(P11SecretKeyFactory.java:265)
    ... 20 more

Any help will be appreciated.

Answer 1

If i am not wrong NSS donot support tls1.2 yet. So you shouldnot initialize the handshake for tls1.2. Some browsers have changed the default TLS version to 1.2. You have to change that to TLS 1.1 and try again.