tomcat classloader bug?

Question

UPDATE: I've learned more about what's going on and added the new info at the bottom.

I have 2 applications running under tomcat. App1 is loaded first, then App2. If App1 runs into any kind of error during startup and fails to load successfully, I get this error during startup of App2:

Caused by: java.security.NoSuchAlgorithmException: No such algorithm: RSA/NONE/OAEPWithSHA1AndMGF1Padding
    at javax.crypto.Cipher.getInstance(DashoA13*..)
    at javax.crypto.Cipher.getInstance(DashoA13*..)
    at com.jp.protection.security.BouncyCastleSecurityProvider.getCipher(BouncyCastleSecurityProvider.java:139)
    at com.jp.protection.security.BouncyCastleSecurityProvider.decode(BouncyCastleSecurityProvider.java:110)
    ... 70 more
Caused by: java.lang.NullPointerException
    at org.bouncycastle.jcajce.provider.util.DigestFactory.getDigest(DigestFactory.java:86)
    at org.bouncycastle.jcajce.provider.asymmetric.rsa.CipherSpi.initFromSpec(CipherSpi.java:83)
    at org.bouncycastle.jcajce.provider.asymmetric.rsa.CipherSpi.engineSetPadding(CipherSpi.java:214)
    at javax.crypto.Cipher$r.a(DashoA13*..)
    ... 74 more

Note that the ultimate cause is a NullPointerException. I downloaded the source for DigestFactory and it looks like this (just the relevant parts excerpted):

package org.bouncycastle.jcajce.provider.util;

public class DigestFactory
{
    private static Set sha1 = new HashSet();

    static
    {           
        sha1.add("SHA1");
        sha1.add("SHA-1");
    }

    public static Digest getDigest(
        String digestName) 
    {
        digestName = Strings.toUpperCase(digestName);

        if (sha1.contains(digestName))  ** line 86 where npe occurs**
        {
            return new SHA1Digest();
        }
    [...]

The only way to get an NPE at line 86 is if sha1 is null. (If digestName was null, the NPE would occur in the call to Strings.toUpperCase. ) And in fact if I put a breakpoint here, under the error scenario the debugger shows sha1 (and all the other similarly statically initialized fields) as null. These fields are private and there are no methods that allow these fields to be modified.

How is this possible? I thought perhaps my DigestFactory source didn't exactly match the jar I was running so the debugger was misleading me, but it should be the right version, and everything else seems to line up.

Under the debugger, I tried calling DigestFactory.getDigest("SHA-1") (using the debugger's evaluate expression) during an earlier phase of App2's initialization, before the exception happens, and it returned successfully. That suggests that either the static fields of DigestFactory are successfully initialized and then somehow later set to null, or another classloader has a different version of the class (which even if the case, doesn't explain how they could be null).

This exception is happening 2 layers deep into third-party code (jproductivity Protection package is using bouncycastle) so my control over the situation is limited. However I would like to understand first of all how this is possible, and hopefully how I can prevent or work around it. Another piece of the mystery is why the first app's error is having any effect on the second app -- under tomcat these should have separate classloaders. But if there's no error in the first app then this problem does not occur in the second app.

UPDATE: Since I posted this I have learned a bit more. When a webapp is stopped by tomcat (in this case because of the startup error), tomcat will null static fields in its classes to avoid memory leaks. So this explains how my immutable static fields were set to null. However, this should not be affecting App2 since it should be using a separate classloader. But I've looked under the debugger, and in fact the same classloader is being used for the DigestFactory class in both webapps. This contradicts all tomcat documentation that I can find. For other classes (my own classes), there are different classloaders. I'm wondering if it has anything to do with DigestFactory being static and immutable, so in theory it wouldn't matter where it came from.

So, as an experiment I removed the jar that contains DigestFactory from both of my webapps and added it to tomcat/lib (so that it's shared and not part of either webapp). This fixed the problem -- its fields were not nulled presumably because it was not part of the offending webapp. However, this approach is undesirable and should not be necessary. Is this a tomcat bug?

Answer 1

After concluding that this had to be a tomcat bug, I upgraded from tomcat 6 to tomcat 7, and that fixed the problem.

Answer 2

What I believe is happening is that bouncycastle gets registered under the applications classloader when you fire up App1. If I remember correctly, the provider is then registered into the JVM classloader through some static initializers or methods.

When your App1 crashes (or redeploys) it's classloader is removed, along with any classes it has loaded, including bouncycastle. The result is that the JVM think it is still there, since its still registered, while actually it isn't.

The solution is to add BouncyCastleProvider to the list of security providers in jre/lib/security/java.security (java 7 location, I think it's in jre/lib/ext in older versions) by adding a line similar to this:

security.provider.[next available number]=org.bouncycastle.jce.provider.BouncyCastleProvider

You might need to add the jar-file there too.

Answer 3

I have found something like the same behaviour when the BouncyCastle provider was inlcuded in my WebApp WEB-INF/lib and I have reloaded my tomcat App. The following excpetion raised in my case:

júl. 09, 2015 7:24:00 DE org.apache.catalina.loader.WebappClassLoader loadClass
INFO: Illegal access: this web application instance has been stopped already.  Could not load org.bouncycastle.jcajce.provider.digest.SHA1$PBEWithMacKeyFactory.  The eventual following stack trace is caused by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no functional impact.
java.lang.IllegalStateException
        at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1612)
        at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1571)
        at org.apache.tomee.catalina.LazyStopWebappClassLoader.loadClass(LazyStopWebappClassLoader.java:129)
        at java.security.Provider$Service.getImplClass(Provider.java:1279)
        at java.security.Provider$Service.newInstance(Provider.java:1237)
        at sun.security.jca.GetInstance.getInstance(GetInstance.java:236)
        at javax.crypto.JceSecurity.getInstance(JceSecurity.java:116)
        at javax.crypto.SecretKeyFactory.getInstance(SecretKeyFactory.java:243)
        at org.bouncycastle.jcajce.util.ProviderJcaJceHelper.createSecretKeyFactory(Unknown Source)
        at org.bouncycastle.jcajce.provider.keystore.pkcs12.PKCS12KeyStoreSpi.calculatePbeMac(Unknown Source)
        at org.bouncycastle.jcajce.provider.keystore.pkcs12.PKCS12KeyStoreSpi.engineLoad(Unknown Source)
        at java.security.KeyStore.load(KeyStore.java:1214)
        at hu.myapp.mypackage.myEJB.setup(myEJB.java:154)

where setup is a @PostConstruct annotated method in myEJB stateless which initialize the BC provider.

@LocalBean
@Stateless
public class myEJB {
  ...
  private BouncyCastleProvider bc = null;
  private KeyStore ks = null;

  @PostConstruct
  protected void setup() {
        bc = new BouncyCastleProvider();
        ks = KeyStore.getInstance("PKCS12", bc);
        ...
  }
  ...
}

When I have moved the provider bcprov-jdk15on-152.jar into the $JAVA_HOME/jre/lib/ext and I have inserted the following row into the $JAVA_HOME/jre/lib/security/java.security file:

  security.provider.11=org.bouncycastle.jce.provider.BouncyCastleProvider

of course provider index in my case was 11 because the 10 was the last provider so in your case it can be different one.

and I have removed the bcprov-jdk15on-152.jar file from my webapp everything worked perfect.

So I think Einar's answer is appropriate.