XML signature different on Domino server

Question

Description

We are trying to set up communications with a service provider that requires that we sign the XML we send. This is done on a Lotus Domino platform using Java 1.6. The problem is that the signature that gets generated is different when running the code using the Domino JRE, than when running it in for example Eclipse using a regular JRE. When run in Domino it triggers a validation error on the receiving end. So using the same code in Eclipse and just targeting the IBM JRE instead results in this error. That makes it pretty clear that there is something in the Domino JRE making this go wrong, we just don't know what or how to get to the bottom of it. Any ideas what setting could be the cause of this difference?

For clarification, note that we are using the same content for all tests and that running i twice in Eclipse yields exactly the same result.

Error message

The error that's returned just says that the signature is invalid: 2013-08-05 16:38:18 Agent Manager: Agent error: ESignClientException: ESigningFacade.getOrder failed. Reason: (RMS returned an error response. TransId [-1971835324952692066] RMS ErrorCode [RMS1002] ErrorText [SignatureVerification failed. The Signature is not valid.])

Code

We are using the service providers API but here's a short example of the code we're running:

public void test() {
    try {
        String xmlrequest = "SOME XML"
        String keyStorePath = "keystore.p12";
        char[] pwdChars = "********".toCharArray();
        PrivateKey privateKey = null;
        String providerName = System.getProperty("jsr105Provider", "org.jcp.xml.dsig.internal.dom.XMLDSigRI");
        XMLSignatureFactory fac = XMLSignatureFactory.getInstance("DOM", (Provider) Class.forName(providerName).newInstance());
        KeyInfoFactory kif = fac.getKeyInfoFactory();

        KeyStore store = KeyStore.getInstance("PKCS12");
        store.load(new FileInputStream(keyStorePath), pwdChars);

        Enumeration ksAliases = store.aliases();
        String keyAlias = null;
        while (ksAliases.hasMoreElements()) {
            String currAlias = (String) ksAliases.nextElement();
            if (store.isKeyEntry(currAlias)) {
                keyAlias = currAlias;
                privateKey = ((PrivateKey) store.getKey(currAlias, pwdChars));
                break;
            }
        }

        Certificate certificate = store.getCertificate(keyAlias);
        CertificateFactory cf = CertificateFactory.getInstance("X.509");
        X509Certificate cert = (X509Certificate) cf.generateCertificate(new   ByteArrayInputStream(certificate.getEncoded()));
        X509Data x5 = kif.newX509Data(Collections.nCopies(1, cert));
        KeyInfo ki = kif.newKeyInfo(Collections.nCopies(1, x5));
        Reference ref = fac.newReference("#object", fac.newDigestMethod("http://www.w3.org/2000/09/xmldsig#sha1", null));

        DocumentBuilderFactory dbf = DocumentBuilderFactory.newInstance();
        dbf.setNamespaceAware(true);
        dbf.setValidating(false);
        dbf.setAttribute("http://java.sun.com/xml/jaxp/properties/schemaLanguage", "http://www.w3.org/2001/XMLSchema");
        DocumentBuilder db = dbf.newDocumentBuilder();
        Document doc = db.parse(new InputSource(new StringReader(xmlrequest)));
        XMLStructure content = new DOMStructure(doc.getFirstChild());

        XMLObject obj = fac.newXMLObject(Collections.nCopies(1, content), "object", null, null);
        SignedInfo si =    fac.newSignedInfo(fac.newCanonicalizationMethod("http://www.w3.org/2001/10/xml-exc-c14n#", (C14NMethodParameterSpec) null),                fac.newSignatureMethod("http://www.w3.org/2000/09/xmldsig#rsa-sha1", null),       Collections.nCopies(1, ref));
        XMLSignature signature = fac.newXMLSignature(si, ki, Collections.nCopies(1, obj), null, null);
        DOMSignContext dsc = new DOMSignContext(privateKey, doc);
        signature.sign(dsc);

        ByteArrayOutputStream os = new ByteArrayOutputStream();
        TransformerFactory tf = TransformerFactory.newInstance();
        Transformer trans = tf.newTransformer();
        trans.transform(new DOMSource(doc), new StreamResult(os));
        os.flush();
        String res = new String(os.toByteArray(), "UTF-8");
        os.close();

    } catch (Exception e) {
        e.printStackTrace();
    }
}

Answer 1

Ok I managed to solve this on my own after a lot of trial and error. Since the provider "org.jcp.xml.dsig.internal.dom.XMLDSigRI" is not included in the IBM JRE it is also not added in the java.security file (jvm/lib/security/java.security). So I added a line for the new provider to the java.security file and then it worked. Or so I thought. It only worked in my test project in Eclipse which was using the IBM JRE. When I tried the same fix on the server it didn't have any impact at all. The list of providers didn't change when I added providers to the java.security file as it did before. So after a bit more testing I found that the provider got registered when I added the provider JAR (xmlsec-1.4.2.jar) to the folder "jvm/lib/ext/". That is how I got it working.