iOS Mobile Device Management - The SCEP server returned an invalid response

Question

I am in the process of writing an open source iOS mobile device management module in Java. For this I am referring the Apple provided Ruby code at [1]. I have set this up and it works fine for me. Now I need to convert this code to Java. So far I have accomplished to do that up to PKIOperation. In the PKI operation I get "The SCEP server returned an invalid response" which I believe is due to wrong response I sent to device upon PKIOperation.

However when I do search on the internet I get this is something to do with the "maxHttpHeaderSize" as I am using the server as Apache Tomcat. Although I increase that since still it does not get resolved.

Here is the code I need to convert - taken from Apple provided Ruby script

   if query['operation'] == "PKIOperation"
    p7sign = OpenSSL::PKCS7::PKCS7.new(req.body)
    store = OpenSSL::X509::Store.new
    p7sign.verify(nil, store, nil, OpenSSL::PKCS7::NOVERIFY)
    signers = p7sign.signers
    p7enc = OpenSSL::PKCS7::PKCS7.new(p7sign.data)
    csr = p7enc.decrypt(@@ra_key, @@ra_cert)
    cert = issueCert(csr, 1)
    degenerate_pkcs7 = OpenSSL::PKCS7::PKCS7.new()
    degenerate_pkcs7.type="signed"
    degenerate_pkcs7.certificates=[cert]
    enc_cert = OpenSSL::PKCS7.encrypt(p7sign.certificates, degenerate_pkcs7.to_der, 
        OpenSSL::Cipher::Cipher::new("des-ede3-cbc"), OpenSSL::PKCS7::BINARY)
    reply = OpenSSL::PKCS7.sign(@@ra_cert, @@ra_key, enc_cert.to_der, [], OpenSSL::PKCS7::BINARY)
    res['Content-Type'] = "application/x-pki-message"
    res.body = reply.to_der
   end

So this is how I written this in Java using Bouncycastle library.

        X509Certificate generatedCertificate = generateCertificateFromCSR(
                privateKeyCA, certRequest, certCA.getIssuerX500Principal()
                        .getName());

        CMSTypedData msg = new CMSProcessableByteArray(
                generatedCertificate.getEncoded());
        CMSEnvelopedDataGenerator edGen = new CMSEnvelopedDataGenerator();
        edGen.addRecipientInfoGenerator(new JceKeyTransRecipientInfoGenerator(
                receivedCert).setProvider(AppConfigurations.PROVIDER));
        CMSEnvelopedData envelopedData = edGen
                .generate(
                        msg,
                        new JceCMSContentEncryptorBuilder(
                                CMSAlgorithm.DES_EDE3_CBC).setProvider(
                                AppConfigurations.PROVIDER).build());

        CMSSignedDataGenerator gen = new CMSSignedDataGenerator();
        ContentSigner sha1Signer = new JcaContentSignerBuilder(
                AppConfigurations.SIGNATUREALGO).setProvider(
                AppConfigurations.PROVIDER).build(privateKeyRA);

        List<X509Certificate> certList = new ArrayList<X509Certificate>();
        CMSTypedData cmsByteArray = new CMSProcessableByteArray(
                envelopedData.getEncoded());
        certList.add(certRA);

        Store certs = new JcaCertStore(certList);

        gen.addSignerInfoGenerator(new JcaSignerInfoGeneratorBuilder(
                new JcaDigestCalculatorProviderBuilder().setProvider(
                        AppConfigurations.PROVIDER).build()).build(
                sha1Signer, certRA));

        gen.addCertificates(certs);

        CMSSignedData sigData = gen.generate(cmsByteArray, true);

        return sigData.getEncoded();

The returned result here will be output in to the servlet output stream with the content type "application/x-pki-message".

It seems I get the CSR properly and I generate the X509Certificate using following code.

    public static X509Certificate generateCertificateFromCSR(
        PrivateKey privateKey, PKCS10CertificationRequest request,
        String issueSubject) throws Exception {

    Calendar targetDate1 = Calendar.getInstance();
    targetDate1.setTime(new Date());
    targetDate1.add(Calendar.DAY_OF_MONTH, -1);

    Calendar targetDate2 = Calendar.getInstance();
    targetDate2.setTime(new Date());
    targetDate2.add(Calendar.YEAR, 2);

    // yesterday
    Date validityBeginDate = targetDate1.getTime();
    // in 2 years
    Date validityEndDate = targetDate2.getTime();

    X509v3CertificateBuilder certGen = new X509v3CertificateBuilder(
            new X500Name(issueSubject), BigInteger.valueOf(System
                    .currentTimeMillis()), validityBeginDate,
            validityEndDate, request.getSubject(),
            request.getSubjectPublicKeyInfo());
    certGen.addExtension(X509Extension.keyUsage, true, new KeyUsage(
            KeyUsage.digitalSignature | KeyUsage.keyEncipherment));

    ContentSigner sigGen = new JcaContentSignerBuilder(
            AppConfigurations.SHA256_RSA).setProvider(
            AppConfigurations.PROVIDER).build(privateKey);

    X509Certificate issuedCert = new JcaX509CertificateConverter()
            .setProvider(AppConfigurations.PROVIDER).getCertificate(
                    certGen.build(sigGen));

    return issuedCert;
}

The generated certificate commonn name is,

Common Name: mdm(88094024-2372-4c9f-9c87-fa814011c525)

Issuer: mycompany Root CA (93a7d1a0-130b-42b8-bbd6-728f7c1837cf), None

[1] - https://developer.apple.com/library/ios/documentation/NetworkingInternet/Conceptual/iPhoneOTAConfiguration/Introduction/Introduction.html

Answer 1

Some thoughts (not a real answer)

1) The "The SCEP server returned an invalid response" could be returned for a huge amount of different reasons. Generally speaking, if SCEP returns anything what can't be parsed by MDM client, it will show this error.

2) Take a look at jSCEP (https://code.google.com/p/jscep/). It's java implementation of SCEP server. And I am pretty sure that it works with iOS (I used it).

• You can both check how they are handling it (as I remember, they are using Bouncy Castle too)

• Also, I would rather include jSCEP in your open source imlpementation than reinvent a bycycle. SCEP RFC has quite a lot of pieces, jSCEP is pretty good with following it.