Trying to understand SSL certificates in Java

Question

To preface, I don't have a good (read: any) background in security, and I think that lack of understanding may be the root of this question.

I was playing around with some code from here dealing with setting up a RTMPS connection. Basically, I was hoping to understand how it worked, and while I don't have any issues with the RTMP protocol, I have trouble following why the SSL code was written how it is. It seems like it would get the certificate from the server, then store it in the cacerts file or the cents folder.

I've been trying to follow the connection process to see where the certificate is retrieved and stored, but haven't been able to figure it out. Here is a section of the code that, as far as I can tell, behaves properly:

Note: SavingTrustManager is just a wrapper around X509TrustManager that also stores the certificate chain.

/**
 * Opens the socket with the default or a previously saved certificate
 *
 * @return A special TrustManager to save the certificate if necessary
 * @throws IOException
 */
private SavingTrustManager openSocketWithCert() throws IOException {
try {
    // Load the default KeyStore or a saved one
    KeyStore ks = KeyStore.getInstance(KeyStore.getDefaultType());
    File file = new File("certs/" + server + ".cert");
    if (!file.exists() || !file.isFile())
        file = new File(System.getProperty("java.home") + "/lib/security/cacerts");

    InputStream in = new FileInputStream(file);
    ks.load(in, passphrase);

    // Set up the socket factory with the KeyStore
    SSLContext context = SSLContext.getInstance("TLS");
    TrustManagerFactory tmf = TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm());
    tmf.init(ks);
    X509TrustManager defaultTrustManager = (X509TrustManager)tmf.getTrustManagers()[0];
    SavingTrustManager tm = new SavingTrustManager(defaultTrustManager);
    context.init(null, new TrustManager[] { tm }, null);
    SSLSocketFactory factory = context.getSocketFactory();

    sslsocket = (SSLSocket)factory.createSocket(server, port);

    return tm;
}
catch (Exception e) {
    // Hitting an exception here is very bad since we probably won't
    // recover
    // (unless it's a connectivity issue)

    // Rethrow as an IOException
    throw new IOException(e.getMessage());
}
}

/**
 * Downloads and installs a certificate if necessary
 *
 * @throws IOException
 */
private void getCertificate() throws IOException {
try {
    SavingTrustManager tm = openSocketWithCert();

    // Try to handshake the socket
    boolean success = false;
    try {
        sslsocket.startHandshake();
        success = true;
    }
    catch (SSLException e) {
        sslsocket.close();
    }

    // If we failed to handshake, save the certificate we got and try
    // again
    if (!success) {
        // Set up the directory if needed
        File dir = new File("certs");
        if (!dir.isDirectory()) {
            dir.delete();
            dir.mkdir();
        }

        // Reload (default) KeyStore
        KeyStore ks = KeyStore.getInstance(KeyStore.getDefaultType());
        File file = new File(System.getProperty("java.home") + "/lib/security/cacerts");

        InputStream in = new FileInputStream(file);
        ks.load(in, passphrase);

        // Add certificate
        X509Certificate[] chain = tm.chain;
        if (chain == null)
            throw new Exception("Failed to obtain server certificate chain");

        X509Certificate cert = chain[0];
        String alias = server + "-1";
        ks.setCertificateEntry(alias, cert);

        // Save certificate
        OutputStream out = new FileOutputStream("certs/" + server + ".cert");
        ks.store(out, passphrase);
        out.close();
        System.out.println("Installed cert for " + server);
    }
}
catch (Exception e) {
    // Hitting an exception here is very bad since we probably won't
    // recover
    // (unless it's a connectivity issue)

    // Rethrow as an IOException
    e.printStackTrace();
    throw new IOException(e.getMessage());
}
}

/**
 * Attempts to connect given the previous connection information
 *
 * @throws IOException
 */
public void connect() throws IOException {
try {
    sslsocket = (SSLSocket)SSLSocketFactory.getDefault().createSocket(server, port);
    in = new BufferedInputStream(sslsocket.getInputStream());
    out = new DataOutputStream(sslsocket.getOutputStream());

    doHandshake();
}
catch (IOException e) {
    // If we failed to set up the socket, assume it's because we needed
    // a certificate
    getCertificate();
    // And use the certificate
    openSocketWithCert();

    // And try to handshake again
    in = new BufferedInputStream(sslsocket.getInputStream());
    out = new DataOutputStream(sslsocket.getOutputStream());

    doHandshake();
}

// Start reading responses
pr = new RTMPPacketReader(in);

// Handle preconnect Messages?
// -- 02 | 00 00 00 | 00 00 05 | 06 00 00 00 00 | 00 03 D0 90 02

// Connect
Map<String, Object> params = new HashMap<String, Object>();
params.put("app", app);
params.put("flashVer", "WIN 10,1,85,3");
params.put("swfUrl", swfUrl);
params.put("tcUrl", "rtmps://" + server + ":" + port);
params.put("fpad", false);
params.put("capabilities", 239);
params.put("audioCodecs", 3191);
params.put("videoCodecs", 252);
params.put("videoFunction", 1);
params.put("pageUrl", pageUrl);
params.put("objectEncoding", 3);

byte[] connect = aec.encodeConnect(params);

out.write(connect, 0, connect.length);
out.flush();

while (!results.containsKey(1))
    sleep(10);
TypedObject result = results.get(1);
DSId = result.getTO("data").getString("id");

connected = true;
}

I think I can't follow how the server's certificate is gathered. It seems like the getCertificate() method is never called from my perspective. Is this because the certificate is stored somewhere already? There's no certs directory, so I'd assume its in the accents file. Would it be called if I cleared my cacerts file? I felt like that would be a bad idea.

Sorry for asking such a vague question, I've just been pondering this code for the past couple days and haven't had much luck with existing resources (javadocs etc). Thanks!

Answer 1

The SSLSocket object has access to the certificates it should trust because it is created, indirectly, by an SSLContext that has been initialized, indirectly, by a KeyStore object which has the locally trusted certificates. When connect() is called on the SSLSocket object, it retrieves the certificate from the server, which it can then compare with the certificates in the KeyStore object to determine whether the certificate chain is legitimate. All of this happens under the covers, so you as an application programmer don't ever need to worry about certificates in the code. This is why the example code's getCertificate() method is not called under normal circumstances.

However, when authentication fails, the example code then, in its getCertificate() method, obtains the server's certificate and installs it in the local keystore. The next attempt to connect to the server will normally then succeed, since the new copy of the certificate the server will send will match the copy just installed in the keystore.

Of course, this defeats the whole authentication scheme, since ultimately this example code is trusting whatever certificate the server sent without verifying it. Most likely the author of this code didn't care about authentication, and just wanted to use an SSL connection, even if it was to a bogus server. That is of course what most people do when they browse to a site with an unrecognized certificate, and just accept the certificate.

What's bad about the code is that it installs the unverified certificate in the local machine's main JRE keystore. That means that all other Java applications run on the machine will thenceforth trust the unverified certificate, which is a major security hole. You might want to avoid doing that in your own code.

Answer 2

Below key points you can google and understand.

1. Trusted certificates and keystores/truststores
2. Concept of private and public key. How browser send the certificate key to server and it gets authenticated.
3. Keytool utility, to convert certificates from one form to another.
4. Importing the trusted certificates and keys to trustore, though java has its "cacerts" file by default, but you can create you own and import certificates into it.
5. Explore certificates, read about thumbprint, algorithms used
6. Create you own custom factory as well.