Android SSLEngine example

Question

I need to work with a TCP socket over TLS for an app I'm working on. I've been through dozens of examples and while I have no problem getting through the handshake, I can't seem to read the input stream through any means (tried a lot, including readline(), reading to character array, etc). every time I try, the app freezes on that spot. If I debug, it never goes to the next line of code.

In an attempted solution, I decided to move over to using an SSLEngine, since that's supposed to be the Java 1.5 answer to java.nio for SSL. However, I have found one example (here: http://docs.oracle.com/javase/7/docs/technotes/guides/security/jsse/samples/sslengine/SSLEngineSimpleDemo.java) which is more than a little confusing to me, and I've not been successful implementing it. When I try, the unwrap() call yields an empty buffer, where I know (from using OpenSSL on the command line) that the service in question pushes data back down the pipe.

Suggestions are welcome, I've burned way too much time on this already. Here's the relevant code:

SSLEngine engine = sslContext.createSSLEngine(uri.getHost(), uri.getPort());
            engine.setUseClientMode(true);
            engine.beginHandshake();
            SSLSession session = engine.getSession();
            int bufferMax = session.getPacketBufferSize();
            int appBufferMax = session.getApplicationBufferSize() + 50;
            ByteBuffer cTo = ByteBuffer.allocateDirect(bufferMax);
            ByteBuffer sTo = ByteBuffer.allocateDirect(bufferMax);

            ByteBuffer out = ByteBuffer.wrap(sessionId.getBytes());
            ByteBuffer in = ByteBuffer.allocate(appBufferMax);


            debug("sending secret");
            SSLEngineResult rslt = engine.wrap(out, cTo);
            debug("first result: " + rslt.toString());
            sTo.flip();
            rslt = engine.unwrap(sTo, in);
            debug("next result" + rslt.toString());

Answer 1

This implementation is missing some key pieces. Namely the handshake can bounce between several states NEED_WRAP, NEED_UNWRAP, NEED_TASK to negotiate a connection. This means you cannot just call one and then the other. You will need to loop over the states until a handshake has completed.

   while (handshaking) {
      switch (state) {
          case NEED_WRAP:
              doWrap();
              break;
          case NEED_UNWRAP:
              doUnwrap();
              break;
          case NEED_TASK:
              doTask();
              break;
        }
    }

A full working example of Java SSL and NIO

Now that said, you should be aware the SSLEngine on Android is broken. Google recommends using threads and blocking sockets according to that thread.

Answer 2

I have written something to make using SSLEngine easier. It can be used with NIO or for other use cases. Available here SSLFacade

Answer 3

unwrap() can yield an empty buffer if what was unwrapped was an SSL handshake message or alert, rather than application data. There's not enough information here to say more. What was the engine status afterwards?

Answer 4

beginHandshake does not proceed the handshake, it is just used to inform the SSLEngine that you want to perform the handshake for the next calls to wrap/unwrap. It's useful when you want to do another handshake. For the initial one, it is not needed as the first call to wrap will initiate the handshake.

Besides, you have to check the result of the wrap and unwrap methods to know if all the data has been correctly encoded. It can happen that you have to call the methods several times to process all the data.

The following link might help: http://onjava.com/onjava/2004/11/03/ssl-nio.html

Or this question: SSL Handshaking Using Self-Signed Certs and SSLEngine (JSSE)