SSLException: Invalid padding Unwrapping TLS Application Data

Question

Several months ago I developed an SSL web server using NIO and the SSLEngine. GET requests work great as do small POST requests (under ~10KB). However, I'm getting sporadic SSLException exceptions, when I POST anything larger than that.

For example, if I POST a small image (~16KB), I can see 3 TLS records representing application data. The first is the HTTP header, and the other two contain the payload. Here's a sense of how small the packets are:

1: 613 bytes
2: 16341 bytes
3: 549 bytes

In my code, I call the SSLEngine to unwrap/decrypt the TLS records (application data). Frequently, the SSLEngine chokes when it tries to unwrap the second record. Here's the error:

javax.net.ssl.SSLException: Invalid padding
    at com.sun.net.ssl.internal.ssl.Alerts.getSSLException(Unknown Source)
    at com.sun.net.ssl.internal.ssl.SSLEngineImpl.fatal(Unknown Source)
    at com.sun.net.ssl.internal.ssl.SSLEngineImpl.readRecord(Unknown Source)
    at com.sun.net.ssl.internal.ssl.SSLEngineImpl.readNetRecord(Unknown Source)
    at com.sun.net.ssl.internal.ssl.SSLEngineImpl.unwrap(Unknown Source)
    at javax.net.ssl.SSLEngine.unwrap(Unknown Source)
    at javaxt.http.servlet.HttpServletRequest.getApplicationData(HttpServletRequest.java:1793)
    ...

    Caused by: javax.crypto.BadPaddingException: Invalid TLS padding: 199
    ...

Here's the code that's throwing the error.

private byte[] getApplicationData() throws IOException {

  //Read the next 5 bytes from the socket channel. This should contain TLS
  //record information.
    if (recordHeader==null){
        recordHeader = ByteBuffer.allocateDirect(5);
        read(recordHeader);
    }

  //Verify that the message contains application data
    recordHeader.rewind();
    if(recordHeader.get()!=23) throw new IOException();

  //Get the length of the TLS record
    recordHeader.position(3);
    int recordLength = Integer.parseInt(getHex(recordHeader) + getHex(recordHeader), 16);
    recordHeader.rewind();

  //Read the TLS record
    ByteBuffer recordData = ByteBuffer.allocateDirect(recordLength);
    read(recordData);
    recordData.rewind();

  //Merge the TLS header and record data into a single buffer
    ByteBuffer tlsRecord = ByteBuffer.allocateDirect(recordLength+recordHeader.capacity());
    tlsRecord.put(recordHeader);
    tlsRecord.put(recordData);
    tlsRecord.rewind();

  //Decrypt the application data
    ByteBuffer output = ByteBuffer.allocateDirect(tlsRecord.capacity());
    SSLEngineResult serverResult = sslEngine.unwrap(tlsRecord, output);
    runDelegatedTasks(serverResult, sslEngine);


    byte[] arr = new byte[output.position()];
    output.rewind();
    output.get(arr);


  //Clean up
    recordHeader.clear();
    recordData.clear();
    tlsRecord.clear();
    output.clear();
    recordHeader = recordData = tlsRecord = output = null;


  //Return array
    return arr;
}

The SSLException is thrown at this line:

SSLEngineResult serverResult = sslEngine.unwrap(tlsRecord, output);

This error occurs sporadically. Sometimes I see it, sometimes I don't. Oftentimes, if I do see the error and simply re-POST the data (e.g. refresh the browser), everything works great (no errors).

I am not an SSL expert by any means so I don't know how best to debug the issue. I'm pretty sure I'm calling the unwrap method correctly. Is the second record somehow corrupt? Has the cypher changed and I need to re-initiate a handshake? What do I do when I encounter this error?

Couple other points:

• I'm using Sun/Oracle JDK 6 and Java Secure Socket Extension (JSSE) that comes bundled with it.
• I'm seeing this error a lot when with Mobile Safari, iOS 6. I rarely have this issue with FireFox.

Thanks in advance!

Answer 1

After a long hiatus, I finally got around to resolving this issue. The problem was that when I was reading the TLS record, I was assuming that my read() method was returning all the requested bytes (i.e. the recordData ByteBuffer was full). That was not true. As a result, the sslEngine.unwrap() call was bombing out.

To resolve the issue, I simply replaced this:

  //Read the TLS record
    ByteBuffer recordData = ByteBuffer.allocateDirect(recordLength);
    read(recordData)

With this:

  //Read the TLS record
    ByteBuffer recordData = ByteBuffer.allocateDirect(recordLength);
    int ttl = read(recordData);
    if (ttl<recordLength){
        while (ttl<recordLength){
            recordData.position(ttl);
            ByteBuffer tmp = ByteBuffer.allocateDirect(recordLength-ttl);
            ttl += read(tmp);
            recordData.put(tmp);
        }
        recordData.rewind();
    }

Once the recordData ByteBuffer was filled, the SSLEngine unwrap() method worked flawlessly.

Answer 2

You're doing this all wrong.

You should just call unwrap() when you want incoming data. unwrap() will tell you to read from the network when necessary (BUFFER_UNDERFLOW) via its return status; it may also tell you to run a task, do a wrap, etc. You should not read the network yourself unprompted, peek into the TLS records, read the length, read the data, reassemble them, and feed them to the SSLEngine. It already does all that. Your code is upside down.

Similarly when writing, just call wrap(). It will tell you when to write to the network (BUFFER_OVERFLOW), run a task, etc.