JetS3t client putObject stopped working with HTTPS hostname invalid exception

Question

Basic put object calls suddenly stopped working (sometimes it succeds). It has been working since long.

Looks like a SSL cert issue.

Stack Trace snippet.

   org.jets3t.service.S3ServiceException: S3 PUT connection failed for '/s3_request_message-38afbd8e-7d65-428a-a708-5d34104ded95-4912660956668093023.xml'
            at org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:516)
            at org.jets3t.service.impl.rest.httpclient.RestS3Service.performRestPut(RestS3Service.java:800)
            at org.jets3t.service.impl.rest.httpclient.RestS3Service.createObjectImpl(RestS3Service.java:1399)
            at org.jets3t.service.impl.rest.httpclient.RestS3Service.putObjectImpl(RestS3Service.java:1317)
            at org.jets3t.service.S3Service.putObject(S3Service.java:1661)
            at org.jets3t.service.S3Service.putObject(S3Service.java:1914)
            at com.amazon.lm.utils.aws.S3Box.putFile(S3Box.java:111)
            at com.amazon.lm.engine.LMEngine.copyRequestS3(LMEngine.java:350)
            at com.amazon.lm.engine.LMEngine.run(LMEngine.java:165)
            at com.amazon.lm.engine.discover.DiscoveryEngine.run(DiscoveryEngine.java:156)
            at com.amazon.lm.engine.discover.GoogleBaseSearch.run(GoogleBaseSearch.java:25)
            at com.amazon.lm.ui.UIDiscoverTask.run(UIDiscoverTask.java:41)
            at java.lang.Thread.run(Thread.java:662)
    Caused by: javax.net.ssl.SSLPeerUnverifiedException: HTTPS hostname invalid: expected 'lm-requests-prod.s3.amazonaws.com', received '*.s3.amazonaws.com'
            at org.apache.commons.httpclient.contrib.ssl.StrictSSLProtocolSocketFactory.verifyHostname(StrictSSLProtocolSocketFactory.java:293)
            at org.apache.commons.httpclient.contrib.ssl.StrictSSLProtocolSocketFactory.createSocket(StrictSSLProtocolSocketFactory.java:215)
            at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707)
            at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361)
            at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387)
            at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
            at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
            at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
            at org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:342)
            ... 12 more

Looks like Java does not like the wildcard domain presented as '*.s3.amazonaws.com'

per Can Java connect to wildcard ssl... wildcards can be problematic with java.

But as said earlier, we have been using it since long time and suddenly started facing this issue, that too intermittently.

We are using following versions:

jdk: 1.6 
jets3: 0.7
openssl:1.0

Has anyone faced this issue? If Yes, Is there any workaround?

Answer 1

This wasn't an issue with the AWSS3JavaClient code, based on the fact that the this problem was happening both with S3 library and with other Java S3 libraries, and the fact that SSL cert verification is done inside the JVM platform library code, not inside our S3 library code.

The problem is that our JVM's keystore didn't have the most recent certificate authorities (CAs) that allow the JVM to form a chain of trust for whatever cert we're getting from the S3 SSL endpoint. This is a fairly common problem with Java and SSL, since the JVM maintains it's own keystore (i.e. it doesn't use certs from the OS).

If you face this problem, try reproducing this issue with other JVMs. Whenever customers have seen this issue in the past, it's been because their local JVM keystore (the keystore ships with the JVM and contains the most recent certs and CAs) has been out of date. Upgrading to the latest JVM version has always fixed this in the past.

Try upgrading your JVM version to recent one, it should help because your keystore must have been expired! :)