Worklight Push notification (APNS) fails with javax.net.ssl.SSLHandshakeException

Question

I have a Worklight 6.1 app on iOS with push notifications. It works fine, until something goes wrong on the Worklight server. All push notifications after that fail until the server is restarted.

The Worklight server is Liberty 8.5.5.0 on Linux x86_64 Using JDK : java-1.7.0-ibm-1.7.0.5.0.x86_64 The Worklight database is Derby (this is a test server)

Everything works great when I start the server.

10 minutes after the server starts, I see:

[2/20/14 19:39:15:319 CST] 0000003e com.notnoop.apns.internal.ApnsFeedbackConnection             W Failed to retreive invalid devices
java.lang.RuntimeException: javax.net.ssl.SSLHandshakeException: com.ibm.jsse2.util.j: End user tried to act as a CA

But that doesn't seem to break anything. Push notifications still work. 30 minutes after server start, I see:

[2/20/14 19:59:48:657 CST] 00000061 com.ibm.ws.webcontainer.util.ApplicationErrorUtils           E SRVE0777E: Exception thrown by application class 'org.jboss.resteasy.core.SynchronousDispatcher.handleApplicationException:365'
org.jboss.resteasy.spi.UnhandledException: org.springframework.dao.InvalidDataAccessApiUsageException: Multiple DISTINCT aggregates are not supported at this time. {SELECT COUNT(DISTINCT t0.DEVICE), COUNT(DISTINCT t1.ID) FROM NOTIFICATION_DEVICE t0 INNER JOIN NOTIFICATION_USER t1 ON t0.USERSUBSCRIPTIONID = t1.ID WHERE (t0.APPLICATIONID IN (?) AND t0.PLATFORM = ?)} [code=30000, state=42Z02]; nested exception is <openjpa-1.2.2-r422266:898935 nonfatal user error> org.apache.openjpa.persistence.ArgumentException: Multiple DISTINCT aggregates are not supported at this time. {SELECT COUNT(DISTINCT t0.DEVICE), COUNT(DISTINCT t1.ID) FROM NOTIFICATION_DEVICE t0 INNER JOIN NOTIFICATION_USER t1 ON t0.USERSUBSCRIPTIONID = t1.ID WHERE (t0.APPLICATIONID IN (?) AND t0.PLATFORM = ?)} [code=30000, state=42Z02]

And any call to the push adapter after that results in:

[2/21/14 19:06:44:038 CST] 00000090 com.notnoop.apns.internal.ApnsConnectionImpl                 I Exception while waiting for error code
java.net.SocketException: Socket is closed
    at com.ibm.jsse2.qc.j(qc.java:301)
    at com.ibm.jsse2.e.read(e.java:32)
    at java.io.InputStream.read(InputStream.java:102)
    at com.notnoop.apns.internal.ApnsConnectionImpl$1MonitoringThread.run(ApnsConnectionImpl.java:114)

[2/21/14 19:06:44:650 CST] 00000085 com.notnoop.apns.internal.ApnsConnectionImpl                 I Failed to send message Message(Id=2; Token=499D9813FBC377CCDE787E2749CDA914F826EDF39B0830D4AFEEF7A5D71A1802; Payload={"aps":{"alert":{"body":"You have 4 available messages","action-loc-key":null},"sound":"","badge":4},"payload":"{\"alias\":\"myNotificationPush\"}"})... trying again after delay
javax.net.ssl.SSLHandshakeException: com.ibm.jsse2.util.j: End user tried to act as a CA

And all the retries after that result in the same failure.

There are some google hits on "End user tried to act as a CA" and "APNS" that suggest this is a JDK 1.5 problem (fixed in 1.6), and that it can be fixed by using the IbmPKIX trust manager, but I am certain that I am using JDK 1.7, and the trust manager in java.security is PKIX

Answer 1

Once again … this took long enough to figure out.

The underlying problem was that the Worklight server was hosted on SoftLayer, and evidently the SoftLayer firewall by default monitors socket connections, and if a connection is idle for ~10 minutes, drops it.

So, the worklight server connected to APN and push worked. Then 10 minutes passed without any push notifications, and the firewall dropped the socket between the Worklight server and the APNS server. All subsequent push requests failed silently because Worklight had a dead socket for the APNS service.

We reconfigured the firewall to allow the connection to APNS to sit idle indefinitely and the problem was resolved.

Answer 2

After upgrading to the Worklight 6.0.0.2 fix pack, we've seen the same issue on Websphere running JDK 7. After some research I found this issue reported for IBM JDK 7. There is an issue with the Entrust certificate that is included in the JDK. I tried the workaround with replacing the cacerts file from the JDK 6 and this worked.

 Replace jre\lib\security\cacerts file in JDK 7 with
 jre\lib\security\cacerts file in JDK 6

http://www-01.ibm.com/support/docview.wss?uid=swg1IV43936

Note: To establish a TLS session with APNs, an Entrust Secure CA root certificate must be installed on the provider’s server. If the server is running OS X, this root certificate is already in the keychain. On other systems, the certificate might not be available. You can download this certificate from the Entrust SSL Certificates website.

https://developer.apple.com/library/ios/documentation/NetworkingInternet/Conceptual/RemoteNotificationsPG/Chapters/CommunicatingWIthAPS.html