Currently New Relic isn't able to detect traffic that's delivered by any protocol other than HTTP, including WebSockets and JMS. If you're willing to do some customization, you can get transactions detected using some annotations and/or custom instrumentation via .XML
The easiest way to sample how this would work is to add the New Relic @Trace annotation to your code and see what's recorded by New Relic https://docs.newrelic.com/docs/java/custom-java-transaction-traces
I'd get in touch with New Relic support directly for further help with this. support.newrelic.com