Spring: Cannot connect to a JMX Server using RMI from behind a firewall

Question

My Spring application is running on a machine that is behind a NAT firewall (pfSense). The machine's internal IP is a.b.c.d, and the NAT IP is w.x.y.z

The Spring configuration's serviceUrl is set to my internal IP (a.b.c.d) on port 1100, and when I start the application, I provide the following switches:

 -Dcom.sun.management.jmxremote -Djava.rmi.server.hostname=w.x.y.z -Dcom.sun.management.jmxremote.port=1099 -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false

As shown above, I set -Djava.rmi.server.hostname=w.x.y.z in order to make it possible to connect to the application through the NAT. I have also opened up the relevant ports on the machine's firewall, and set up a port forward from w.x.y.z:1100 to a.b.c.d:1100

When I then try to connect to the app from outside the network using jconsole on w.x.y.z:1100, I get a java.io.IOException: jmxrmi

 java.io.IOException: jmxrmi
at sun.tools.jconsole.ProxyClient.checkSslConfig(ProxyClient.java:236)
at sun.tools.jconsole.ProxyClient.<init>(ProxyClient.java:127)
at sun.tools.jconsole.ProxyClient.getProxyClient(ProxyClient.java:483)
at sun.tools.jconsole.JConsole$3.run(JConsole.java:524)
 Caused by: java.rmi.NotBoundException: jmxrmi
at sun.rmi.registry.RegistryImpl.lookup(RegistryImpl.java:136)
at sun.rmi.registry.RegistryImpl_Skel.dispatch(Unknown Source)
at sun.rmi.server.UnicastServerRef.oldDispatch(UnicastServerRef.java:409)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:267)
at sun.rmi.transport.Transport$1.run(Transport.java:177)
at sun.rmi.transport.Transport$1.run(Transport.java:174)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:556)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:811)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:670)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:275)
at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:252)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:378)
at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source)
at sun.tools.jconsole.ProxyClient.checkSslConfig(ProxyClient.java:234)

If I try to connect using jconsole on w.x.y.z:1099, I get a java.rmi.ConnectException (Connection refused). How can I expose my JMX MBeans outside my NATted firewall?

 java.rmi.ConnectException: Connection refused to host: w.x.y.z; nested exception is: 
java.net.ConnectException: Operation timed out
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:619)
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216)
at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:129)
at javax.management.remote.rmi.RMIServerImpl_Stub.newClient(Unknown Source)
at javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2373)
at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:297)
at sun.tools.jconsole.ProxyClient.tryConnect(ProxyClient.java:355)
at sun.tools.jconsole.ProxyClient.connect(ProxyClient.java:313)
at sun.tools.jconsole.VMPanel$2.run(VMPanel.java:292)
Caused by: java.net.ConnectException: Operation timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at java.net.Socket.connect(Socket.java:528)
at java.net.Socket.<init>(Socket.java:425)
at java.net.Socket.<init>(Socket.java:208)
at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:40)
at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:147)
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:613)
... 9 more

Interestingly enough, I can connect to both ports (1099 and 1100) on w.x.y.z using telnet.

The relevant portion of my Spring configuration is:

<bean id="registry" class="org.springframework.remoting.rmi.RmiRegistryFactoryBean">
    <property name="port" value="1100" />
</bean>
<bean id="serverConnector" class="org.springframework.jmx.support.ConnectorServerFactoryBean" depends-on="registry">
    <property name="server" ref="mbeanServer" />
    <property name="objectName" value="connector:name=rmi" />
    <property name="serviceUrl" value="service:jmx:rmi://a.b.c.d:1100/jndi/rmi://a.b.c.d:1100/server" />
    <property name="environment">
        <props>
            <prop key="jmx.remote.jndi.rebind">true</prop>
        </props>
    </property>
</bean>
<bean id="mbeanServer" class="org.springframework.jmx.support.MBeanServerFactoryBean">
    <!-- indicate to first look for a server -->
    <property name="locateExistingServerIfPossible" value="true"/>
</bean>

Note: I am not using any servlet containers.

EDIT: The first answer provided me with the basis for the solution to my specific problem, which can be seen in the second answer.

Answer 1

Elaborating on the solution I identified using the links and pointers anttix gave me in his answer.

So, as mentioned in the answer above, using the system properties com.sun.management.jmxremote.port and com.sun.management.jmxremote.rmi.port is the basis for the solution.

I changed my Spring config to remove the registry and serverConnector beans, as this was not allowing me to register port 1099 (PermissionExceptions resulted in my being unable to bind to port 1099), and used the platform mbean server.

The Spring config I used was just this:

 <bean id="mbeanServer" class="java.lang.management.ManagementFactory" factory-method="getPlatformMBeanServer"/>

Then, I started the application using the following switches:

 -Dcom.sun.management.jmxremote -Djava.rmi.server.hostname=w.x.y.z -Dcom.sun.management.jmxremote.port=1099 -Dcom.sun.management.jmxremote.rmi.port=1099 -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false

Setting the java.rmi.server.hostname to the NAT VIP w.x.y.z is critical as we need to be able to execute remote operations on locally created remote objects from outwith the network.

Note: This solution will only work on Java 7u4 or higher.

Answer 2

The initial port you define with com.sun.management.jmxremote.port is called a registry port and is only used to start negotiation and determine next port(s) to use for "real" communication. Java RMI mechanism uses dynamically allocated ports and in general is not compatible with firewalls.

What port is used by Java RMI connection?

That said, for JMX it is possible to work around it

a) Use system properties to lock both ports (requires Java 7)

com.sun.management.jmxremote.port
com.sun.management.jmxremote.rmi.port

b) Use custom code to request a specific port. See JConsole over ssh local port forwarding

See also:

• Why Java opens 3 ports when JMX is configured?
• http://realjenius.com/2012/11/21/java7-jmx-tunneling-freedom/