This does not seem like an VNC issue- either the ports are not open, they are being blocked b a firewall, or there is some kind of permission/authentication issue with the app itself related to monitoring it.
In order to eliminate VNC as the cause (and use localhost in a local connection on jconsole), do "ssh -X REMOTHOST -n jconsole" and see. This will also eliminate the overhead of running the full X server and VNC.
Also on linux you can find out what process holds a port open by doing: netstat -ap | grep PORT_NUMBER on the remote host you want to run on.
Colin