unfortunately the amount of Xmx must fit into you virtual memory.
you may add a huge swap space to get more virtual memory for your processes. I observed, that processes tend to stay in swap space once they got memory pages from there. a little trick is to add another swap of same size as your first and turn the first one off. pages will swap in back into physical RAM. add this as a regular maintainance task.