Have you tried specifying a larger heap size when running FitNesse. Start with the following:
java -Xmx500 -jar fitnesse-standalone.jar -p 8080
The default value is 64MB, so you can start as low as 200 and keep going up from there until your suite runs successfully.
Obviously change the port number to match your setup.