-XX:+ExitOnOutOfMemoryError
works as expected with JRockit R28.2.2:
$ jrockit-jdk1.6.0_29/bin/java -Xmx20m -XX:+ExitOnOutOfMemoryError OOM
java.lang.OutOfMemoryError: allocLargeObjectOrArray: [B, size 40976
at jrockit/vm/Allocator.allocLargeObjectOrArray(JIZ)Ljava/lang/Object;(Native Method)
at jrockit/vm/Allocator.allocObjectOrArray(Allocator.java:349)
at jrockit/vm/Allocator.allocArray(Allocator.java:257)
at OOM.<init>(OOM.java:3)
at OOM.main(OOM.java:9)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
[ERROR] Exit on OutOfMemory requested. Exiting.
JRockit aborted: Exit requested on OOM (51)
Which version of JRockit are you using? Did you spell the parameter correctly?