I don't know anything about netbeans or how to control its predexing behavior, but it's a simple matter to run dx on a jar from the command line
dx --dex --output=classes.dex blah.jar
Or, since you need to control the memory options
dx -JXmx1G --dex --output=classes.dex blah.jar
You can find dx at <android-sdk>/platform-tools/dx
On the newer versions of sdk , you can find dx at
<android-sdk>/build-tools/android-<ver>/