I found the answer to this problem today:
I didn't notice it when copying configuration details into Apple's Jar Bundler, but I had this in the VMOptions parameter (Info.plist file):
<key>Java</key>
<dict>
[...]
<key>VMOptions</key>
<string>-Djava.awt.headless=true</string>
[...]
</dict>
Passing -Djava.awt.headless=true
to the Java VM was the problem. After removing this parameter, the application behaved just as I wanted. It doesn't have a window but a simple menu with quit option is there, so users can easily stop the application.