I had to set "Try to obtain root privileges if admin user" under OSX in the "Request Privileges" action in the startup folder under the installer section.
This is still a genuine bug in install4j (that there's no error message when you don't do this), but at least there's a way to make it work.