I've finally got Glassfish, JackRabbit and JAAS working together so that I can create a Subject using my custom LoginModule that JackRabbit then uses to create a session. Below are the steps that I took to resolve this issues described in my original question:
Instead of using the JackRabbit RAR (Model 2) I now include the JackRabbit Jars inside my war (Model 1). This allowed me to implement my own custom AccessManager that does not rely on the JackRabbit typed principles. The biggest disadvantage of this approach is that I now have to create and shutdown the repository myself. The solution I went with was an ApplicationScoped CDI Producer that creates the repo and the shuts it down in the dispose method. This makes it easy to inject the repo into the classes.
I solved the issue with JackRabbit finding the Subject in Glassfish by patching jackrabbit-core. It appears this issue has been around for some time see (JCR-3188), and a patch has been provided but never included in the source code. I applied the patch to 2.6.4 and JackRabbit is now able to find and use the Subject in Glassfish.