After looking at Jenkins LOGS I learned a few things :
- This plugin (github pull-request builder) fully replaces all the other build-triggers (including git and SCM poll). I was using all three, and when I thought this plugin was working because it was building correctly, was actually the other triggers.
- You must set your project github URL on the github plugin. This plugin depends on the github plugin to work. This is what was preventing it from working in my configuration.
After you fix #2, you can remove the triggers from SCM Poll and Git to make sure the pull-request plugin is triggering the build. When it does, it actually adds a link to the pull request to the activity feed in Jenkins (very nice) and updates the status on github correctly.