Regarding the issue :BUILD FAILED C:\jbpm-installer\build.xml:508: JBoss application server did not start within 5 minutes
Check the jboss serverlog for issues.
One thing I got was that, the jbpm 6 default installation tries to download a sample jbpm application project which is hosted in github. If we have no access to gihub it would the server startup would wait and would fail.
You can also increase the timeout ( But for sure if there is no other problem jboss boots up faster than the 5 mins limit ) in the build.xml goto target and change the attribute maxwait in the waitfor tag.
For an assurance, I could get the jbpm environment up with the install.demo in windows 8 64 bit machine, the only problem was the connectivity with github. And the only difereence was I was using Java 1.6