You can use web services in jBPM, at least in version 5 and 6. I believe the tooling support in eclipse is minimal or non-existent however. Don't know about the support in web console. See Maciej's blog for details on that.
It is possible to use custom objects and own libraries in jBPM. You just have to make the libraries available to the jBPM runtime. If you're using your own application where you run jBPM, just include the libraries in the CLASSPATH. For the jBPM console, you will have to include the libraries in the lib/ directory of the war, or inside WEB-INF/classes if you have .class files.
Not sure if there is out of the box support for exposing jBPM services over web service. If all you want to do is orchestrate other web services and expose yourself as a web service, you could consider using a BPEL engine rather, like Apache ODE