The problem was caused by a wrong Content-Type
.
When I started making a sample form on a separate orbeon I got a error message in the log:
Body received with non-XML media type for replace="instance": text/html
Changing the service response Content-Type to application/xml
solved the problem.