it seems there is a manual workaround possible: http://community.jaspersoft.com/jasperreportsr-server/issues/3716#comment-808481
we implemented a servlet filter class to try to set the Content-Disposition header of the response in the cases when we knew that the MIME type was wrongly set. As we knew that the response was flushed after being processed by the web service end point, we set the header BEFORE being processed as
Content-Disposition: attachment; filename='filename.extension'
. This turned out to work, and we were able to download the file with an appropriate file extension.
but they also mention that it would work with a v5.6.0 although it did not in our tests (see comment above: Opening file content resource (Excel) of JasperReports Server / Tomcat with Internet Explorer displays binary data inline)
v5.6.0, and apparently on this release the MIME type of the response was correctly set, so we finally get to a proper solution for our problem.