I had a second look at the web.xml file for the urlrewrite entry. I found that Atlassian had added specific settings for the filter to kick in. So "my" rules never actually kicked in. I changed these to /* and verified that it now worked as expected.
<filter-mapping>
<filter-name>UrlRewriteFilter</filter-name>
<!--<url-pattern>/s/*</url-pattern>-->
<url-pattern>/*</url-pattern>
</filter-mapping>
Now I have to figure out Atlassians intention with the /s/* (I know what they do, but I do not know why they added this. Test installation seems to run just fine).