After further investigation we found websphere was scanning jars resource annonations and this happens on 1st request. There is configuration on websphere that will make sure this scanning doesn't happen. If that is setup correctly it works as expected.
The property is document as such like this
com.ibm.ws.webcontainer.SkipMetaInfResourcesProcessing = true
we got this from http://pic.dhe.ibm.com/infocenter/wasinfo/v8r0/index.jsp?topic=%2Fcom.ibm.websphere.nd.multiplatform.doc%2Finfo%2Fae%2Fae%2Frweb_jsp_staticfile.html