PWC1231 and PWC6344

Question

I have two servers running on Glassfish 2.1 both have the same web app.

Two times this error occurred: Some jsp pages stop displaying only showing a blank page, and the following errors are printed in the logs...

PWC1231: Servlet.service() for servlet jsp threw exception java.io.FileNotFoundException: /path/to/jsp/file/jsp_file.jsp.java (Permission denied) at java.io.FileOutputStream.open(Native Method) at java.io.FileOutputStream.(FileOutputStream.java:179) at java.io.FileOutputStream.(FileOutputStream.java:70) at org.apache.jasper.compiler.AntJavaCompiler.getJavaWriter(AntJavaCompiler.java:213) at org.apache.jasper.compiler.Compiler.generateJava(Compiler.java:173) at org.apache.jasper.compiler.Compiler.compile(Compiler.java:409) at org.apache.jasper.JspCompilationContext.compile(JspCompilationContext.java:592) at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:344) at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:470) at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:364) at javax.servlet.http.HttpServlet.service(HttpServlet.java:831) at org.apache.catalina.core.ApplicationFilterChain.servletService(ApplicationFilterChain.java:411) at org.apache.catalina.core.ApplicationDispatcher.doInvoke(ApplicationDispatcher.java:855) at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:703) at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:542) at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:474) at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:366) at org.apache.struts.action.RequestProcessor.doForward(RequestProcessor.java:1056) at org.apache.struts.tiles.TilesRequestProcessor.doForward(TilesRequestProcessor.java:261) at org.apache.struts.action.RequestProcessor.processForwardConfig(RequestProcessor.java:388) at org.apache.struts.tiles.TilesRequestProcessor.processForwardConfig(TilesRequestProcessor.java:316) at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:231) at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1164) at org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:415) at javax.servlet.http.HttpServlet.service(HttpServlet.java:738) at javax.servlet.http.HttpServlet.service(HttpServlet.java:831) at org.apache.catalina.core.ApplicationFilterChain.servletService(ApplicationFilterChain.java:411) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:317) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:198) at com.my.app.filtro.FiltroCallcenter.doFilter(FiltroCallcenter.java:90) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:230) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:198) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:288) at org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:271) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:202) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:632) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577) at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:94) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:206) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:632) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:571) at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1080) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:150) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:632) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:571) at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1080) at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:272) at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.invokeAdapter(DefaultProcessorTask.java:637) at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.doProcess(DefaultProcessorTask.java:568) at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.process(DefaultProcessorTask.java:813) at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.executeProcessorTask(DefaultReadTask.java:341) at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:263) at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:214) at com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:265) at com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:106)

Followed by this:

PWC6344: Unable to create output writer for file /path/to/jsp/file/jsp_file.jsp.java|#]

Sometimes only the PWC6344 error is printed and sometimes both, the PWC1231 error is always followed by PWC6344 (which somewhat makes sense, because that exception is thrown when an IOException occurs).

Well, both times this errors happened, the only thing I did was stop and start the instance, and everything was alright again, also this error has only occurred in one of the servers.

Whats is happening ?... Or how can I diagnose what is causing this so I can fix the problem instead of stopping and re starting for eternity...

UPDATES:

I was looking into the possibility of this been a file descriptor problem, as suggested by sbridges but!, the maximum number of file handlers is 811975 with 4520 opened files in one server and 359532 with only 6894 in the other.

So, I guess its safe to say this is not the problem!

Does someone have another theory?

Answer 1

It looks like the permissions are set incorrectly and you can't write the compiled jsp page to disk,

/path/to/jsp/file/jsp_file.jsp.java (Permission denied) 

Are the permissions correct on that directory/file?

Answer 2

Rename jsp_file.jsp.java to jsp_file.jsp.

Answer 3

PWC6344: Unable to create output writer for file /path/to/jsp/file/jsp_file.jsp.java|#]

This can also happen if the underlying operating system has run out of file handlers/descriptors which are required in order to open a file for reading or writing. I'm not closely familiar with CentOS, but Google hints that it has a "relatively low" limit of 1024 given the lot of problems related to it in the search results. Among the results you'll see a lot of questions/answers as to how to increase it, such as the following blog:

Increase the number of file descriptors on Centos and Fedora Linux

Raising the number of file descriptors for a regular user on CentOS/Fedora/Redhat is surprisingly difficult to learn how to do. There are lots of incomplete walk throughs on the web, some with typos and other problems.

Here are the steps that worked for me to raise the open file descriptor limit from 1024 (the default) to 65535:

1. As root, edit /etc/sysctl.conf and add the line:

    fs.file-max = 512000
   

2. At the bash prompt, run:

    $ sysctl -p
   

   That will cause the settings to take effect. You can also cat 512000 > /proc/sys/fs/file-max but that may reset on reboot.

3. Edit /etc/security/limits.conf and add the following:

    * - nofile 65535
   

   See the inline comments for more details on what that does and how to make it more restrictive if you prefer.

4. As root, run

    $ ulimit -n 65535
   

   and make sure you have no errors. To double check, run ulimit -n and make sure the response is 65535.

5. Ensure that PAM authentication is turned on for SSH, or else when you try to connect as a regular user, you won’t see the new limits. Edit /etc/ssh/sshd_config and make sure you have:

    UsePAM yes
   

   Restart SSH /sbin/service sshd restart if you made any changes.

6. Login as a regular user with a new SSH session & shell and run:

    $ ulimit -n 65535
   

   Run ulimit -n again to check and good luck!