Java and Tomcat - CodeCache is full. Compiler has been disabled

Question

My Situation

• I have developed a web service for document analysis
• This web service reads txt and pdf documents
• For reading pdf files I am using a java library developed by my university advisor
• I am using Java 7
• I am currently deploying it on Tomcat v7.0
• If I generate a war file from my application and host it on a local tomcat on my pc, or if I start my application on tomcat via eclipse ee, the reading of a pdf file makes no problem

My Problem

• I have set up myself a little server (not the best hardware, dual core cpu, 4g ram), installed tomcat, java and the rest that seemed necessary to me.
• When I now deploy a war file on the servers tomcat using the tomcat manager everything works fine until I try to let the application read a pdf file.
• After the application has read several pages I get an error that the CodeCache has become full and the compile has been disabled because of this.

Example error output during a pdf import

This log shows how the pdf is being imported and on somepoint the error messages start. I have marked the erorr message fat.

...

May 31, 2012 11:15:40 AM infovis.structure.pdf.PDFImport process INFO: Processing Page 13

May 31, 2012 11:15:40 AM infovis.structure.pdf.PDFImport process INFO: Processing Page 14

May 31, 2012 11:15:41 AM infovis.structure.pdf.PDFImport process INFO: Processing Page 15

Java HotSpot(TM) 64-Bit Server VM warning: CodeCache is full. Compiler has been disabled. Java HotSpot(TM) 64-Bit Server VM warning: Try increasing the code cache size using -XX:ReservedCodeCacheSize= Code Cache [0x00007fa43437e000, 0x00007fa4347fe000, 0x00007fa43737e000) total_blobs=1858 nmethods=1318 adapters=490 free_code_cache=44631Kb largest_free_block=45618688 Java HotSpot(TM) 64-Bit Server VM warning: CodeCache is full. Compiler has been disabled. Java HotSpot(TM) 64-Bit Server VM warning: Try increasing the code cache size using -XX:ReservedCodeCacheSize= Code Cache [0x00007fa43437e000, 0x00007fa4347fe000, 0x00007fa43737e000) total_blobs=1859 nmethods=1318 adapters=490 free_code_cache=44631Kb largest_free_block=45618688

May 31, 2012 11:16:19 AM infovis.structure.pdf.PDFImport process INFO: Processing Page 16

May 31, 2012 11:16:20 AM infovis.structure.pdf.PDFImport process INFO: Processing Page 17

Java HotSpot(TM) 64-Bit Server VM warning: CodeCache is full. Compiler has been disabled. Java HotSpot(TM) 64-Bit Server VM warning: Try increasing the code cache size using -XX:ReservedCodeCacheSize= Code Cache [0x00007fa43437e000, 0x00007fa4347fe000, 0x00007fa43737e000) total_blobs=1860 nmethods=1318 adapters=490 free_code_cache=44630Kb largest_free_block=45618688 May 31, 2012 11:17:07 AM infovis.structure.pdf.PDFImport process INFO: Processing Page 18 Java HotSpot(TM) 64-Bit Server VM warning: CodeCache is full. Compiler has been disabled. Java HotSpot(TM) 64-Bit Server VM warning: Try increasing the code cache size using -XX:ReservedCodeCacheSize= Code Cache [0x00007fa43437e000, 0x00007fa4347fe000, 0x00007fa43737e000) total_blobs=1861 nmethods=1318 adapters=490 free_code_cache=44633Kb largest_free_block=45618688

and so on...

What I have tried so far

I have tried to change the cache sizes in the tomcat configuration on my server (I'm not the best in working with the linux shell). I have tried to increase the CodeCache Size, as well as the size of the other caches but the problem still occures. I have checked my code for possible leaks but did not find any yet (Remember, if I start it via eclipse I dont get this message, so this might indicate a tomcat(?) configuration problem). I have also tried to set the parameter "UseCodeCacheFlushing" that should enforece to empty the codecache when it's becoming full, but somehow it doesn't effect the applications breakdown.

My tomcat server configuration

I have read that the default CodeCache size is 32MB or 64MB when it's a 64 bit application. I tried reserving 512mb (maybe I did something wrong in the configuration?) but the problem of course happened again.

You may pass JVM startup parameters to Java here. If unset, the default options will be: -Djava.awt.headless=true -Xmx128m -XX:+UseConcMarkSweepGC

Use "-XX:+UseConcMarkSweepGC" to enable the CMS garbage collector (improved response time). If you use that option and you run Tomcat on a machine with exactly one CPU chip that contains one or two cores, you should also add the "-XX:+CMSIncrementalMode" option. JAVA_OPTS="-Djava.awt.headless=true -Xmx3g -Xms2g -XX:+UseCodeCacheFlushing -XX:+UseG1GC -XX:MaxPermSize=512m -XX:ReservedCodeCacheSize=512m"

My Thoughts on this

During my research I found some notes that CodeCache related problems may indicate memory leak problems, when there is a programming failure, in which the result is that the garbage collector can't empty the cache. This might be possible and sadly I don't have the sourcecode of the library in which I read the pdf. But on the other hand I have no problems in reading a 650 pages pdf on my desktop pc on a local tomcat (quadcore, 4x 3.0ghz, also 4g ram) and this confuses me.

Could it be that this is just a tomcat problem that can get solved if I use another server for deployment, like glassfish?

Can anyone of you help me or has any ideas or suggestions? Maybe I did some configuration wrong? I'm not that experienced in working with tomcat or other servers, so any help is really welcome.

Thanks a lot for every answer and thought that you're sharing with me.

Answer 1

The solution for me was to switch from Tomcat to Glassfish as application server for deploying my application.

After that switch I never re-experienced this CodeCache behaviour.

For making sure this problem is solved I also observed the java-vm running on my server (using jconsole with remote). I didn't see any suspicious behaviour anymore.

Answer 2

Try another JVM like JRockit where you shouldn't have this problem. If it is Tomcat - try look if it logs something about leaks in its logs - it usually does so for classloader problems.