Slow Dynamic GSP Reloading in Production on AIX

Question

We are using Grails 2.2.4, WebSphere 8.0.0.5 all running on AIX 6.1.0.0. Websphere is using the IBM JDK:

Java(TM) SE Runtime Environment (build pap6460_26sr3ifix-20121005_02(SR3+IV27268+IV27928+IV28217+IV25699))

IBM J9 VM (build 2.6, JRE 1.6.0 AIX ppc64-64 20120919_122629 (JIT enabled, AOT enabled)

J9VM - R26_Java626_SR3_iFix_1_20120919_1316_B122629

JIT - r11.b01_20120808_24925ifx1

GC - R26_Java626_SR3_iFix_1_20120919_1316_B122629 J9CL - 20120919_122629)

JCL - 20120713_01

The problem is that using:

grails.gsp.enable.reload = true
grails.gsp.view.dir="/path/to/gsp/views"

is slow, and by that I mean a good 20 seconds to render a small GSP. What's interesting is that in our local development environments it takes 2 seconds.

We've isolated this problem by having a controller that does nothing except call render(..) on a blank GSP with nothing in the model, so I can only assume it's the compilation but I could be wrong.

Has anyone come across other instances where rendering GSPs is extremely slow, or have any suggestions, perhaps it's some sort of weird JDK issue on AIX?

In addition to the bounty, whoever answers this correctly gets free waffles.

EDIT Just noticed this the other day: there are three environments with the same WAS config and setup and one of them works fine, so it is definitely some sort of environment issue.

Answer 1

Turned out the problem was https://plugins.grails.org/plugin/sergiomichels/grails-melody-plugin. Who knew!

Answer 2

I agree with your suspicion that it's compilation time. Perhaps you grails.gsp.view.dir is slow - a networked file system perhaps?

The real answer, unfortunately, is to not enable GSP reload in production. It's clearly meant as a development convenience, and is not intended to perform well in production.

Answer 3

Make sure sitemesh is being preprocessed

grails.views.gsp.sitemesh.preprocess=true

Also I suspect this to be locking issue rather than compiling issue.

To at least reduce this problem set the following configs

grails.gsp.reload.interval= time in milliseconds.

something high depending on your comfort. Perhaps every hour?

If your file is changing the last modified time too quickly you need to lower the granularity by

grails.gsp.reload.granularity= Time in milliseconds. 

Limit the number of classes being reloaded by

grails.reload.excludes 

and

grails.reload.includes

Also remember the view path MUST end in slash. I didn't see that in the example you provided.

Answer 4

While I can't tell you what is the cause for your problem, I can point you to a tool that could help you look into the performance problem.

The Grails Miniprofiler plugin is a fantastic tool for examining end to end performance, all the way to the view (which is where you believe your problem is).

On some GSPs on a project of mine, I was surprised to see how heavy some views got with respect to the SQL calls (via lazy loading).

You may have suspicions for where the problems lie, and maybe you're hitting an obscure bug on that specific platform, but having hard numbers to point to bottlenecks is extremely useful.

For what it's worth, I haven't seen the problem you've described in any of my OS/environments. But I do remember experiencing serious pain with trying to deploy to JBoss 6.1 (since resolved) so I am sensitive to the idea that Grails could have issues on some environments.

Best of luck...

Grails Miniprofiler plugin

Answer 5

We are having the same problem on our application that tried to render a Birt report on Gsp page(it takes some times 5 minutes on prod server,We used Tomcat and Mysql) , not answer but a suggestion you might need to use the grails cache plugin ,that has a capability of cache gsp pages based on you requirement.

grails.org/plugin/cache‎