we think we found the answer. We saw a similar bug reported in the glassfish jira:https://java.net/jira/browse/GLASSFISH-16254. It seems to be a bug with glassfish 3.0.1.
They had an out of memory error when glassfish monitoring for thread pool and http service was turned on which is the exact setup we had.
We turned off the glassfish monitoring and now we are running stable for 1 week without any out of memory.
Thanks for everyone's help!