Tracing memory leak in Ruby on Rails 3 / Postgres / Apache Passenger application

StackOverflow https://stackoverflow.com/questions/5369623

Question

Hello,

we have recently updated an application to Rails 3.0.4 (3.0.5 on online devel server). Most of the changes from 2.3.10 to 3.0.4 were due to obsolete or outdated plugins and gems, and were solvable with relative ease. But one thing makes me go mad:

Every single web request, in development mode, causes the server process to allocate about 50-60 MB more memory than before. This memory is not freed after the request, at least not all of it. After 10-20 requests, every Ruby instance consumed over 500 MB of RAM, while our previous Rails 2.3.10 instances were rarely above 200 MB.

This makes it impossible to run our 1300 tests, because the devel machine's 4GB of RAM is filled before the end of the tests. It only happens in development mode with cache_classes = false. If I switch cache_classes to true, the Rails instances will consume about 200MB of memory, and then stay there. However, during tests, even with cache_classes = true, memory usage will grow.

I queried ObjectSpace and found out that with each request, about 3500 new Proc, up to 50'000 new Strings and 3000 new Hashes and Arrays are created and not freed. These strings (when dumped) contained my whole source code including plugins and gems, documentation, source code comments and path names. (Why?)

To find the cause for this, here's what I tried: (After every change, I hammered the apps with ab -n 50.)

  1. I created a fresh Rails 3 application with a single resource and controller and SQLite3 DB. Memory usage started at 60 MB and stayed below 80 MB.
  2. I changed 'sqlite3' to 'pg' and pointed the new Rails 3 app to my existing Postgres DB. Memory usage started at 110 MB and did not grow beyond 130MB. (Side question: Why does the Postgres gem use so much more memory than the SQLite3 gem?)
  3. I copied over my Gemfile and Gemfile.lock from the broken Rails3 app to the bare bones app and ran bundle install. No change, memory stayed at about 115MB, no matter how many requests were made.
  4. I created an empty "def FooController; def foo; render :text => 'foo' end; end" in the broken Rails3 app. Memory usage grew more slowly, but it still never stopped growing after requests.
  5. I deleted every route except for the FooController route. No change.
  6. I disabled all Gems except for the following: pg, rails, aasm, will_paginate, geokit-rails3, koala, omniauth, paperclip. No change.
  7. I disabled every before_filter and after_filter in ApplicationController and every nonessential include in environment.rb. I also synced boot.rb, environment.rb and application.rb with my bare-bones Rails 3 app, except for five relatively simple observers, autoloading files in /lib and filter_parameters. No change. Every new request still consumed an additional 10-50 MB of RAM.

If you have an idea what is going wrong here, and where the memory leak could be, I would really appreciate any help. I am running Rails 3.0.4 on OS X Snow Leopard, Rails 3.0.5 on Debian Lenny, and

Thank you!

Coming closer:

I have removed every plugin, every gem, every extension and everything that I did not personally write myself, so that my application is basically naked. Especially, I removed these plugins: acts_as_list, acts_as_tree, asset_packager, forgot_password, fudge_form, fudge_scaffold, paperclippolymorph, query_trace, rails_upgrade, repeated_auto_complete-0.1.0, role_requirement, to_select, validates_url, and ym4r_gm.

Now my application - only the above FooController still works! - starts up with 65MB and never goes beyond 75MB of RAM, even after hammering it with ab -n 1000 -c1 (1000 HTTP requests to /foo using ApacheBench). Unfortunately, without the plugins, this is also the only URI that works at all.

After some digging, it seems that a combination between the Restful Authentication and Acts As State Machine (AASM) plugins causes the memory leak. See also https://github.com/Satish/restful-authentication/issues#issue/11. I'm not sure yet why, and just doing "include AASM" in my bare-bones project does not cause RAM usage growth just by itself.

I will investigate further.

Culprit found

It is AASM. In Rails 3 it seems to leak AASM::xxx object instances. see

Second culprit found

There was another memory leak in rspec. This made my tests almost unbearably slow, even after removing AASM, because two parallel running rspec tasks (using https://github.com/grosser/parallel_tests) took almost 3GB of memory at the end. See https://github.com/rspec/rspec-core/issues/#issue/321.

Licensed under: CC-BY-SA with attribution
Not affiliated with StackOverflow
scroll top