After days of debugging, reading tons of forums and posts I have found the solution:
a modest class variable string that grows until a memory leak occours.
Some useful notes that I have earned in my trip:
Curb vs HTTParty
Between these two gems that perform curl requests, the best in term of performance is Curb
.
http://bibwild.wordpress.com/2012/04/30/ruby-http-performance-shootout-redux/
Pay attention at the class variables
My problem was a debug/info variable string class that continues growing, avoid to use class variable that are never collected by the garbage collector.
In my specific case was:
@status = "#{@status} Warning - response is empty for #{description}\n"
Perform some manual garbage collection
Perform some manual GC.start
at the critical point to ensure to free the memory that are no more necessary. Remember that calling GC.start
doesn't perform an instantaneous call to the garbage collector, it only suggests it.
Calling ActiveRecords array
When calling big ActiveRecords use .find_each
, e.g.:
Model.find_each(:batch_size => 50) do |row|
This perform a query only for 50 (or something smaller than default value) row every time, better than calling a single query with 1k row. (I guess that the default batch_size
is 1000).
Useful links: