Why does iterating over data result in massive memory usage?

Question

I've got ten users, each with thousands (the counts are exactly 98956, 6954, 159732, 0, 40176, 541819, 218012, 52380, 93203, and 432) of path items. I need to iterate over each user, and work with their paths. I'm finding that each iteration increases memory usage, and after only a few users I'm using over a gigabyte of RAM.

I've simplified the code down to just gathering the paths, and the excessive memory usage is still occurring:

User.find_each do |user|
  points = user.paths.map { |path| [path.latitude, path.longitude, path.id] }
end

As somewhat of a side-note, I'm also experiencing this excessive memory usage with delayed_job, where I have one job for each user. So, rather than gathering multiple users with User.find_each do |user|, it's just grabbing one user with user = User.find(id). The memory usage grows from job to job, until eventually delayed_job exits (and I haven't been able to find any logging that corresponds with this - it simply goes away).

Is there anything that I can do to manage memory usage?

I'm using ruby-1.9.3-p484 and Rails 3.2.16

Answer 1

I think you use find_each wrong! Since you have too few users with a lot of paths, try to iterate a user's path with find_each so that not all paths of a user are loaded to memory.

User.find_each do |user|
  points = []
  user.paths.find_each { |path| points << [path.latitude, path.longitude, path.id] }
end