Using Ruby to convert a range to an array and running out of memory

Question

I'm doing something that seems simple at first

Ruby Code:

(0...693530740).to_a

This results in

NoMemoryError: failed to allocate memory

Well i'm stumped. Is there any way to change how much memory the ruby interpreter can use? I don't see how. I have no way around this as it has to exist in my code. Is there a better way? Is there perhaps a lower level language that can do this? Any ideas are welcome. I have already tried using jRuby and giving the jvm 15GB of memory but no luck.

Thank you for reading this question

Answer 1

Why do you need to make an array with hundreds of millions of items? Do you really need to have all those items in memory at the same time, or can you process them one by one, allowing the ones which have already been processed to be garbage-collected? Something like this:

(0...693530740).each do |n|
  # do something
end

Answer 2

"I have no way around this as it has to exist in my code"

Alter your code so that you don't need it. You only have one piece of meaningful data here, the number 693530740, which fits just fine into 8 bytes. It is very unlikely that you really need to expand it into that huge array. Most of Ruby's array methods that you might think you need will have equivalents (using Range, or Enumerator) that work without needing to instantiate such a list of numbers.

If you have trouble seeing what kind of re-design would avoid the large array, then post a new question - here on Stack Overflow if the design can be outlined in a short description and a few lines of code. Perhaps to codereview.stackexchange.com if it not possible to demonstrate your algorithm in a small piece of code.

Answer 3

So you want to allocate an array with ~650 million numbers. which is ~2.6GB . Note that this memory must be continuous. Without knowing how much physical memory you have, I think this is the main reason you can't do it