Ruby : The best way to manage a large 3d array

Question

I would like to know what is the best way to manage a large 3d array with something like :

x = 1000 y = 1000 z = 100

=> 100000000 objects

And each cell is an object with some amount of data.

Simple methods are very loooooong even if all data are collapsed (I a first tryed an array of array of array of objects)

class Test

    def initialize
        @name = "Test"
    end

end

qtt = 1000*1000*100
Array.new(qtt).each { |e| e = Test.new }

I read somewhere that DB could be a good thing for such cases.

What do you think about this ?

What am I trying to do ?

This "matrix" represents a world. And each element is a 1mx1mx2m block who could be a different kind (water, mud, stone, ...) Some block could be empty too.

But the user should be able to remove blocks everywhere and change everything around (if they where water behind, it will flow through the hole for exemple.

In fact what I wish to do is not Minecraft be a really small clone of DwarfFortress (http://www.bay12games.com/dwarves/)

Other interesting things

In my model the groud is at level 10. It means that [0,10] is empty sky in most of cases. Only hills and parts of mountains could be present on those layers.

Underground is basicaly unknown and not dug. So we should not have to add instances for unused blocks.

What we should add from the beginning to the model : gems, gold, water who could stored without having to store the adjacent stone/mood/earth blocks.

At the beginning of the game, 80% of the cube doesn't need to be loaded in memory.

Each time we dig we create new blocks : the empty block we dug and the blocks around.

The only things we should index is :

• underground rivers
• underground lakes
• lava rivers

Answer 1

Holding that many objects in memory is never a good thing. A flat-file or database-centric approach would be a lot more efficient and easier to maintain.

What I would do - The object-oriented approach

Store the parameters of the blocks as simple data and construct the objects dynamically.

Create a Block class to represent a block in the game, and give it variables to hold the parameters of that particular block:

class Block
  # location of the Block
  attr_accessor :x, :y, :z

  # an individual id for the Block
  attr_accessor :id

  # to define the block type (rock, water etc.)
  attr_accessor :block_type

  # and add any other attributes of a Block...
end

I'd then create a few methods that would enable me to serialise/de-serialise the data to a file or database.

As you've stated it works on a board, you'd also need a Board class to represent it that would maintain the state of the game as well as perform actions on the Block objects. Using the x, y, z attributes from each Block you can determine its location within the game. Using this information you can then write a method in the Block class that locates those blocks adjacent to the current one. This would enable you to perform the "cascading" effects you talk about where one Block is affected by actions on another.

Accessing the data efficiently

This will rely entirely on how you choose to serialise the Block objects. I would probably choose a binary format to reduce unnecessary data reads and store the objects via their id parameter, and then use something like MMIO to quickly do random-access reads/writes on a large data file in an Array-like manner. This will allow you to access the data quickly and efficiently, without the memory overhead. How you read the data will relate to your adjacent blocks method above.

You can of course also choose the DB storage route which will allow you to isolate the Blocks and do lookups on particular blocks in a higher-level manner, however that might give you a bit of extra overhead.

It sounds like an interesting project, I hope this helps a bit! :)

P.S With regards to the comment above by @Linuxious about choosing a different language. Yes this might be true in some cases, but a skilled programmer never blames his tools. A program is only as efficient as the programmer makes it...unless you're writing it in Java ;)