MemoryMappedFile or serialization, speed on very large object

Question

I have a project that has a fairly complicated, nested (references calling the referenced objects up and down) object (class) stored in a dictionary like:

public static Dictionary<string, Object1> DObject { get; set; }

Object1 is a complicated class. It is not like a "address book" or "person info" class. The class itself has arrays, dictionaries of another object. Those object references class above it, etc.

Because the size of the DObject can be 1GB+ in memory, I'm going to serialize it to a file using BinaryFormatter. Because I need to load this object, I was thinking of using MemoryMappedFile. New keys and values might be added to the dictionary. Objects might have more data in them (added/updated), etc. Does the MMF change size? How do I access a certain key in the DObject in the memory mapped file? Is there a searching mechanism like a hashtable in memory so I can find a certain key like a dictionary and get its value? How is this MMF work anyway?

My ideal thought is. Have a large file on disk (2GB+). I update the object in the dictionary fast on the disk like having a memory saved on the disk. Everything is fast. Key-value look up all the way. I look up, edit the value, save, etc... I would have to have quick access to this 2GB+ object any time I want. If the WCF server restarts, I would need quick access to this 2GB+ size object. That's why I was thinking 1) serialization 2). load and read from/to MMF. My main concern is speed at this moment. I can't be reloading this 2GB data from scratch everytime I debug the project in other words (it takes long time).

Any suggestions, ideas and thoughts on how I should handle this situation.

Answer 1

You can't keep DObject itself in a memory mapped file. .NET objects tend to move around all the time, and you can't force them to stay in one place in memory (you could pin them, but you really don't want to). You can serialize the object and write it to file, but that would mean writing more than 1GB of data each time.

Why not go the standard way, and 'serialize' your object to a set of database tables?