Persistence of boost multi_index_container stored in a file

Question

Can I rely on boost multi_index_container allocated within memory mapped file? Will be this kind of "database" portable between computers with the same endianess?

Answer 1

No, this is not safe. The address of the first byte in the memory mapped file cannot be guaranteed to be the same between invocations.

Boost's memory mapped file takes a hint, but it is only a hint.

So in the first invocation, the memory might be located at 0xBAADF00D. On the second invocation, it might be located at 0xF00DBAAD. Pointers within the structure would no longer be valid, as they would point to memory around 0xBAADF00D rather than around 0xF00DBAAD.

As a serious problem, it might usually work in testing, as the hint might usually be obeyed -- but sometimes there will already be stuff at that memory address, and the hint will have to be ignored.

(There are other parts that make this really hard -- but the above makes it pretty infeasible).

Now, strategies like this can work, but require very intrusive introspection of the data structures in question. You'd have to do a pass like DLL readdressing in order to get the data structures to have the correct offsets, and by that point you might as well just serialize it flat.

Answer 2

With the help of Boost.Interprocess's memory mapped files and a little care you can persist multi_index_containers; the problem of mappings being located at different addresses can be overcome using offsets rather than pointers, and Boost.MultiIndex is prepared to handle that: see this example where a sort of simplistic database is implemented with access from several processes at a time.

This said, using memory mapped files for persistence is an extremely fragile approach, as it will only work if these parameters are the exactly the same:

• machine architecture
• operating system version
• compiler version
• compilation parameters (which might affect padding, for instance)
• version of all relevant libs used (Boost included), as ABIs might change across versions
• some other aspects I forgot about :-)

Everything considered, using Boost.Serialization as suggested by @alfC is probably the best solution for portable persistence across different environments. If you're concerned about speed, a mixed approach like this might work:

• When in session and within the context of several processes (ideally, out of the same executable) using the same structure in the same machine, memory-map the container.
• Once the session is finished (or every so often to guard against crashes etc.), portably persist using Boost.Serialization.

Answer 3

If what you want is to transport a multi_index from one computer to another the best thing is to use Boost.Serialization, which works out of the box:

http://www.boost.org/doc/libs/1_60_0/libs/multi_index/doc/tutorial/creation.html#serialization

#include <boost/archive/text_oarchive.hpp>
#include <boost/archive/text_iarchive.hpp>
#include <fstream>
...
void save(const employee_set& es)
{
  std::ofstream ofs("data");
  boost::archive::text_oarchive oa(ofs);
  oa<<es;
}

void load(employee_set& es)
{
  std::ifstream ifs("data");
  boost::archive::text_iarchive ia(ifs);
  ia>>es;
}
...
employee_set es; // a multi_index type
... // fill it with data
save(es);
...
employee_set restored_es;
load(restored_es);