How do I share locks between two processes working on the same region of shared memory?

Question

I'm wondering how to do this (using C++98). Here's my scenario: I have process A and process B. Process A allocates a large buffer in shared memory and splits it into a fixed number of chunks. It then uses a series of structs like these to represent each chunk:

struct Chunk
{
    Lock lock; //wrapper for pthread_attr_t and pthread_mutex_t
    char* offset; //address of the beginning of this chunk in the shared memory buffer
};

The Lock when constructed does this:

pthread_mutexattr_init(&attrs);
pthread_mutexattr_setpshared(&attrs, PTHREAD_PROCESS_SHARED);
pthread_mutex_init(&lock,&attrs); //lock is pthread_mutex_t and attrs is pthread_mutexattr_t

the lock method when called does this:

pthread_mutex_lock(&lock);

It uses placement new when creating the above "Chunk"s into the beginning of the shared memory buffer like this:

char* mem; //pointer to the shared memory
Chunks[i] = new (mem) Chunk; //for i = {1..num chunks}
mem += sizeof(Chunk);

Then it assigns offsets and proceeds to write into the rest of the buffer through it's lifetime. Each time it is writing in the chunk corresponding to one of the aboves it grabs the chunks lock and releases it when done.

Now process B comes up and maps the same shared memory buffer into memory and attempts to retrieve the chunks like this:

Chunk** chunks = reinterpret_cast<Chunk**)(mem); //mem being the pointer into the shared memory

then it tries to operate on the shared memory by scanning around the different chunks and also trying to make use of the lock if needed.

I am getting strange crashes when I run this in which the chunks** is garbage and am wondering if the Lock would work across the processes as well or if there are any other caveats I'm ignoring in the simple steps above? Is having the SHARED pthread attr enough or do I need to use an entirely different breed of lock?

Answer 1

When you pull a region of shared memory into a process, it generally will NOT be located at the same virtual address as in other processes that access the shared memory. So you can't just store the raw pointers into the shared memory and expect them to work meaningfully.

So in your case, even though the Chunk is in shared memory, the offset pointer within each chunk is not meaningful in any other process.

One solution is to use offsets from the beginning of the shared memory chunk

struct Chunk {
    pthread_mutex_t  lock;
    size_t           offset;
};

char *base; // base address of shared memory
char *mem;  // end of in-use shared memory

Chunk *chunks = reinterpret_cast<Chunk *>(mem);  // pointer to array in shared memory
for (int i = 0; i < num_chunks; i++) {
    // initialize the chunks
    new(mem) Chunk;
    mem += sizeof(Chunk); }
// set the offsets to point at some memory
for (int i = 0; i < num_chunks; i++) {
    chunks[i].offset = mem - base;
    mem += CHUNK_SIZE; // how much memory to allocate for each chunk?
}

Now in B you can just do

Chunk *chunks = reinterpret_cast<Chunk *>(base);

but in either process, to access the data of a chunk, you need base + chunks[i].offset

Alternately, you can use a package that manages the shared memory allocation for you and ensures that it gets mapped at the same address in every process.

Answer 2

In addition to my comments I also provide a very basic offset_ptr implementation. I agree that making a simple project dependent on something like boost may be an overkill but even until reaching a certain project size where you decide to switch to a more serious set of libraries it is worth wrapping critical things like offset pointers. Wrapping helps you to centralize your offset handler code and also to guard yourself with asserts. A simple offset_ptr template that doesn't handle all cases can still be much better than hand-coded offset+pointer manipulator code that is copy pasted everywhere and its basic implementation without attempting to be comprehensive is:

template <typename T, typename OffsetInt=ptrdiff_t>
class offset_ptr
{
    template <typename U, typename OI> friend class offset_ptr;
public:
    offset_ptr() : m_Offset(0) {}
    offset_ptr(T* p)
    {
        set_ptr(p);
    }
    offset_ptr(offset_ptr& other)
    {
        set_ptr(other.get_ptr());
    }
    template <typename U, typename OI>
    offset_ptr(offset_ptr<U,OI>& other)
    {
        set_ptr(static_cast<T*>(other.get_ptr()));
    }
    offset_ptr& operator=(T* p)
    {
        set_ptr(p);
        return *this;
    }
    offset_ptr& operator=(offset_ptr& other)
    {
        set_ptr(other.get_ptr());
        return *this;
    }
    template <typename U, typename OI>
    offset_ptr& operator=(offset_ptr<U,OI>& other)
    {
        set_ptr(static_cast<T*>(other.get_ptr()));
        return *this;
    }
    T* operator->()
    {
        assert(m_Offset);
        return get_ptr();
    }
    const T* operator->() const
    {
        assert(m_Offset);
        return get_ptr();
    }
    T& operator*()
    {
        assert(m_Offset);
        return *get_ptr();
    }
    const T& operator*() const
    {
        assert(m_Offset);
        return *get_ptr();
    }
    operator T* ()
    {
        return get_ptr();
    }
    operator const T* () const
    {
        return get_ptr();
    }

private:
    void set_ptr(const T* p)
    {
        m_Offset = p ? OffsetInt((char*)p - (char*)this) : OffsetInt(0);
    }
    T* get_ptr() const
    {
        return m_Offset ? (T*)((char*)this + m_Offset) : (T*)nullptr;
    }

private:
    OffsetInt m_Offset;
};

offset_ptr<int> p;
int x = 5;

struct TestStruct
{
    int member;
    void func()
    {
        printf("%s(%d)\n", __FUNCTION__, member);
    }
};

TestStruct ts;
offset_ptr<TestStruct> pts;

int main()
{
    p = &x;
    *p = 6;
    printf("%d\n", x);

    ts.member = 11;
    if (!pts)
        printf("pts is null\n");
    pts = &ts;
    if (pts)
        pts->func();
    pts = nullptr;
    if (!pts)
        printf("pts is null again\n");

    // this will cause an assert because pts is null
    pts->func();
    return 0;
}

It may contain some operator functions that are pain in the ass to write if you are not used to implement pointer stuff but this is really simple compared to a full fledged pointer implementation of a complex lib like boost, isn't it? And it makes the pointer (offset) manipulator code much nicer! There is no excuse for not using at least a wrapper like this!

Wrapping the locks and other stuff is also fruitful even if you do it for yourself without external libs because after writing/using the locks with bare native api for 2-3-4 times it truns out that your code will look much nicer if you use a simple wrapper with a ctor/destructor/lock()/unlock() not to mention the centralized lock/unlock code that can be guared with asserts and occasionally you can put in debug info to the lock class (like the id of the last locker thread to debug deadlocks easier...).

So don't omit wrappers even if you are rolling your code without std or boost.