An easier way would probably just to mmap
the relevant offset of /dev/mem
in your userspace program directly. This allows you to access the physical memory layout by seeking into it.
AFAIK, this is what the RaspberryPi developers have done to make GPIO memory mapped I/O registers available to userspace programs.