I worked it out, the error message was misleading, it wasn't that I had too many mqueues open, it was that the mqueues had allocated for these-selves most of the memory assigned to the mqueue system. So when I tried to open a new mqueue it couldn't get enough memory for it and failed to open.
Looking at the code for the other processes I should be able to free up the space by just reducing the maxmsg value. Some are using queue sizes of 1000, the one above I copy pasted from another process is set to 512 messages, which seems very excessive. Did a quick test by halving that value in two processes and now I have my 6th queue running.