I didn't realize at first that the foreign keys are referencing a index rather than a field. So by creating a new primary key on the same field didn't mean that the foreign keys switched to the new index.
I now dropped the unique constraint using cascade and re-added all the foreign keys, they are now using the primary key.