I've decided to have extra modules making the relationships so then I have a related field on each of my current modules pointing to a module in the middle where I can customise fields anyway I want "and" I will block the related fields from modification based on user and if the field has been set already etc.
This is an obvious solution but I wanted to have less modules for plain and simple OCD. Once I convinced my brain that not being able to customise the relationship with "control" fields was even worse for OCD than having more modules everything settled down!