Life is easier in the Entity Framework if you use foreign keys:
Why does Entity Framework Reinsert Existing Objects into My Database?
Making Do with Absent Foreign Keys
And you are absolutely correct to say that the foreign key is not really relevant to the domain model. It is part of the persistence model.
So you need to decide which camp to join. Are you a purist or a pragmatist? Separate domain models and persistence models or not?