Actually the solution to my problem was very easy, it just needed a bit more reading. When the match request is created a property playerGroup can be set to restrict only the players from a group to connect to players from the same group.
But it seems that setting this property when when inviting a friend does not work, it works only on auto-match. So if anyone plans on doing that, you have to find a way around it.
GKMatchRequest *request = [[GKMatchRequest alloc] init];
request.minPlayers = 2;
request.maxPlayers = 4;
request.playerGroup = MyMap_Forest | MyRulesCaptureTheFlag;
If your game sets the playerGroup property, only players whose requests share the same playerGroup value are automatched by Game Center. The value of a player group is arbitrary. For example, you could define different playerGroup values to implement any of the following filters:
A game could restrict players based on skill level. A game that provides multiple game modes could use it to filter players into the specific game they want to play. A game that provides bonus content through in-app purchase could match players who own the same content with each other.
Setting playerGroup to 0 or not setting it at all means that there is not a player group for that match so anyone can connect to anyone.