Affinity groups are an availability and scalability concept that is physically implemented within the fabric controller, and the Azure data centre, however they (Microsoft) sees fit. For example, affinity groups help scalability by co-locating services in the same rack (or switch, or something).
Since there should be no availability and scalability dependencies (logical or physical) between your different environments, there should be no need to share affinity groups. If one of your environments, say staging, was also a backup/recovery environment for production - then affinity groups would come into play. But if that was the case, the staging environment would be in a separate data centre (region) anyway.
Virtual networks and other IaaS should, at least architecturally, follow a similar pattern. If you have three environments (production, staging, and test) you should have three different virtual networks, and three different VPN gateways. This may be difficult to get past the on-prem networking guys (who are more familiar with physical networking equipment), but you have to separate them - you don't want test data clogging up or breaking the production (virtual) network do you?