Every registered Voters is called when something calls isGranted.
The fact is that the framework itself (or a bundle f.e) calls isGranted on the request.
You have to use supportsClass, supportsAttribute, ... in order to check if the object is the one you're waiting for, and if not return a VoterInterface::ABSTAIN value.
Take a look at existing implementations (in the framework itself (like RoleVoter) or here: https://github.com/KnpLabs/KnpRadBundle/blob/develop/Security/Voter/IsOwnerVoter.php#L35-L45