Try:
credentials = SignedJwtAssertionCredentials(
'my@developer.gserviceaccount.com',
key,
sub='superadmin@mydomain.com',
scope=['https://www.googleapis.com/auth/admin.directory.user',])
You don't need both scopes, use readonly if you're doing read operations only, use the above if you're doing read and write.
sub= defines which Google Apps account the service account should impersonate to perform the directory operations, it's necessary and the account needs to have the right permissions.
Lastly, be sure that you've granted the service account's client_id access to the directory scopes you need in the Control Panel. The steps to do this are listed in the Drive documentation, just sub in the correct scope(s) for Admin Directory.