The above code will work if you replace the provider block with:
var provider = new AssertionFlowClient(GoogleAuthenticationServer.Description, certificate)
{
ServiceAccountId = SERVICE_ACCOUNT_EMAIL,
Scope = DirectoryService.Scopes.AdminDirectoryOrgunit.GetStringValue(),
ServiceAccountUser = SERVICE_ACCOUNT_USER //"my.admin.account@my.domain.com"
};
I had seen this in another post and tried it with my standard user account and it didn't work. Then I read something that suggested everything had to be done with an admin account. So, I created a whole new project, using my admin account, including creating a new service account, and authorising it. When I tried it, it worked. So, then I put the old service account details back in but left the admin account in. That worked, too.