Ok so actually it was kinda of hidden but I found it.
So you need to set the redirect uri to the "out of band" uri set up within the credentials.
urn:ietf:wg:oauth:2.0:oob
The Client doesn't actually sets it to that value but I guess google defaults it to it.
The current signet gem does not allow redirect_uri to be set to that value so I added a PR to the main repo, In the mean time I'm using my fork and it works fine.