You could use error pages to internally route traffic to a different location in the server block, and then use that location block to proxy to varnish.
example:
server {
error_page 519 = @google;
if ($http_user_agent ~* Googlebot) {
return 519;
}
location @google {
proxy_pass http://varnish;
}