In Google API console for your project (https://code.google.com/apis/console/)
Make sure that your "Public API access" or "Key for server application" has allowed the IP address of your server.
Also make sure you have the proper "Redirect URIs", if you get refused, look at your php logs, I had this:
PHP Fatal error: Uncaught exception 'Google_ServiceException' with message 'Error calling GET https://www.googleapis.com/mirror/v1/timeline?maxResults=3&key=: (403) Access Not Configured. Please use Google Developers Console to activate the API for your project.'
use your $base_url + /oauth2callback.php to find out what the correct Redirect URI is.
I managed to solve the issue by adding the IP address and REDIRECT URIs and it runs both on my local machine and on the server. (same code, same key)
PS: I want to thank Scarygami, disabling the $client->setDeveloperKey($key); helped me find the solution.