Question

Here's the story:

I use Pentaho Kettle for querying web stats from Google Analytics via Simple API Access. For this request I need the API key. Now this API key turns invalid every now and then (I am not sure what the rythm is) and then the requests fail of course. So I want to generate a new one, receive it and make it available to the ETL-job and its GA steps.

My plan is to do this via Java embedded in one or more "User Defined Java Class" steps via google-api-java-client and a Service Account. The Kettle job generates a new API key, receives it and provides the API key via a field or directly as a paramter.

But primarily I am interested in a general Java solution for the described use case. If somebody has a Kettle-solution that would be even better but I mention those details chiefly to put the question into a context.

Question: How do I generate a new API key for Google Analytics Simple API Access and receive it using google-api-java-client via OAuth2 and without user-interaction (fully automated)?

Bounty: I would appreciate an answer that either provides a Java program or a detailed sequence of API calls. The output of the program is a functioning API key appliable for Simple API Access. Given the expected work involved I chose the highest possible bounty.


I registered a Service Account so I have the following IDs & Co. available:

  • Client ID
    • 123.apps.googleusercontent.com
  • E-Mail address
    • 123@developer.gserviceaccount.com
  • Public key fingerprints
    • abcxxx
  • client_secrets.json
    • {...}
  • private key
    • abcxxx-privatekey.p12

client_secrets.json:

{"web":{
  "auth_uri":            "https://accounts.google.com/o/oauth2/auth",
  "token_uri":           "https://accounts.google.com/o/oauth2/token",
  "client_email":        "123@developer.gserviceaccount.com",
  "client_x509_cert_url":"https://www.../123@developer.gserviceaccount.com",
  "client_id":           "123.apps.googleusercontent.com",
  "auth_provider_x509_cert_url":"https://www.googleapis.com/oauth2/v1/certs"
}}
Was it helpful?

Solution

A couple of things before the code:

  1. Technically if you're using OAuth 2.0 for authentication then Simple API access doesn't apply at all. So you don't need to worry about the API Key. The instructions from Pentaho are old and don't apply unless you're using a non-OAuth 2.0 authentication mechanism (Which isn't what you should be doing)

  2. What you mostly like want to use is a Service Account for authentication which does not require any human interaction.

Register the App

If you haven't already registered your application with the Google Cloud Console, then set up a project and application in the Cloud Console. The system guides you through the process of choosing or creating a project and registering a new application, and it automatically activates the API for you.

If you've already registered your application with the Cloud Console, then follow this procedure instead:

Go to the Google Cloud Console. Select a project. In the sidebar on the left, select APIs & auth. In the displayed list of APIs, make sure the Analytics API status is set to ON. In the sidebar on the left, select Registered apps. Select an application. In either case, you end up on the application's credentials page.

To set up a service account, expand the Certificate section. Then select Generate Certificate. (If you already have a certificate, you can add a new key by selecting Generate New Key.) A dialog box appears; select Download private key to proceed. When it's finished downloading, select View public key.

After downloading the file and closing the dialog, you will be able to get the Service Account's email address.

Download the Java Client Library for Analytics

Visit https://developers.google.com/api-client-library/java/apis/analytics/v3

Go to the Add Library to Your Project section at the bottom and download the Google Analytics API v3 Client Library for Java.

Add Service Account Email to GA Account

Add the service account email to the Google Analytics account(s) you want to access through the using the API.

Sample Java App to Get # of Visits

The following code snippet is for the Core Reporting API. It authenticates using the service account. It then retrieves the visits for a Analytics profile and outputs the number.

To use this then you should obviously replace the Service account email and the path to the private key file. Also you need to import the Java client for analytics and the relevant JARS from the libs folder.

Also refer to the Google Analytics Core Reporting API docs to learn to query the API with dimensions and metrics, segments, filters, etc.

You should be able to customize this for use with your application.

import java.io.IOException;
import java.security.GeneralSecurityException;
import java.util.HashSet;
import java.util.Set;

import com.google.api.client.googleapis.auth.oauth2.GoogleCredential;
import com.google.api.client.http.HttpTransport;
import com.google.api.client.http.javanet.NetHttpTransport;
import com.google.api.client.json.jackson2.JacksonFactory;
import com.google.api.services.analytics.Analytics;
import com.google.api.services.analytics.AnalyticsScopes;
import com.google.api.services.analytics.model.GaData;

public class AnalyticsSample  {

    private static final String APPLICATION_NAME = "APP_NAME";

    private static final String SERVICE_ACCOUNT_EMAIL = "<YOUR_SERVICE_ACCOUNT_EMAIL>@developer.gserviceaccount.com";

    /** Path to the Service Account's Private Key file */
    private static final String SERVICE_ACCOUNT_PKCS12_FILE_PATH = "/path/to/-privatekey.p12";

    private static Analytics service;

    public static Analytics getAnalyticsService() throws GeneralSecurityException, IOException {
        Set<String> scopes = new HashSet<String>();
        scopes.add(AnalyticsScopes.ANALYTICS_READONLY); // You can set other scopes if needed 

        HttpTransport httpTransport = new NetHttpTransport();
        JacksonFactory jsonFactory = new JacksonFactory();
        GoogleCredential credential = new GoogleCredential.Builder()
            .setTransport(httpTransport)
            .setJsonFactory(jsonFactory)
            .setServiceAccountId(SERVICE_ACCOUNT_EMAIL)
            .setServiceAccountScopes(scopes)
            .setServiceAccountPrivateKeyFromP12File(
                    new java.io.File(SERVICE_ACCOUNT_PKCS12_FILE_PATH))
            .build();
        Analytics service = new Analytics.Builder(httpTransport, jsonFactory, null)
            .setHttpRequestInitializer(credential)
            .setApplicationName(APPLICATION_NAME)
            .build();
        return service;
    }

     public static void main(String[] args) {
         try {
             service = getAnalyticsService();
             GaData result = service.data().ga().get(
                     "ga:<YOUR PROFILE ID>",
                     "2013-10-16", // Start date
                     "2013-10-17", // End Date
                     "ga:visits").execute();  // Add Dimensions and metrics

             System.out.println(result.getRows().get(0).get(0)); // Just an example of output

         } catch (IOException e) {
             System.err.println(e.getMessage());
         } catch (GeneralSecurityException e) {
             System.err.println(e.getMessage());
         } catch (Throwable t) {
             t.printStackTrace();
         }
         System.exit(1);
     }

}

OTHER TIPS

i'm not a Java guy, but i am doing pretty much exactly what you're asking for using Go and the Google API. i can give you an overview of the process, hopefully this helps/sends you in the right direction? :)

overview:

  • download the certificate from your registered app in the google cloud console
  • convert the certificate from PCS12 encoding into PEM
  • craft a JWT (JSON web token) using that key, along with the email address (the foobarbazxxxxetc@developer.gserviceaccount.com one) and the API scope that you want to use)
  • add the same email address with appropriate permissions to the analytics account you want to use
  • use the JWT to get an oauth token
  • create an analytics service, authenticating using the oauth token
  • now you can talk to core reporting API

EDIT: no key regeneration neeeded. the token lasts 1 hour, and cannot be refreshed. but that doesn't matter because the next time you authenticate using the key, you'll get a new token lasting an hour.

EDIT 2: i'm including clean compiling Go code with docs/pre-reqs/comments. hope this helps a bit more!

package main

// overview of crqs.go:
// this example app and supporting documentation enables you to query analytics data via the Core Reporting API in a completely headless manner, with
// no "popup" or user intervention required. it doesn't "handle" the returned data very well, as that will depend on what metrics/dimensions your
// query has. better handling of this is left as an exercise for the reader. :) follow the pre-reqs, change the ga* consts specific to your own app,
// and you should be good to go. uses google-api-go-client, analytics/v3, oauth, and oauth/jwt which you can download and install from
// https://code.google.com/p/goauth2/ and http://code.google.com/p/google-api-go-client/

// docs/pre-reqs:
// 1. create a Project within the Google Cloud Console (https://cloud.google.com/console#/project)
// 2. within that Project, ensure you've turned on the Analytics API (APIs & Auth --> Analytics --> On)
// 3. within that Project, create an application of type "Web Application" (APIs & Auth --> Registered Apps)
// 4. click on the application name. expand the "Certificate" section. you will see that the email/public keys
//    have not been generated. click "Generate". download the private key (.p12 file). also, note the private key password, and
//    the email address, you will need these later. you will set the string const gaServiceAcctEmail to this email address.
// 5. download the JSON file, too. you will set the const gaServiceAcctSecretsFile to the location of this file.
// 6. now we need to convert the .p12 key to a PEM key. in a unix shell:
//    $ openssl pkcs12 -in privatekeyfilename.p12 -nodes -nocerts > privatekeyfilename.pem
//    it will ask you for the Import password. enter the private key password you were given in step 4. you should see
//    "MAC verified OK", and you will have the PEM key. you will set the const gaServiceAcctPEMKey to the location of this file.
// 7. goto Google Analytics. in Admin section specific to the GA account/property you wish to query, give Read & Analyze permissions to the
//    email address you were given in step 4.
// 8. now we need the Table ID of the GA account/property mentioned in step 7. login to the Google Analytics Query Explorer 2 at
//    http://ga-dev-tools.appspot.com/explorer/ then select the appropriate account/property/profile in the dropdown menus. you will see
//    the Table ID in the "*ids" box. it looks like "ga:1234556". you will set the string const gaTableID to this value.
// 9. that should be all you need to do! compile and "go".

// example runs:
// 1. shows total pageview count for all URLs (starting at default date october 1st through today).
// $ ./crqs
// gaStartDate=2013-10-01, gaEndDate=2013-10-18
// gaMetrics=ga:pageviews
// gaFilter=ga:pagePath=~^/
// len(gaData.Rows)=1, TotalResults=1
// row=0 [25020179]
//
// 2. shows unique pageview count per URL for top 5 URLs starting in "/movies" (starting at default date october 1st through today),
// in descending order.
// $ ./crqs -m=ga:uniquePageviews -d=ga:pagePath -s=-ga:uniquePageviews -f=ga:pagePath=~^/movies -x=5
// gaStartDate=2013-10-01, gaEndDate=2013-10-18
// gaMetrics=ga:uniquePageviews
// gaDimensions=ga:dimensions=ga:pagePath
// gaSortOrder=ga:sort=-ga:uniquePageviews
// gaFilter=ga:pagePath=~^/movie
// len(gaData.Rows)=5, TotalResults=10553
// row=0 [/movies/foo 1005697]
// row=1 [/movies/bar 372121]
// row=2 [/movies/baz 312901]
// row=3 [/movies/qux 248761]
// row=4 [/movies/xyzzy 227286]
//
// 3. shows unique pageview count per URL for top 5 URLs from Aug 1 --> Sep 1, in descending order.
// $ ./crqs -sd=2013-08-01 -ed=2013-09-01 -m=ga:uniquePageviews -d=ga:pagePath -s=-ga:uniquePageviews -x=5
// gaStartDate=2013-08-01, gaEndDate=2013-09-01
// gaMetrics=ga:uniquePageviews
// gaDimensions=ga:dimensions=ga:pagePath
// gaSortOrder=ga:sort=-ga:uniquePageviews
// len(gaData.Rows)=5, TotalResults=159023
// row=0 [/ 4280168]
// row=1 [/foo 2504679]
// row=2 [/bar 1177822]
// row=3 [/baz 755705]
// row=4 [/xyzzy 739513]

import (
    "code.google.com/p/goauth2/oauth"
    "code.google.com/p/goauth2/oauth/jwt"
    "code.google.com/p/google-api-go-client/analytics/v3"
    "encoding/json"
    "flag"
    "fmt"
    "io/ioutil"
    "log"
    "net/http"
    "os"
    "time"
)

// constants
const (
    // don't change these
    dateLayout string = "2006-01-02" // date format that Core Reporting API requires
    // change these! add in your own app and analytics-property-specific values
    gaServiceAcctEmail       string = "your-applications-email-address@developer.gserviceaccount.com"           // email address of registered application
    gaServiceAcctSecretsFile string = "/path/to/client_secret_your-application.apps.googleusercontent.com.json" // registered application JSON file downloaded from Google Cloud Console
    gaServiceAcctPEMKey      string = "/path/to/your-applications-converted-privatekey.pem"                     // private key (PEM format) of registered application
    gaScope                  string = "https://www.googleapis.com/auth/analytics.readonly"                      // oauth 2 scope information for Core Reporting API
    gaTableID                string = "ga:12345678"                                                             // namespaced profile ID of the analytics account/property/profile from which to request data
)

// globals
var (
    // vars for the runtime flags
    gaDimensions string
    gaEndDate    string
    gaFilter     string
    gaMetrics    string
    gaMaxResults int64
    gaSortOrder  string
    help         bool
    gaStartDate  string
)

// types

// struct to read the registered application's JSON secretsfile into
type GAServiceAcctSecretsConfig struct {
    ClientEmail  string   `json:"client_email"`
    ClientId     string   `json:"client_id"`
    ClientSecret string   `json:"client_secret"`
    RedirectURIs []string `json:"redirect_uris"`
    Scope        string
    AuthURI      string `json:"auth_uri"`
    TokenURI     string `json:"token_uri"`
}

// func: init()

// initialisation function for the command line flags/options.
func init() {
    flag.BoolVar(&help, "h", false, "Show all command line arguments.")
    flag.StringVar(&gaDimensions, "d", "", "GA query dimensions")
    flag.StringVar(&gaEndDate, "ed", "", "GA query end date")
    flag.StringVar(&gaFilter, "f", "", "GA filter to apply")
    flag.Int64Var(&gaMaxResults, "x", 10000, "GA maximum # of results to return (default: 10000)")
    flag.StringVar(&gaMetrics, "m", "ga:pageviews", "GA metrics you want to query (default: ga:pageviews)")
    flag.StringVar(&gaSortOrder, "s", "", "GA query reply sort order")
    flag.StringVar(&gaStartDate, "sd", "2013-10-01", "GA query start date (default: 2013-10-01)")
}

// func: readConfig()

// reads in the registered application's JSON secretsfile and crafts an oauth.Config which
// it returns to the calling func. exits hard if either the JSON secretsfile load fails,
// or if the unmarshal fails.
func readGAServiceAcctSecretsConfig() *oauth.Config {
    configfile := new(GAServiceAcctSecretsConfig)
    data, err := ioutil.ReadFile(gaServiceAcctSecretsFile)
    if err != nil {
        log.Fatal("error reading GA Service Account secret JSON file -", err)
    }
    err = json.Unmarshal(data, &configfile)
    if err != nil {
        log.Fatal("error unmarshalling GA Service Account secret JSON file -", err)
    }
    return &oauth.Config{
        ClientId:     configfile.ClientId,
        ClientSecret: configfile.ClientSecret,
        Scope:        gaScope,
        AuthURL:      configfile.AuthURI,
        TokenURL:     configfile.TokenURI,
    }
} //

// func: main()

// the main function. duh.
func main() {
    // grok the command line args
    flag.Usage = usage
    flag.Parse()
    if help {
        flag.Usage()
    }
    // read in the registered application's JSON secretsfile and get an oauth.Config in return.
    oauthConfig := readGAServiceAcctSecretsConfig()
    // read in the registered applications private PEM key.
    pemKey, err := ioutil.ReadFile(gaServiceAcctPEMKey)
    if err != nil {
        log.Fatal("error reading GA Service Account PEM key -", err)
    }
    // create a JWT (JSON Web Token).
    jsonWebToken := jwt.NewToken(gaServiceAcctEmail, oauthConfig.Scope, pemKey)
    // form the JWT claim set.
    jsonWebToken.ClaimSet.Aud = oauthConfig.TokenURL
    // create a basic httpclient. we will use this to send the JWT.
    httpClient := &http.Client{}
    // build the request, encode it, and then send the JWT. we get a nifty oauth.Token in return.
    oauthToken, err := jsonWebToken.Assert(httpClient)
    if err != nil {
        log.Fatal("error asserting JSON Web Token -", err)
    }
    // build the oauth transport
    oauthTransport := oauth.Transport{Config: oauthConfig}
    // set the oauthTransport's token to be the oauthToken
    oauthTransport.Token = oauthToken
    // create a new analytics service by passing in the httpclient returned from Client() that can now make oauth-authenticated requests
    analyticsService, err := analytics.New(oauthTransport.Client())
    if err != nil {
        log.Fatal("error creating Analytics Service -", err)
    }
    // create a new analytics data service by passing in the analytics service we just created
    dataGaService := analytics.NewDataGaService(analyticsService)

    // w00t! now we're talking to the core reporting API. hard stuff over, let's do a simple query.

    // if no gaEndDate specified via command line args, set it to today's date.
    if gaEndDate == "" {
        t := time.Now()
        gaEndDate = t.Format(dateLayout)
    }
    // print the query start & end dates.
    fmt.Printf("gaStartDate=%s, gaEndDate=%s\n", gaStartDate, gaEndDate)
    fmt.Printf("gaMetrics=%s\n", gaMetrics)
    // setup the query
    dataGaGetCall := dataGaService.Get(gaTableID, gaStartDate, gaEndDate, gaMetrics)
    // setup the dimensions (if any).
    if gaDimensions != "" {
        dimensions := fmt.Sprintf("ga:dimensions=%s", gaDimensions)
        fmt.Printf("gaDimensions=%s\n", dimensions)
        dataGaGetCall.Dimensions(gaDimensions)
    }
    // setup the sort order (if any).
    if gaSortOrder != "" {
        sortorder := fmt.Sprintf("ga:sort=%s", gaSortOrder)
        fmt.Printf("gaSortOrder=%s\n", sortorder)
        dataGaGetCall.Sort(gaSortOrder)
    }
    // setup the filter (if any).
    if gaFilter != "" {
        filter := fmt.Sprintf("%s", gaFilter)
        fmt.Printf("gaFilter=%s\n", filter)
        dataGaGetCall.Filters(filter)
    }
    // setup the max results we want returned.
    dataGaGetCall.MaxResults(gaMaxResults)
    // send the query to the API, get gaData back.
    gaData, err := dataGaGetCall.Do()
    if err != nil {
        log.Fatal("API error -", err)
    }
    // how much data did we get back?
    fmt.Printf("len(gaData.Rows)=%d, TotalResults=%d\n", len(gaData.Rows), gaData.TotalResults)
    // a lazy loop through the returned rows
    for row := 0; row <= len(gaData.Rows)-1; row++ {
        fmt.Printf("row=%d %v\n", row, gaData.Rows[row])
    }
}

// func: usage()

// prints out all possible flags/options when "-h" or an unknown option is used at runtime.
// exits back to shell when complete.
func usage() {
    fmt.Printf("usage: %s [OPTION] \n", os.Args[0])
    flag.PrintDefaults()
    os.Exit(2)
}
Licensed under: CC-BY-SA with attribution
Not affiliated with StackOverflow
scroll top