Getting 401 uploading file into a table with a service account

Question

I am using nodejs and the REST API to interact with bigquery. I am using the google-oauth-jwt module for JWT signing.

I granted a service account write permission. So far I can list projects, list datasets, create a table and delete a table. But when it comes to upload a file via multipart POST, I ran into two problems:

• gzipped json file doesn't work, I get an error saying "end boundary missing"
• when I use uncompressed json file, I get a 401 unauthorized error

I don't think this is related to my machine's time being out of sync since other REST api calls worked as expected.

var url = 'https://www.googleapis.com/upload/bigquery/v2/projects/' + projectId + '/jobs';
    var request = googleOauthJWT.requestWithJWT();
    var jobResource = {
        jobReference: {
            projectId: projectId,
            jobId: jobId
        },
        configuration: {
            load: {
                sourceFormat: 'NEWLINE_DELIMITED_JSON',
                destinationTable: {
                    projectId: projectId,
                    datasetId: datasetId,
                    tableId: tableId
                },
                createDisposition: '',
                writeDisposition: ''
            }
        }
    };
    request(
            {
                url: url,
                method: 'POST',
                jwt: jwtParams,
                headers: {
                    'Content-Type': 'multipart/related'
                },
                qs: {
                    uploadType: 'multipart'
                },
                multipart: [
                    {
                        'Content-Type':'application/json; charset=UTF-8',
                        body: JSON.stringify(jobResource)
                    },
                    {
                        'Content-Type':'application/octet-stream',
                        body: fileBuffer.toString()   
                    }
                ]
            },
            function(err, response, body) {
                console.log(JSON.parse(body).selfLink);
            }
        );

Can anyone shine some light on this?

P.S. the documentation on bigquery REST api is not up to date on many things, wish the google guys can keep it updated

Update 1:

Here is the full HTTP request:

POST /upload/bigquery/v2/projects/239525534299/jobs?uploadType=multipart HTTP/1.1
content-type: multipart/related; boundary=71e00bd1-1c17-4892-8784-2facc6998699
authorization: Bearer ya29.AHES6ZRYyfSUpQz7xt-xwEgUfelmCvwi0RL3ztHDwC4vnBI
host: www.googleapis.com
content-length: 876
Connection: keep-alive

--71e00bd1-1c17-4892-8784-2facc6998699
Content-Type: application/json

{"jobReference":{"projectId":"239525534299","jobId":"test-upload-2013-08-07_2300"},"configuration":{"load":{"sourceFormat":"NEWLINE_DELIMITED_JSON","destinationTable":{"projectId":"239525534299","datasetId":"performance","tableId":"test_table"},"createDisposition":"CREATE_NEVER","writeDisposition":"WRITE_APPEND"}}}
--71e00bd1-1c17-4892-8784-2facc6998699
Content-Type: application/octet-stream

{"practiceId":2,"fanCount":5,"mvp":"Hello"}
{"practiceId":3,"fanCount":33,"mvp":"Hello"}
{"practiceId":4,"fanCount":71,"mvp":"Hello"}
{"practiceId":5,"fanCount":93,"mvp":"Hello"}
{"practiceId":6,"fanCount":92,"mvp":"Hello"}
{"practiceId":7,"fanCount":74,"mvp":"Hello"}
{"practiceId":8,"fanCount":100,"mvp":"Hello"}
{"practiceId":9,"fanCount":27,"mvp":"Hello"}

--71e00bd1-1c17-4892-8784-2facc6998699--

Answer 1

I figured this out myself. This is one of those silly mistakes that would have you stuck for the whole day and at the end when you found the solution you would really knock on your own head.

I got the 401 by typing the selfLink URL in the browser. Of course it's not authorized.

Answer 2

You are most likely sending duplicate content-type headers to the Google API.

I don't have the capability to effortlessly make a request to Google BigQuery to test, but I'd start with removing the headers property of your options object to request().

Remove this:

headers: {
  'Content-Type': 'multipart/related'
},

The Node.js request module automatically detects that you have passed in a multipart array, and it adds the appropriate content-type header. If you provide your own content-type header, you most likely end up with a "duplicate" one, which does not contain the multipart boundary.

If you modify your code slightly to print out the actual headers sent:

var req = request({...}, function(..) {...});

console.log(req.headers);

You should see something like this for your original code above (I'm using the Node REPL):

> req.headers
{ 'Content-Type': 'multipart/related',
  'content-type': 'multipart/related; boundary=af5ed508-5655-48e4-b43c-ae5be91b5ae9',
  'content-length': 271 }

And the following if you remove the explicit headers option:

> req.headers
{ 'content-type': 'multipart/related; boundary=49d2371f-1baf-4526-b140-0d4d3f80bb75',
  'content-length': 271 }

Some servers don't deal well with multiple headers having the same name. Hopefully this solves the end boundary missing error from the API!