Google BigQuery PHP Load Data from URL

Question

I am connecting to an API, and getting a report in a TSV format. I am needing to upload the report to Google BigQuery, but all the documentation I have found so far loads data from Google Cloud Storage. Is there a way to load data from a seperate URL?

Here is the code I have thus far:

$service = new Google_BigqueryService($client);
// Your project number, from the developers.google.com/console project you created
// when signing up for BigQuery
$project_number = '*******';

// Information about the destination table
$destination_table = new Google_TableReference();
$destination_table->setProjectId($project_number);
$destination_table->setDatasetId('php_test');
$destination_table->setTableId('my_new_table');

// Information about the schema for your new table
$schema_fields = array();
$schema_fields[0] = new Google_TableFieldSchema();
$schema_fields[0]->setName('Date');
$schema_fields[0]->setType('string');

$schema_fields[1] = new Google_TableFieldSchema();
$schema_fields[1]->setName('PartnerId');
$schema_fields[1]->setType('string');

....

$destination_table_schema = new Google_TableSchema();
$destination_table_schema->setFields($schema_fields);

// Set the load configuration, including source file(s) and schema
$load_configuration = new Google_JobConfigurationLoad();
$load_configuration->setSourceUris(array('EXTERNAL URL WITH TSV'));
$load_configuration->setDestinationTable($destination_table);
$load_configuration->setSchema($destination_table_schema);


$job_configuration = new Google_JobConfiguration();
$job_configuration->setLoad($load_configuration);

$load_job = new Google_Job();
$load_job->setKind('load');        
$load_job->setConfiguration($job_configuration);

$jobs = $service->jobs;
$response = $jobs->insert($project_number, $load_job);

I realize that this is meant for Google Cloud Storage, but I do not want to use it, if I am just going to pass data through it and delete it within the hour.

Is there PHP code that I can use that will allow me to use external URLs and load data from them?

Answer 1

As Pentiuum10 mentioned above, BigQuery doesn't support reading from non-Google Cloud Storage URLs. The logistics involved would be tricky ... we'd need credentials to access the data, which we don't really want to have to be responsible for. If this is a popular request, we might end up supporting external paths that are either unrestricted or support oauth2. That said, we haven't had a lot of users asking for this so far.

Feel free to file a feature request via the public issue tracker here: https://code.google.com/p/google-bigquery/issues/.