TB of data need to uploaded to Bigquery

StackOverflow https://stackoverflow.com/questions/23117543

  •  04-07-2023
  •  | 
  •  

Question

We have TB of data need to uploaded to bigquery. I remember one of the video from Felipe Hoffa mentioning that we can send a hard drive overnight to Google and they can take care of it. Can Google Bigquery team provide more info on it is possible?

Was it helpful?

Solution

This is the Offline Import mechanism from Google Cloud Storage. You can read about it here: https://developers.google.com/storage/docs/early-access

Basically, you'd use this mechanism to import to Google Cloud Storage, then run BigQuery import jobs to import to BigQuery from there.

Depending on how many TB of data you are importing, you might just be better off uploading directly to Google Cloud Storage. Gsutil and other tools can do resumable uploads.

If you are talking about 100s of TB or more, you might want to talk to a Google Cloud Support person about your scenarios in detail. They may be able to help you optimize your usage of BigQuery and Cloud Storage.

Licensed under: CC-BY-SA with attribution
Not affiliated with StackOverflow
scroll top