Importing CouchBase dump to BigQuery
-
07-10-2020 - |
Domanda
After creating a CouchBase dump and copying it to a Google Storage bucket I tried importing the data to BigQuery.
I failed miserably, apparently the design.json
or the files that appear in the same directory as data-0000.cbb
are not working when I'm trying the json
import.
Do I need to convert the dump somehow? Is there any other way to migrate the data from CB to BQ?
Thank you all in advance.
Soluzione
I managed to export the couchbase data to CSV, after converting to CSV I discovered the files were corrupted and there were some entries I had to give up on.
When I first tried there was no "Ignore Errors" option in BigQuery so I couldn't import the files at all, after several weeks I tried again and found the "Ignore Errors" option, I immediately imported my files, first each one of them to a single DB, later on combined them together, it looks great now but it was not tested for compliance yet, I lost several records (a little less than 20) out of 80 gigs of data.