bq query without verbose option

StackOverflow https://stackoverflow.com/questions/22410119

  •  14-06-2023
  •  | 
  •  

Question

The idea is to use the bq tool to feed an R session with data stored in the BigQuery system in csv format. The output of the bq tool shows the data plus unwanted messages such as the job status and warning messages of an update. Is there any flag so the bq tool will only return the data of the query in csv without any other messages? 

> cmdbq <- "bq --format=csv query \"SELECT id, SEC_TO_TIMESTAMP(timestamp) AS time FROM publicdata:samples.wikipedia LIMIT 5\""
> r1 <- shell(cmdbq,intern=TRUE)
> r1
 [1] "\rWaiting on bqjob_r1b11c1f4_00000144c13cea58_1 ... (0s) Current status: DONE   " "id,time"                                                                         
 [3] "18524,2009-08-03 09:13:51"                                                        "2252318,2008-11-24 00:41:42"                                                     
 [5] "3430772,2007-01-11 05:43:19"                                                      "11170233,2007-05-11 12:13:36"                                                    
 [7] "1008805,2007-06-19 20:34:13"                                                      ""                                                                                
 [9] "There are available updates for some Cloud SDK components.  To "                  "install them, please run:"                                                       
[11] " $ gcloud components update"                                                      ""                                                                                
> r2 <- read.csv(text=paste(r1[2:7],collapse="\n"),stringsAsFactors=FALSE)
> r2
        id                time
1    18524 2009-08-03 09:13:51
2  2252318 2008-11-24 00:41:42
3  3430772 2007-01-11 05:43:19
4 11170233 2007-05-11 12:13:36
5  1008805 2007-06-19 20:34:13
>

Note: I am aware about existance of the bigrquery package but I want to use this approach because this part of the code is going to be running in a foreach parallelized loop and it simplifies the authentication step for each node.

Was it helpful?

Solution

The message about needing to update is printed to stderr. This is done so that if you capture stdout only, it will not interfere with scraping the output.

Alternatively, you can permanently disable this message by running

$ gcloud config set --section component_manager disable_update_check true

although then you will need to manually check for updates in the future.

OTHER TIPS

If you use bq -q it will suppress the job status messages. The gcloud message, unfortunately, is added in a wrapper and not bq itself, so I don't know of a way to disable it, other than to run gcloud components update

Licensed under: CC-BY-SA with attribution
Not affiliated with StackOverflow
scroll top