Importing csv data into OpenTSDB

Question

I have successfully installed OpenTSDB ontop of a Cloudera Hadoop/HBase cluster.

My question is, I have reams of historical 1 minute stock data that looks like this:

"Date","Time","Open","High","Low","Close","Volume" 
12/30/2002,0930,24.53,24.65,24.53,24.65,762200
12/30/2002,0931,24.65,24.68,24.52,24.6,90400

.....

From the documentation in the QS Guide, it says in the Batch imports section:

./tsdb import your-file

When I try this on my data it throws an unhelpful exception.

Any hints on how to import this into OpenTSDB? Thanks.

Answer 1

You need to write a script to transform your CSV in something that is in the OpenTSDB format. The general format for OpenTSDB is metric timestamp value tags

For instance your sample could be written as follows:

stock.open 1041269400 24.53 symbol=XXX
stock.high 1041269400 24.65 symbol=XXX
stock.low 1041269400 24.53 symbol=XXX
stock.close 1041269400 24.65 symbol=XXX
stock.volume 1041269400 762200 symbol=XXX
stock.open 1041269460 24.65 symbol=XXX
stock.high 1041269460 24.68 symbol=XXX
stock.low 1041269460 24.52 symbol=XXX
stock.close 1041269460 24.6 symbol=XXX
stock.volume 1041269460 90400 symbol=XXX

Although since it appears that you're working with 1-minute periods, the open/close is redundant, so maybe this would be more appropriate:

stock.quote.1m 1041269340 24.53 symbol=XXX
stock.quote.1m 1041269400 24.65 symbol=XXX
stock.quote.1m 1041269460 24.6 symbol=XXX

Answer 2

I wrote a small csv importer for opentsdb.

https://github.com/soeren-lubitz/csv-to-opentsdb

It works for CSV-Files in form of

Timestamp,Foo,Bar
1483342774,42.1,23.2

Hope this helps. Feedback would be appreciated.