Best Vertica Data load Techniques?

Question

hy everybody !! -i am having some problems in seeing how does a data is loaded into Vertica !! I need to know if there are any Verticas dba's around here ! How do you guys do it ??

I mean i have other RDBMS as front end and Vertica runs in the back doing all the hard work etc.. Since transactional with vertica does not work that well !! (update speed is bad).

So please i need some help with this !! like a replication tool or if its done with scripts or by ETL(which tool is better)! THX

Answer 1

I mean i have other RDBMS as front end and Vertica runs in the back doing all the hard work etc.

I don't understand that this means about the relationship between data on the other systems and data on Vertica. What is your overall data strategy? Do you have monolithic files lying around that need to get block loaded? Your complaint about Vertica update speed suggests you might have lots of small updates?

If you have big files, best practice is actually to load using vsql itself, with the file sitting on one of the Vertica nodes. That is the fastest, anyways. You can write a loader that streams the big files over JDBC from a remote machine. This is how my company does it, and it's reasonably fast.

If you have lots of small rows that you want to send in frequently, Vertica is not as good with this. We also do this, but it is way slower.

Answer 2

I use Talend Open Studio (http://www.talend.com/)

they cooperates!

Answer 3

Try to use Vertica COPY command with Direct Option . It is for BULK loading .

Answer 4

For oracle, look at wisdowforce. For MySQL look at Tungsten replicator. Or you can use UDL/External tables. In the end, there are a ton of ways and much more detail would be needed to determine what is best for you.

Answer 5

Use ETL tool like Talend to sync data periodically from OLTP database to Vetica. This can handle multiple heterogeneous data source and complex transformation.

If you are using VoltDB (sister of Vertica for OLTP), add export only tables in VoltDB and sync data to Vertica.

Answer 6

If you are interested in setting up HP Vertica data inserts or updates using Talend (ETL tool) then I have a blog post you might be interested in: HP Vertica use in Talend