Importing csv into multiple mysql databases from rails app

Question

I have a CSV file consisting of 78,000 records.Im using smarter_csv (https://github.com/tilo/smarter_csv) to parse the csv file. I want to import that into a MySQL database from my rails app. I have the following two questions

1. What would be the best approach to quickly importing such a large data-set into MySQL from my rails app ?. Would using resque or sidekiq to create multiple workers be a good idea ?

2. I need to insert this data into a given table which is present in multiple databases. In Rails, i have a model talk to only one database. So how can i scale the solution to talk to multiple mysql databases from my model ?

Thank You

Answer 1

One way would be to use the native interface of the database application itself for importing and exporting; it would be optimised for that specific purpose.

For MySQL, the mysqlimport provides that interface. Note that the import can also be done as an SQL statement and that this executable provides a much saner interface for the underlying SQL command.

As far as implementation goes, if this is a frequent import exercise, the sidekiq/resque/cron job is the best possible approach.

[EDIT]

The SQL command referred to above is the LOAD DATA INFILE as the other answer points out.

Answer 2

Performance wise probably the best method is the use MYSQL's LOAD DATA INFILE syntax and execute an import command on each database. This requires the data file to be local to each database instance.

As the other answer suggests, mysqlimport can be used to ease the import as the LOAD DATA INFILE statement syntax is highly customisable and can deal with many data formats.