Stream to a file in ruby when exporting a large database table to a csv file

Question

I export the following from a mysql database using a little ruby script and the mysql2 gem

require 'csv'
require 'mysql2'

begin
    db = Mysql2::Client.new( :host => ".", :username => "xyz", :password => 'xyz', :database => 'xyz')

    results = db.query 'select * from mytable'

    CSV.open("C:/tmp/mytable.csv", "w") do |csv|
        csv << results.fields

        results.each do |row|
            csv << row.values
        end
    end
end

This works fine for small tables but I think I should be doing things differently for big tables. I could be wrong here but it seems to read the entire table out to memory and then write it all in one go. If i've got a huge table with tens of millions of rows I'd like to stream to the file constantly.

How would I go about doing that?

I tried running this against a table with ~10 million rows and got the following

[FATAL] failed to allocate memory

---

I've found this, will report back if it works

result = client.query("SELECT * FROM really_big_Table", :stream => true)

Answer 1

The answer was indeed this, the problem was reading the database table

results = db.query( 'select * from mytable',  :stream => true, :cache_rows => false )

The writing of the csv was done asynchronously

Answer 2

This isn't perhaps the Ruby solution you are looking for, but it is one solution:

MySQL supports direct output to a CSV file. In your case if all you need is a CSV dump, then the most efficient way is probably to run a query like this:

SELECT * FROM mytable
  INTO OUTFILE 'C:\\tmp\\mytable.csv'
  FIELDS TERMINATED BY ','
  ENCLOSED BY '"'
  LINES TERMINATED BY '\r\n'

This works if the MySQL server is on the same computer as the output file, since it will be the server that writes the file, not the client.

http://www.tech-recipes.com/rx/1475/save-mysql-query-results-into-a-text-or-csv-file/