Read compressed csv file on-the-fly

Question

I have wrote some csv file and compress it, using this code:

arr = (0...2**16).to_a
File.open('file.bz2', 'wb') do |f|
  writer = Bzip2::Writer.new f
  CSV(writer) do |csv|
    (2**16).times { csv << arr }
  end
  writer.close
end

I want to read this csv bzip2ed file (csv files compressed with bzip2). These files uncompressed look like:

1,2
4,12
5,2
8,7
1,3
...

So I tried this code:

Bzip2::Reader.open(filename) do |bzip2|
  CSV.foreach(bzip2) do |row|
    puts row.inspect
  end
end

but when it is executed, it throws:

/Users/foo/.rvm/rubies/ruby-2.1.0/lib/ruby/2.1.0/csv.rb:1256:in `initialize': no implicit conversion of Bzip2::Reader into String (TypeError)
from /Users/foo/.rvm/rubies/ruby-2.1.0/lib/ruby/2.1.0/csv.rb:1256:in `open'
from /Users/foo/.rvm/rubies/ruby-2.1.0/lib/ruby/2.1.0/csv.rb:1256:in `open'
from /Users/foo/.rvm/rubies/ruby-2.1.0/lib/ruby/2.1.0/csv.rb:1121:in `foreach'
from worm_pathfinder_solver.rb:79:in `block in <main>'
from worm_pathfinder_solver.rb:77:in `open'
from worm_pathfinder_solver.rb:77:in `<main>'

Question:

What is wrong? How should I do?

Answer 1

Based on the brief docs you'll probably need send the read method on bzip2 object (not tested):

Bzip2::Reader.open(filename) do |bzip2|
  CSV.foreach(bzip2.read) do |row|
    #               ^^^^
    puts row.inspect
  end
end

Answer 2

CSV.foreach assumes you're passing a file path to open. If you want to pass a stream to CSV you need to be more explicit and use CSV.new. This code will process a gzipped file:

Zlib::GzipReader.open(filename) do |gzip|
  csv = CSV.new(gzip)
  csv.each do |row|
    puts row.inspect
  end
end

Answer 3

My guess would be that CSV tries to convert the Bzip2::Reader to a string but doesn't know how and simply throws the exception. You can manually read the data into a string and then pass THAT to CSV.

Though it's strange since it could handle Bzip2::Writer just fine.