Creating data parser in Ruby

Question

I need to create a data parser that will read in a couple CSVs (predefined format), parse, check for validity, and then, if all goes according to plan, commit the data to the database. Finally, report any errors that the validity checker didn't catch that were ultimately fatal. It would be extremely helpful if, on most errors, it could still commit the error free rows and at least report the rows that were not committed.

Does anyone know of a gem, any other resource, or any advice that I could use to get me started?

This is going to be part of a larger Padrino app, and each CSV will be POSTed to a route that is meant for this type of processing.

Thank you for your help!

Answer 1

You can use the csv library and just parse the data. If something goes wrong rescue. Other ways of checking sanity depend on your data and if you can predict valid from invalid data. That's the route I'd go.

If you read via the csv library you can convert data directly to objects and then use your faviourite orm to save the data to a db.

Answer 2

For parsing CSV, you can use FasterCSV to handle parsing and catching errors.

For importing the data, use an existing ORM (ActiveRecord, Sequel, DataMapper) to handle database interaction logic in your controller.

You may also find these interesting:

1. http://www.ruby-forum.com/topic/178753
2. http://blog.pioneeringsoftware.co.uk/2010/07/13/import-csv-files-in-rails-3