Invalid byte sequence in UTF-8, CSV import, Rails 4

Question

I have a rake task that populates my database from a CSV file:

require 'csv'
namespace :import_data_csv do
desc "Import teams from csv file"
task import_data: :environment do

  CSV.foreach(file, :headers => true) do |row|
  #various import tasks

This had been working properly, but with a new CSV file, I'm getting the following error on the 6th row of the CSV file:

Invalid byte sequence in UTF-8

I have looked through the row and can't seem to find any irregular characters.

I've also attempted a couple other fixes recommended on stackoverflow: - Changing the CSV.foreach to:

reader = CSV.open(file, "r")
reader.each do |row|

And changing:

CSV.foreach(file, headers => true) do |row|

to:

CSV.foreach(file, encoding: "r:ISO-8859-1", :headers => true) do |row|

None of these seem to correct the issue.

Suggestions?

Answer 1

I solved this by saving the file as a MDOS CSV, instead of the standard CSV file or the Windows CSV format.

Answer 2

The answer for me was to take the CSV file and save it to a text file. Then replace the tabs with commas. Then save the file as UTF-8 encoded. Finally, change the .txt to .csv and make sure it works in Excel BUT DON'T save it in Excel. Just close it when you see it looks correct. Then upload it.

A long non-programatic solution, but for my purposes it's sufficient.

Source is here: https://help.salesforce.com/apex/HTViewSolution?id=000003837&language=en_US