How do I encode files to UTF-8 for Rails 3?
-
15-07-2021 - |
Question
I've been working on outlook imports (linked in exports to outlook format) but I'm having troubles with encoding. The outlook format CSV I get from exporting my LinkedIn contacts are not in UTF-8. Letters like ñ
cause an exception in the mongoid_search
gem when calling str.to_s.mb_chars.normalize
. I think encoding is the issue, because when I call mb_chars
(see first code example). I am not sure if this is a bug in the gem, but I was advised to sanitize the data nonetheless.
From File Picker, I tried using their new, community-supported gem to upload CSV data. I tried three encoding detectors and transcoders:
- Ruby port of a Python lib
chardet
- Didn't work as expected
- The port still contained Python code, preventing it from running in my app
rchardet19
gem- Detected
iso-8859
with.8/1
confidence. - Tried to transcode with Iconv, but crashed on "illegal characters" at
ñ
- Detected
Charlock_Holmes
gem- Detected
windows-1252
with33/100
confidence - I assume that's the actual encoding, and
rchardet
gotiso-8859
because this ones based of that. - This gem uses ICU and has a maintained branch "bundle-icu" which supports Heroku. When I try to transcode using
charlock
, I get the errorU_FILE_ACCESS_ERROR
, an ICU error code meaning "could not open file"
- Detected
Anybody know what to do here?
Solution
Ruby 1.9 has encoding built in, have you tried:
s.force_encoding 'utf-8'
mb_chars is a wrapper for ruby 1.8, so you shouldn't need it.
See duplicate