How to remove non-printable/invisible characters in ruby?

Question

Sometimes I have evil non-printable characters in the middle of a string. These strings are user input, so I must make my program receive it well instead of try to change the source of the problem.

For example, they can have zero width no-break space in the middle of the string. For example, while parsing a .po file, one problematic part was the string "he is a man of god" in the middle of the file. While it everything seems correct, inspecting it with irb shows:

 "he is a man of god".codepoints
 => [104, 101, 32, 105, 115, 32, 97, 32, 65279, 109, 97, 110, 32, 111, 102, 32, 103, 111, 100] 

I believe that I know what a BOM is, and I even handle it nicely. However sometimes I have such characters on the middle of the file, so it is not a BOM.

My current approach is to remove all characters that I found evil in a really smelly fashion:

text = (text.codepoints - CODEPOINTS_BlACKLIST).pack("U*")

The most close I got was following this post which leaded me to :print: option on regexps. However it was no good for me:

"m".scan(/[[:print:]]/).join.codepoints
 => [65279, 109] 

so the question is: How can I remove all non-printable characters from a string in ruby?

Answer 1

try this:

>>"aaa\f\d\x00abcd".gsub(/[^[:print:]]/,'.')
=>"aaa.d.abcd"

Answer 2

Ruby can help you convert from one multi-byte character set to another. Check into the search results, plus read up on Ruby String's encode method.

Also, Ruby's Iconv is your friend.

Finally, James Grey wrote a series of articles which cover this in good detail.

One of the things you can do using those tools is to tell them to transcode to a visually similar character, or ignore them completely.

Dealing with alternate character sets is one of the most... irritating things I've ever had to do, because files can contain anything, but be marked as text. You might not expect it and then your code dies or starts throwing errors, because people are so ingenious when coming up with ways to insert alternate characters into content.

Answer 3

Codepoint 65279 is a zero-width no-break space. It is commonly used as a byte-order mark (BOM).

You can remove it from a string with:

my_new_string = my_old_string.gsub!("\xEF\xBB\xBF".force_encoding("UTF-8"), '')

A fast way to check if you have any invisible characters is to check the length of the string, if it's higher than what you can see in IRB, you do.