Profanity filter with fuzzy search in ruby rails

Question

I am a rails newbie.

I am using profanity_filter ruby gem to filter the foul words in my content application..

profanity_filter, if at all there is a foul word, lets say "foulword" it returns "f******d"

If any user plays smart and types "foulwoord" or "foulwordd" or "foulllword" etc it does not detect as a foul word.

Is there a way to make sure it detects these user-smart-foul-words?

Looking forward for help!

Thank you!

Answer 1

How many foul words do you need to filter?

One approach would be to use something like Diff::LCS (from the diff-lcs gem) to check how many letters are different between the word being checked and each foul word. If you have a large number of foul words to check, this could be very slow. One thing you could do to make it much faster would be to include a dictionary of "good" words. Keep the "good" dictionary in a Set, and before checking each content word, first test whether it is in the dictionary. If so, you can move on. (If you want to make checking the dictionary very fast, keep it in a search trie.)

Further, if you check a word and find that it is OK, you could add it to the dictionary so you don't need to check the same word again. The danger here is that the dictionary may grow too large. If this is a problem, you could use something similar to a "least recently used" cache which, when the dictionary becomes too big, would discard "good" words which have not been seen recently.

Another approach would be to generate variants on each foul word, and store them in a "bad" dictionary. If you generate each word which differs by 1 letter from a foul word, there would be about 200-500 for each foul word. You could also generate words which differ from a foul word only by changing the letter "o" to a zero, etc.

No matter what you do, you are never going to catch 100% of "bad" words without ever mistakenly flagging a "good" word. If you can get a filter which catches an acceptably high percentage of "bad" words, with an acceptably low rate of false positives, that will be "success".

If you are doing this for a web site, I suggest that rather than blocking content with "bad" words, you automatically flag it for moderator attention. If allowing obscene content to go up on the site even briefly is unacceptable, you could delay displaying flagged content until after a moderator has looked at it. This will avoid the Scunthorpe problem with @Blorgbeard mentioned in his comment.