User expectations and unicode normalization

Question

This is a bit of a soft question, feel free to let me know if there's a better place for this.

I'm developing some code that accepts a password that requires international characters - so I'll need to compare an input unicode string with a stored unicode string. Easy enough.

My question is this - do users of international character sets generally expect normalization in such a case? My Google searches show some conflicts in opinion from 'always do it' (http://unicode.org/faq/normalization.html) to 'don't bother'. Are there any pros/cons to not normalizing? (i.e., less likely to able guess a password, etc.)

Answer 1

I would recommend that if your password field accepts Unicode input (presumably UTF-8 or UTF-16), that you normalize it before hashing and comparing. If you don't normalize it, and people access it from different systems (different operating systems, or different browsers if it's a web app, or with different locales), then you may get the same password represented with different normalization. This would mean that your user would type the correct password, but have it rejected, and it would not be obvious why, nor would they have any way to fix it.

Answer 2

I wouldn't bother for a couple reasons:

1. You're going to make things less secure. If two or more characters are all represented in your DB as the same thing, then that means there are fewer possible passwords for the site. (Though this probably isn't a huge deal, since the number of possible passwords is pretty huge.)
2. You will be building code into your program that does complicated work that is (probably) part of a library you didn't write...and eventually somebody won't be able to log in as a result. Better in my mind to keep things simple, and to trust that people using different character sets know how to type them properly. That said, I've never implemented this in an international password form, so I couldn't tell you what the standard design pattern is.