Which form of unicode normalization is appropriate for text mining?

Question

I've been reading a lot on the subject of Unicode, but I remain very confused about normalization and its different forms. In short, I am working on a project that involves extracting text from PDF files and performing some semantic text analysis.

I've managed to satisfactorily extract the text using a simple python script, but now I need to make sure that all equivalent orthographic strings have one (and only one) representation. For example, the 'fi' typographic ligature should be decomposed into 'f' and 'i'.

I see that python's unicodedata.normalize function offers several algorithms for normalizing unicode code points. Could someone please explain the difference between:

• NFC
• NFKC
• NFD
• NFKD

I read the relevant wikipedia article, but it was far too opaque for my feeble brain to understand. Could someone kindly explain this to me in plain English?

Also, could you please make a recommendation for the normalization method best adapted to a natural language processing project?

Answer 1

Characters like é can be written either as a single character or as a sequence of two, a regular e plus the accent (a diacritic). Normalization chooses consistently among such alternatives, and will order multiple diacritics in a consistent way.

Since you need to deal with ligatures, you should use "compatibility (de)composition", NFKD or NFKC, which normalizes ligatures. It's probably ok to either use composed or decomposed forms, but if you also want to do lossy matching (e.g., match é even if the user types plain e), you could use the compatibility decomposition NFKD and discard the diacritics for loose matching.