String search with Turkish dotless i

Question

When searching the text Çınaraltı Café for the text Ci using the code

NSStringCompareOptions options =
    NSCaseInsensitiveSearch |
    NSDiacriticInsensitiveSearch |
    NSWidthInsensitiveSearch;
NSLocale *locale = [NSLocale localeWithLocaleIdentifier:@"tr"];
NSRange range = [haystack rangeOfString:needle 
                                options:options
                                  range:NSMakeRange(o, haystack.length)
                                 locale:locale];

I get range.location equals NSNotFound.

It's not to do with the diacritic on the initial Ç because I get the same result searching for alti where the only odd character is the ı. I also get a valid match searching for Cafe which contains a diacritic (the é).

The apple docs mention this situation as notes on the locale parameter and I think I'm following them. Though I guess I'm not because it's not working.

How can I get a search for 'i' to match both 'i' and 'ı'?

Answer 1

I did this and seems to work well for me.. hope it helps!

NSString *cleanedHaystack = [haystack stringByReplacingOccurrencesOfString:@"ı"
                                                                withString:@"i"];
cleanedHaystack = [cleanedHaystack stringByReplacingOccurrencesOfString:@"İ"
                                                             withString:@"I"];

NSString *cleanedNeedle = [needle stringByReplacingOccurrencesOfString:@"ı"
                                                            withString:@"i"];
cleanedNeedle = [cleanedNeedle stringByReplacingOccurrencesOfString:@"İ"
                                                         withString:@"I"];

NSUInteger options = (NSDiacriticInsensitiveSearch |
                      NSCaseInsensitiveSearch |
                      NSWidthInsensitiveSearch);
NSRange range = [cleanedHaystack rangeOfString:cleanedNeedle
                                       options:options];

Answer 2

I don't know whether this helps as an answer, but perhaps explains why it's happening.

I should point out I'm not an expert in this matter, but I've been looking into this for my own purposes and been doing some research.

Looking at the Unicode collation chart for latin, the equivalent characters to ASCII "i" (\u0069) do not include "ı" (\u0131), whereas all the other letters in your example string are as you expect, i.e.:

• "c" (\u0063) does include "Ç" (\u00c7)
• "e" (\u0065) does include "é" (\u00e9)

The ı character is listed separately as being of primary difference to i. That might not make sense to a Turkish speaker (I'm not one) but it's what Unicode have to say about it, and it does fit the logic of the problem you describe.

In Chrome you can see this in action with an in-page search. Searching in the page for ASCII i highlights all the characters in its block and does not match ı. Searching for ı does the opposite.

By contrast, MySQL's utf8_general_ci collation table maps uppercase ASCII I to ı as you want.

So, without knowing anything about iOS, I'm assuming it's using the Unicode standard and normalising all characters to latin by this table.

As to how you match Çınaraltı with Ci - if you can't override the collation table then perhaps you can just replace i in your search strings with a regular expression, so you search on Ç[iı] instead.

Answer 3

I wrote a simple extension in Swift 3 for Turkish string search.

let turkishSentence = "Türkçe ya da Türk dili, batıda Balkanlar’dan başlayıp doğuda Hazar Denizi sahasına kadar konuşulan Altay dillerinden biridir."
let turkishWannabe = "basLayip"

let shouldBeTrue = turkishSentence.contains(turkishString: turkishWannabe, caseSensitive: false)
let shouldBeFalse = turkishSentence.contains(turkishString: turkishWannabe, caseSensitive: true)

You can check it out from https://github.com/alpkeser/swift_turkish_string_search/blob/master/TurkishTextSearch.playground/Contents.swift

Answer 4

As Tim mentions, we can use regular expression to match text containing i or ı. I also didn't want to add a new field or change the source data as the search looks up huge amounts of string. So I ended up a solution using regular expressions and NSPredicate.

Create NSString category and copy this method. It returns basic or matching pattern. You can use it with any method that accepts regular expression pattern.

- (NSString *)zst_regexForTurkishLettersWithCaseSensitive:(BOOL)caseSensitive
{
    NSMutableString *filterWordRegex = [NSMutableString string];
    for (NSUInteger i = 0; i < self.length; i++) {
        NSString *letter = [self substringWithRange:NSMakeRange(i, 1)];
        if (caseSensitive) {
            if ([letter isEqualToString:@"ı"] || [letter isEqualToString:@"i"]) {
                letter = @"[ıi]";
            } else if ([letter isEqualToString:@"I"] || [letter isEqualToString:@"İ"]) {
                letter = @"[Iİ]";
            }
        } else {
            if ([letter isEqualToString:@"ı"] || [letter isEqualToString:@"i"] ||
                [letter isEqualToString:@"I"] || [letter isEqualToString:@"İ"]) {
                letter = @"[ıiIİ]";
            }
        }
        [filterWordRegex appendString:letter];
    }
    return filterWordRegex;
}

So if the search word is Şırnak, it creates Ş[ıi]rnak for case sensitive and Ş[ıiIİ]rnak for case insensitive search.

And here are the possible usages.

NSString *testString = @"Şırnak";

// First create your search regular expression.
NSString *searchWord = @"şır";
NSString *searchPattern = [searchWord zst_regexForTurkishLettersWithCaseSensitive:NO];

// Then create your matching pattern.
NSString *pattern = searchPattern; // Direct match
// NSString *pattern = [NSString stringWithFormat:@".*%@.*", searchPattern]; // Contains
// NSString *pattern = [NSString stringWithFormat:@"\\b%@.*", searchPattern]; // Begins with

// NSPredicate
// c for case insensitive, d for diacritic insensitive
NSPredicate *predicate = [NSPredicate predicateWithFormat:@"self matches[cd] %@", pattern]; 
if ([predicate evaluateWithObject:testString]) {
    // Matches
}

// If you want to filter an array of objects
NSArray *matchedCities = [allAirports filteredArrayUsingPredicate:
    [NSPredicate predicateWithFormat:@"city matches[cd] %@", pattern]];

You can also use NSRegularExpression but I think using case and diacritic insensitive search with NSPredicate is much more simpler.