Unexpected (case-insensitive) string sorting in Elasticsearch

Question

I have a list of console platforms that I'm sorting in Elasticsearch.

Here is the mapping for the "name" field:

{
    "name": {
        "type": "multi_field",
        "fields": {
            "name": {
                "type": "string",
                "index": "analyzed"
            },
            "sort_name": {
                "type": "string",
                "index": "not_analyzed"
            }
        }
    }
}

When I execute the following query

{
  "query": {
    "match_all": {}
  },
    "sort": [
        {
          "name.sort_name": { "order": "asc" }
        }
    ],
    "fields": ["name"]
}

I get these results:

{
    "took": 1,
    "timed_out": false,
    "_shards": {
        "total": 3,
        "successful": 3,
        "failed": 0
    },
    "hits": {
        "total": 17,
        "max_score": null,
        "hits": [
            {
                "_index": "platforms",
                "_type": "platform",
                "_id": "1393602489",
                "_score": null,
                "fields": {
                    "name": "GameCube"
                },
                "sort": [
                    "GameCube"
                ]
            },
            {
                "_index": "platforms",
                "_type": "platform",
                "_id": "1393602490",
                "_score": null,
                "fields": {
                    "name": "Gameboy Advance"
                },
                "sort": [
                    "Gameboy Advance"
                ]
            },


    {
            "_index": "platforms",
            "_type": "platform",
            "_id": "1393602498",
            "_score": null,
            "fields": {
                "name": "Nintendo 3DS"
            },
            "sort": [
                "Nintendo 3DS"
            ]
        },

        ...remove for brevity ...

        {
            "_index": "platforms",
            "_type": "platform",
            "_id": "1393602493",
            "_score": null,
            "fields": {
                "name": "Xbox 360"
            },
            "sort": [
                "Xbox 360"
            ]
        },
        {
            "_index": "platforms",
            "_type": "platform",
            "_id": "1393602502",
            "_score": null,
            "fields": {
                "name": "Xbox One"
            },
            "sort": [
                "Xbox One"
            ]
        },
        {
            "_index": "platforms",
            "_type": "platform",
            "_id": "1393602497",
            "_score": null,
            "fields": {
                "name": "iPhone/iPod"
            },
            "sort": [
                "iPhone/iPod"
            ]
        }
    ]
}

Everything is sorted as expected except the iPhone/iPod result is at the end (instead of after GameBoy Advance) - why does the / in the name have an effect on the sorting?

Thanks

Answer 1

Okay so I discovered the reason wasn't anything to do with the /. ES will sort by capital letters then lower case letters.

I added a custom analyzer to the settings of the index creation:

{
    "analysis": {
        "analyzer": {
            "sortable": {
                "tokenizer": "keyword",
                "filter": [
                    "lowercase"
                ]
            }
        }
    }
}

Then in the field mapping I added 'analyzer': 'sortable' to the sort_name multi field.

Answer 2

Use Normalizer with keyword to handle the sort

https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-normalizers.html#analysis-normalizers

PUT index_name
{
  "settings": {
    "analysis": {
      "normalizer": {
        "my_normalizer": {
          "type": "custom",
          "char_filter": ["quote"],
          "filter": ["lowercase", "asciifolding"]
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "name": {
        "type": "keyword",
        "normalizer": "my_normalizer"
      }
    }
  }
}

Search query may be modified like this

{
  "query": {
    "match_all": {}
  },
    "sort": [
        {
          "name.sort_name": { "order": "asc" }
        }
    ],
    "fields": "name.keyword"
}

Answer 3

According to https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-normalizers.html (ElasticSearch 7.16) ...

Elasticsearch ships with a lowercase built-in normalizer.

So you can define an additional field (in the example below named "lowersortable"):

PUT /myindex/_mapping
{
  "properties": {
    "myproperty": {
      "type": "text",
      "fields": {
        "lowersortable": {
          "type": "keyword",
          "normalizer": "lowercase"
        }
      }
    }
  }
}

... and use this field myproperty.lowersortable for sorting in the search query.