Solr: strip punctuation before index

Question

I am having a problem with striping punctuation from the solr index When the punctuation sign follow right after a word then this word is not indexed properly.

For example: if we index "hello, John", the asset won't be found by keyword "hello" while there will be no issue if we remove comma after word "hello".

Is there any FilterFactory that suppose to strip punctuation? Any ideas?

Thanks, Bogdan.

Answer 1

You can use the solr.PatternReplaceFilterFactory to strip beginning and trailing punctuation with this:

<filter class="solr.PatternReplaceFilterFactory"
    pattern="^\p{Punct}*(.*?)\p{Punct}*$"
    replacement="$1"/>

And if you wanted to strip all punctuation at the beginning and end, except (for example) the dollar-sign in front of a word, you could use this:

<filter class="solr.PatternReplaceFilterFactory"
    pattern="^[\p{Punct}&&[^$]]*(.*?)\p{Punct}*$"
    replacement="$1"/>

Answer 2

This is done with the WordDelimiterFilterFactory. Set generateWordParts=1.

There is also the PatternTokenizerFactory that could be used, but I have never tried it.

Answer 3

Use PatternReplaceFilterFactory

<!-- remove punctuation -->
    <filter class="solr.PatternReplaceFilterFactory" pattern="^(\p{Punct}*)(.*?)(\p{Punct}*)$" replacement="$2"/>
    <filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.StandardFilterFactory"/>
    <filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
  </analyzer>

...