How to configure Hibernate Search to find words with accents

Question

I need to implement a global search, on a website which I am implementing using Spring (4.0.2)/Hibernate(4.3.1)/MySQL. I have decided to use Hibernate Search(4.5.0) for this.

This seems to be working fine, but only when I do a search for an exact pattern.

Imagine I have the following text on an indexed field: "A história do Capuchinho e do Lobo Mau"

1) If I search for "história" or "lobo mau", the query will retrieve the corresponding indexed entity, as I would have expected.

2) If I search for "historia" or "lobos maus" the search will not retrieve the entity.

As far as I have read, it should be possible to configure Hibernate Search to perform a much smarter search than this. Can anyone point me on the right direction to achieve this? See below key aspects of the implementation I executed. Thanks!

This is the "parent" indexed entity

@Entity
@Table(name="NEWS_HEADER")
@Indexed
public class NewsHeader implements Serializable {

static final long serialVersionUID = 20140301L;

private int                 id;
private String              articleHeader;
private String              language;
private Set<NewsParagraph>  paragraphs = new HashSet<NewsParagraph>();

/**
 * @return the id
 */
@Id
@Column(name="ID")
@GeneratedValue(strategy=GenerationType.AUTO)
@DocumentId
public int getId() {
    return id;
}
/**
 * @param id the id to set
 */
public void setId(int id) {
    this.id = id;
}
/**
 * @return the articleHeader
 */
@Column(name="ARTICLE_HEADER")
@Field(index=Index.YES, analyze=Analyze.YES, store=Store.NO)
public String getArticleHeader() {
    return articleHeader;
}
/**
 * @param articleHeader the articleHeader to set
 */
public void setArticleHeader(String articleHeader) {
    this.articleHeader = articleHeader;
}
/**
 * @return the language
 */
@Column(name="LANGUAGE")
public String getLanguage() {
    return language;
}
/**
 * @param language the language to set
 */
public void setLanguage(String language) {
    this.language = language;
}
/**
 * @return the paragraphs
 */
@OneToMany(mappedBy="newsHeader", fetch=FetchType.EAGER, cascade=CascadeType.ALL)
@IndexedEmbedded
public Set<NewsParagraph> getParagraphs() {
    return paragraphs;
}
// Other standard getters/setters go here

And this the IndexedEmbedded entity

@Entity
@Table(name="NEWS_PARAGRAPH")
public class NewsParagraph implements Serializable {

static final long serialVersionUID = 20140302L;

private int         id;
private String      content;
private NewsHeader  newsHeader;

/**
 * @return the id
 */
@Id
@Column(name="ID")
@GeneratedValue(strategy=GenerationType.AUTO)
public int getId() {
    return id;
}
/**
 * @param id the id to set
 */
public void setId(int id) {
    this.id = id;
}
/**
 * @return the content
 */
@Column(name="CONTENT")
@Field(index=Index.YES, analyze=Analyze.YES, store=Store.NO)
public String getContent() {
    return content;
}
// Other standard getters/setters go here

This is my search method, implemented on my SearchDAOImpl

public class SearchDAOImpl extends DAOBasics implements SearchDAO {
    ...
    public List<NewsHeader> searchParagraph(String patternStr) {

    Session session = null;

    Transaction tx;

    List<NewsHeader> result = null;

    try {
        session = sessionFactory.getCurrentSession();
        FullTextSession fullTextSession = Search.getFullTextSession(session);
        tx = fullTextSession.beginTransaction();

        // Create native Lucene query using the query DSL
        QueryBuilder queryBuilder = fullTextSession.getSearchFactory()
            .buildQueryBuilder().forEntity(NewsHeader.class).get();

        org.apache.lucene.search.Query luceneSearchQuery = queryBuilder
            .keyword()
            .onFields("articleHeader", "paragraphs.content")
            .matching(patternStr)
            .createQuery();

        // Wrap Lucene query in a org.hibernate.Query
        org.hibernate.Query hibernateQuery = 
            fullTextSession.createFullTextQuery(luceneSearchQuery, NewsHeader.class, NewsParagraph.class);

        // Execute search
        result = hibernateQuery.list();

    } catch (Exception xcp) {
        logger.error(xcp);
    } finally {

        if ((session != null) && (session.isOpen())) {
            session.close();
        }
    }
    return result;
}
...
}

Answer 1

You could configure, or you can use a standard language analyzer, such as PortugueseAnalyzer. I'd recommend starting from the existing analyzer, and creating you own if necessary, using it as a starting point for tweaking the filter chain.

You can set this in using the @Analyzer annotation for the field:

@Field(index=Index.YES, analyze=Analyze.YES, store=Store.NO, analyzer = @Analyzer(impl = org.apache.lucene.analysis.pt.PortugueseAnalyzer.class))

Or you can set that analyzer as the default for the class, if you place an @analyzerannotation are the head of the class instead.

Answer 2

This is what I have ended up doing, to resolve my problem.

Configure an AnalyzerDef at the entity level. Within it, use LowerCaseFilterFactory, ASCIIFoldingFilterFactory and SnowballPorterFilterFactory to achieve the type of filtering I needed.

@AnalyzerDef(name = "customAnalyzer",
  tokenizer = @TokenizerDef(factory = StandardTokenizerFactory.class),
  filters = {
    @TokenFilterDef(factory = LowerCaseFilterFactory.class),
    @TokenFilterDef(factory = ASCIIFoldingFilterFactory.class),
    @TokenFilterDef(factory = SnowballPorterFilterFactory.class)
})
public class NewsHeader implements Serializable {
...
}

Add this notation for each of the fields I want indexed, either in the Parent entity or its IndexedEmbedded counterpart, to use the defined above analyzer.

@Field(index=Index.YES, store=Store.NO)
@Analyzer(definition = "customAnalyzer")

You will need to either re-index, or re-insert your entities, for the analyser to take effect.

Answer 3

If you want to search accent character and also find same normal keyword in result then you must have to implement ASCIIFoldingFilterFactory class in analyzer like

@AnalyzerDef(name = "customAnalyzer",
tokenizer = @TokenizerDef(factory = StandardTokenizerFactory.class),

filters = {
  @TokenFilterDef(factory = LowerCaseFilterFactory.class),
  @TokenFilterDef(factory = ASCIIFoldingFilterFactory.class),
  @TokenFilterDef(factory = StopFilterFactory.class, params = {
          @Parameter(name="words", value= "com/ik/resource/stoplist.properties" ),
          @Parameter(name="ignoreCase", value="true")
      })
})

@Analyzer(definition = "customAnalyzer") apply on entity or fields