speed up data access in a search form

Question

In a spring mvc app using hibernate and JPA, I have a keyword class which enables users to search records in an underlying database. The JSP has a search box in a dandelion datatable, and the user types in text to reduce the number of keyword options, until the search results contain a short-enough list of keywords from which the user can select the word to use for their search.

The problem is that there are 80,000 keywords in the underlying data table, and it takes 20 minutes for my app to send all those keywords to the JSP, even after I put an index on the underlying database table for the field used in the search. Is there some way to set it up so that keywords are only sent to the JSP when at least 3 characters of text has been entered into the search box?

Here is my controller method:

@RequestMapping(value = "/search", method = RequestMethod.GET)
public String processFindForm(
        ConcWordKey wordkey, HttpServletRequest request, BindingResult result, Map<String, Object> model) {

    Collection<ConcWordKey> results = this.clinicService.findKeyWord("");
    System.out.println("results.size() is: "+results.size());
    model.put("selections", results);
    return "search";
}  

And here is the relevant part of the JSP:

<datatables:table id="keywords" data="${selections}" cdn="true" row="kw" theme="bootstrap2" 
      cssClass="table table-striped" paginate="true" info="false" 
      cssStyle="width: 150px;" align="left" dom="frtp">
    <datatables:column title="Name" cssStyle="width: 150px;" display="html">
        <spring:url value="/search?name={name}" var="searchUrl">
            <spring:param name="name" value="${kw.name}"/>
        </spring:url>
        <a href="${fn:escapeXml(searchUrl)}"><c:out value="${kw.name}"/></a>
    </datatables:column>
</datatables:table>  

Here is the JPA method that populates the selections:

@SuppressWarnings("unchecked")
public Collection<ConcWordKey> findKeyWord(String str) {
    Query query = this.em.createQuery("SELECT DISTINCT wk FROM ConcWordKey wk WHERE wk.name LIKE :wd");
    query.setParameter("wd", str + "%");
    Collection<ConcWordKey> results = query.getResultList();
    return results;
}

Answer 1

I'm afraid the poor client-side performance come from the method your are using for loading the data. I think 80000 records is too much for a DOM source, especially with the search fired on keypress.

Before the v0.10.0: I recommend you to switch to an AJAX source (with paging enabled). A bit more work is required to set things up but it should have a significant impact on performance. Note that the JSP sample app does use the same technologies (Spring, JPA) as yours. Feel free to take a look.

However, if you really want to keep loading data with a DOM source, you could simply unbind the keypress event handler, as suggested in this post. To implement it, you'll need the extra JavaScript feature of Dandelion-Datatables.

Starting from the v0.10.0: the filtering feature has been improved in multiple ways:

• a time delay (500ms by default, configurable) is always applied before firing the search
• a min character length can be also configured
• or you can fire the search on a click event instead

Of course, using an AJAX source is still recommended but the above features will improve the UX even more. I'll update this anwser with the right links to the new docs and sample apps once this version released.

Hope this helps!

(Disclaimer required by StackOverflow: I'm the author of Dandelion)

EDIT: Dandelion 0.10.0 released. Links added

Answer 2

EhCache may not be what you need first. It provides ability to handle your application cache, which means keep in memory the most used objects, to access them faster (from memory rather then querying them) when you need them. So if you have like 80 000 keywords, that sounds like you have 80 000+ search possibilities, which means that :

• It'll take a lot of memory to keep them all in the cache
• You must search all of them once to have them in the cache

What you may need, though, is an underlying index system, like Lucene (or Solar if you have multiple instances), which will help you speed up your queries. Then, your searches will be faster, no matter they have been fired before or not.

Of course, you can use the indexing system with a caching system to have even better performances. My point is indexing would be more important than caching at first.

Some other ideas :

• Be sure to enable your database cache (lvl1 cache) so that your queries will be faster
• Add a database index on your keyword column (also speeds up a lots the select queries)

Good luck !

EDIT : Reviewing the problem, an autocomplete functionality would be better. For example : http://johnderinger.wordpress.com/2013/01/10/html-autocomplete-with-jpa-rest-and-jquery/