Server-side Pagination: total row count for expensive query?

Question

I have a simple query using server-side pagination. The issue is the WHERE Clause makes a call to an expensive function and the functions argument is the user input, eg. what the user is searching for.

SELECT 
  * 
FROM 
  ( SELECT /*+ FIRST_ROWS(numberOfRows) */ 
      query.*, 
      ROWNUM rn FROM 
        (SELECT            
            myColumns 
          FROM 
            myTable 
            WHERE expensiveFunction(:userInput)=1  
          ORDER BY id ASC
        ) query
    )
WHERE rn >= :startIndex 
AND ROWNUM <= :numberOfRows

This works and is quick assuming numberOfRows is small. However I would also like to have the total row count of the query. Depending on the user input and database size the query can take up to minutes. My current approach is to cache this value but that still means the user needs to wait minutes to see first result.

The results should be displayed in the Jquery datatables plugin which greatly helps with things like serer-side paging. It however requires the server to return a value for the total records to correctly display paging controls.

What would be the best approach? (Note: PHP)

I thought if returning first page immediately with a fake (better would be estimated) row count. After the page is loaded do an ajax call to a method that determines total row count of the query (what happens if the user pages during that time?) and then update the faked/estimated total row count.

However I have no clue how to do an estimate. I tried count(*) * 1000 with SAMPLE (0.1) but for whatever reason that actually takes longer than the full count query. Also just returning a fake/random value seems a bit hacky too. It would need to be bigger than 1 page size so that the "Next" button is enabled.

Other ideas?

Answer 1

One way to do it is as I said in the comments, to use a 'countless' approach. Modify the client side script in such a way that the Next button is always enabled and fetch the rows until there are none, then disable the Next button. You can always add a notification message to say that there are no more rows so it will be more user friendly.

Considering that you are expecting a significant amount of records, I doubt that the user will paginate through all the results.

Another way is to schedule a cron job that will do the counting of the records in the background and store that result in a table called totals. The running intervals of the job should be set up based on the frequency of the inserts / deletetions.

Then in the frontend, just use the count previously stored in totals. It should make a decent aproximation of the amount.

Answer 2

Depends on your DB engine. In mysql, solution looks like this :

    mysql> SELECT SQL_CALC_FOUND_ROWS * FROM tbl_name
        -> WHERE id > 100 LIMIT 10;
    mysql> SELECT FOUND_ROWS();

Basically, you add another attribute on your select (SQL_CALC_FOUND_ROWS) which tells mysql to count the rows as if limit clause was not present, while executing the query, while FOUND_ROWS actually retrieves that number.

For oracle, see this article : How can I perform this query in oracle

Other DBMS might have something similar, but I don't know.