What is the most efficient way to paginate my site when querying with SQL?

Question

I am trying to paginate the results of an SQL query for use on a web page. The language and the database backend are PHP and SQLite.

The code I'm using works something like this (page numbering starts at 0)

http://example.com/table?page=0

page = request(page)
per = 10 // results per page
offset = page * per

// take one extra record so we know if a next link is needed
resultset = query(select columns from table where conditions limit offset, per + 1)

if(page > 0) show a previous link
if(count(resultset) > per) show a next link

unset(resultset[per])

display results

Are there more efficient ways to do pagination than this?

One problem that I can see with my current method is that I must store all 10 (or however many) results in memory before I start displaying them. I do this because PDO does not guarantee that the row count will be available.

Is it more efficient to issue a COUNT(*) query to learn how many rows exist, then stream the results to the browser?

Is this one of those "it depends on the size of your table, and whether the count(*) query requires a full table scan in the database backend", "do some profiling yourself" kind of questions?

Answer 1

i'd suggest just doing the count first. a count(primary key) is a very efficient query.

Answer 2

I've opted to go with the COUNT(*) two query method, because it allows me to create a link directly to the last page, which the other method does not allow. Performing the count first also allows me to stream the results, and so should work well with higher numbers of records with less memory.

Consistency between pages is not an issue for me. Thank you for your help.

Answer 3

There are several cases where I have a fairly complex (9-12 table join) query, returning many thousands of rows, which I need to paginate. Obviously to paginate nicely, you need to know the total size of the result. With MySQL databases, using the SQL_CALC_FOUND_ROWS directive in the SELECT can help you achieve this easily, although the jury is out on whether that will be more efficient for you to do.

However, since you are using SQLite, I recommend sticking with the 2 query approach. Here is a very concise thread on the matter.

Answer 4

I doubt that it will be a problem for your users to wait for the backend to return ten rows. (You can make it up to them by being good at specifying image dimensions, make the webserver negotiate compressed data transfers when possible, etc.)

I don't think that it will be very useful for you to do a count(*) initially.

If you are up to some complicated coding: When the user is looking at page x, use ajax-like magic to pre-load page x+1 for improved user experience.

A general note about pagination: If the data changes while the user browses through your pages, it may be a problem if your solution demands a very high level of consistency. I've writte a note about that elsewhere.