paginating over big number of records

Question

I've got a big number of records and I display only part using a pagination. Which is ok. Since I have many pages I'm obtaining count on each page, is there a tecnique to avoid this?

I've got a redis as a session store and a postgres database.

I was thinking to perhaps save a query result count in a session (I have only one search page).

I'm sure this is a frequent occurrence for someone who is doing this a long time.

What can be done to elegantly solve this? (Not to get the count on every page that is)

Answer 1

You can use infinite scrolling. This is what major sites like facebook use.

http://railscasts.com/episodes/114-endless-page-revised?view=asciicast

Answer 2

Use sorted_set to store your records, you can fetch records from a specific index with a limit.

Answer 3

Use NOTIFY and LISTEN from your Postgres table triggers and a seperate server-side process (for example a python script).

Keep your redis data up-to-date, make sure to handle rolled back transactions.

You can use a redis ZSET (sorted set) for storing an easily paginatable 'index'. If you have several ways of ordering your data, you can have a HSET with your actual data, which you reference from your ZSETs by id. In a redis Lua script, you can get a quick result back in one call that collects the HSET data using the ZSET id's.

Kind regards, TW