Displaying very large amount of records at once in Rails on Heroku

Question

I've got more than 6000 User rows with multiple associations to display more than 15 columns for each row.

Whenever I invoke this action to retrieve, Heroku seems to refuse processing after a while and stops.

I've tried find_each but this doesn't seem to work so I'm setting up Resque with RedisToGo.

It seems to work fine with other queries especially useful for creating and updating large amount of records but how would I efficiently display this to the user?

Would I just use something like Resque Status to display status to the user on the page with currently processing then display when it's ready?

How do real world applications deal with this kind of issue?

Answer 1

Loading all the data up front is as inefficient as you can get

Regardless of the cost in server resources, it's just bad system design. The most efficient & better way will be to load only the data you need to display:

Imagine if Google loaded all the results for your query each time. It would not only degrade your experience, but overload their servers to the point of making it unusable

---

Pagination

You'd be much better using a pagination solution:

• will_paginate
• kaminari

Both of these work in a similar way - by applying their .page() & .per() methods on your query, the gems only return the data for that page (like Google)

If you wanted to then provide an ajax pagination solution, you'd be able to create the feeling of all the data being available, without the massive overhead:

Answer 2

I think heroku has a timeout of 30 secounds. And probably your request takes more than that.

You should watch out for n+1 queries. Are you using joins or include to use less queries?

Also in this case it's usual to use pagination. If you don't want to have multiple pages you can simply load pages with ajax. Maybe an endless scroll is what you need. This way you avoid long request which is really a bad practice.