Loading all the data up front is as inefficient as you can get
Regardless of the cost in server resources, it's just bad system design. The most efficient & better way will be to load only the data you need to display:
Imagine if Google loaded all the results for your query each time. It would not only degrade your experience, but overload their servers to the point of making it unusable
Pagination
You'd be much better using a pagination solution:
Both of these work in a similar way - by applying their .page()
& .per()
methods on your query, the gems only return the data for that page (like Google)
If you wanted to then provide an ajax pagination solution, you'd be able to create the feeling of all the data being available, without the massive overhead: