Rendering huge tables in symfony

Question

I have to create a backend page in symfony that is expected to produce a table with several thousands of lines.

I know that normally this is "not nice", but in this case I can't add paging buttons and similar stuff because this page is used as a way to export data outside from the website: in other words, it doesn't matter if it takes minutes to be rendered, but when visible this page has to contain ALL the data I need. I will then take care of potential security issues by restricting this page to the only user allowed to access it.

However, if I rely on the usual symfony MVC structure, I get out of memory errors simply because I need to store all the data in a buffer before sending it to the twig template.

Do you know if it exists a simple way to render this page without incurring in memory exaustion issues? In plain PHP I would have simply created an iteration extracting a portion of the results from the query and rendering a portion of the table every time, but can I do something similar by following the symfony Controller->twig Template structure?

Answer 1

A traditional way to do it, is to use two variants of the query.

1. Includes paging, but setting the range and max results.

2. Doesn't.

In practice, the first query extends the second but adding the range filters, but when you request an export, the results of the second are streamed as a file, allowing for very large downloads. Furthermore you can select each row individually and dump it.

You should be able to use $query->iterate() to fetch one at a time, although I haven't tested the behavior of this method, the documentation implies it doesn't load them all at once.

http://www.doctrine-project.org/api/orm/2.4/source-class-Doctrine.ORM.Query.html#563-577

A useful export form would be CSV whic you can write to using PHPs inbuilt functions.

Answer 2

If you do not need to include every field in your table, especially the largest ones, you can use a partial query to return only the fields you need to display. I have used this to return over 20k rows of data on a single page with 5 or 6 short text fields that aligned on one row.

$q = $em->createQuery("select partial u.{id,name} from MyApp\Domain\User u");

See the doctrine2 docs here