How to use less memory while running a task in Symfony 1.4?

Question

I'm using Symfony 1.4 and Doctrine.

So far I had no problem running tasks with Symfony. But now that I have to import a pretty big amount of data and save them in the database, I get the infamous

"Fatal Error: Allowed memory size of XXXX bytes exhausted"

During this import I'm only creating new objects, setting a few fields and saving them.

I'm pretty sure it has something to do with the number of objects I'm creating when saving data. Unsetting those objects doesn't do anything though.

Are there any best practices to limit memory usage in Symfony?

Answer 1

I've come across this, and there's a couple of techniques I found really helped with Doctrine's extensive memory usage.

1: Where possible, hydrate your Doctrine query results down to an array. You can do this as follows eg:

$query = self::createQuery("q")->
  ...
  ->setHydrationMode(Doctrine::HYDRATE_ARRAY)
  ->execute();

This forces Doctrine to NOT create large objects, but instead reduces it to an array. Obviously bear in mind that if you do this, you lose the ability to call methods etc, so this is only good if you're using it for reading field values etc.

2: Free your results after execution. This is documented in a tiny area of the Doctrine docs, but it's really helped out the import task I was using:

$query->free();

That's it. You can also do this on objects that you've created, eg $myObj->free(); and this forces Doctrine to remove all the circular references that it creates. Note that circular references are automatically freed from PHP 5.3 onwards on deletion of an object via the PHP scope or unset(), but before that you'll need to do it yourself.

Unsetting variables after you've used them also helps, although do this in conjunction with the free() method above as mentioned, as unset() won't clear the circular refs otherwise.

Answer 2

Try this :

Doctrine_Manager::connection()->setAttribute(Doctrine_Core::ATTR_AUTO_FREE_QUERY_OBJECTS, true );

as mentioned on

php/symfony/doctrine memory leak?

Answer from Jordan Feldstein not mine.

Answer 3

Another hint for reducing the amount of memory used inside a task is to disable the query profiler. A big number of queries tends to make the task using more and more memory.

To do so create a new task environment in your database.yml config file by adding the following lines:

task:
  doctrine:
    class: sfDoctrineDatabase
    param:
      profiler: false

Then setup your task to run in "task" environment. It should help to keep memory usage stable if your queries are in a loop.

Answer 4

Sorry I know this is a late answer, but could help someone.

Another potentially huge memory saver is to make sure Symfony's debug mode is not enabled for that task. In a couple of long-running tasks I added this line and it cut down the RAM usage about 40% even after I had optimized things like hydration mode.

sfConfig::set('sf_debug', false);

Answer 5

I've had the same problem with PHP batch jobs for symfony--if they run for a long time and use a lot of data they tend to balloon, and even if I made one wrapper that invoked many separate PHP processes, it didn't help.

Because of this, I've rewritten my bigger batch jobs with Perl's DBI and they're reliable and manageable.

I'm not suggesting this is the best answer, just sympathizing and offering my experience. There's probably a way to make PHP behave better.

Answer 6

Caution with fetchOne() on Doctrine Query. This function call will not append "Limit 1" on SQL

If you just need to get one records from DB, make sure:

$q->limit(1)->fetchOne() 

The memory usage is tremendous dropped on large table.

You could see fetchOne() will fetch from DB as a collection first then return the first element.

public function fetchOne($params = array(), $hydrationMode = null)
{
    $collection = $this->execute($params, $hydrationMode);

    if (is_scalar($collection)) {
        return $collection;
    }

    if (count($collection) === 0) {
        return false;
    }

    if ($collection instanceof Doctrine_Collection) {
        return $collection->getFirst();
    } else if (is_array($collection)) {
        return array_shift($collection);
    }

    return false;
}

Answer 7

Also worth looking into:

gc_collect_cycles — Forces collection of any existing garbage cycles

Answer 8

Also try to limit the (select) fields in your query only to those you really need.

for example use something like:

$query = self::createQuery("q")->
  ->select('id','title','price')
  ...

instead of:

$query = self::createQuery("q")->
  ->select('*')
  ...