Endless loading huge php file with high memory_limit

Question

Intro

Currently I am working with XML files. With PHP I combine several XML files. This is a big operation that exceeds the default memory_limit of PHP.

When I extend the limit with ini_set('memory_limit', '1024M'); the fatal error is gone and the script doesn't exceed the limit anymore.

The problem I have now is that after combining the XML files my script is buggy. Browsers are loading endless, I am still able to do everything serverside. Like e-mail myself, adding rows to the database. But everything clientside just dont work. Like a simple echo 'Hello World'; or header('location: http://www.google.com');

While combining the files I log the steps.

$stores = $this->Store->getAll();
$this->log(2, 'all', 1337, 'Start processing ' . count($stores) . ' stores ';
foreach($stores as $store)
{
    $this->log(2, $store->id, 1337, 'Start processing store ';
    $this->process_store($store->id, false);
}
$this->log(2, 'all', 1337, 'Done with processing ' . count($stores) . ' stores ';
header('location: ' . BASE_URL . '/datatest');
die();

The strange part is that my script DOES log the last log Done with processing. Anything I do at serverside works.

But the header (redirect) just doesn't. My browser is just endless loading.

Questions

• What is going wrong?
• Do browsers disable communication with the server after some time?

Answer 1

I noticed that the server and client disconnect from each others.

Finaly I found the solution, which is realy ugly!

I had to set a sleep(1); inside the foreach loop to keep the server and client connected.

Omg this is slower then before :'(

Answer 2

You could circumvent the memory problem by doing a recursive XML merge using simpleXML

Code

<?php
private function mergeXML(&$base, $add)
{
    $new = $base->addChild($add->getName());
    foreach ($add->attributes() as $a => $b) {
        $new[$a] = $b;
    }
    foreach ($add->children() as $child) {
        $this->mergeXML($new, $child);
    }
}?>

Usage

<?php mergeXML($xmlElement, simplexml_load_string($toMerge)) ?>

So you would start out and merge XML 1 with XML2 to TempXML1, then merge TempXML1 with XML3, etc. This way, you just go down the list of xml files without loading all of them into memory.

Answer 3

Are you sure that BASE_URL . '/datatest' is a correct URL? You should also put a capital L to Location to ensure the redirect works properly.

Try this without the entire merging to verify that your header redirection works at first.