Load big XML files to mySQL database (PHP)

Question

For a new project I need to load big XML files (200MB+) to a mySQL database. There are +- 20 feeds i need to match with that (not all fields are the same).

Now when i want to catch the XML I get this error:

Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 171296569 bytes) in E:\UsbWebserver\Root\****\application\libraries\MY_xml.php on line 21

Is there an easy solution for this? It's not possible to get te feed in parts of a few MB's each.

Thank you very much!

P.s. has somebody an idea to match xml-feeds easy?

Answer 1

This should fix it:

ini_set('memory_limit', -1);

Answer 2

If you have to parse big xml files (or big files in general), it's a better strategy to write a stream based function than one that requires everything to be in memory at the same time. PHP has a class called XmlReader, that allows you to scan through a file, one node at a time. It may not be trivial to change your code to use it, but you might consider making the change.

Answer 3

I wrote a blog entry about that a while ago. The solution was to use XMLReader to read chunks of the XML into an object and do things with it then.

Answer 4

Add ini_set('memory_limit', '150MB') into your script.

Answer 5

Is PHP the only programming language you would consider for the job? If you use Java, then there are more and efficient ways to load XML..