D
D
DYLAN2019-02-15 20:59:09
PHP
DYLAN, 2019-02-15 20:59:09

How to process XML over 1GB?

What's the point, there is a site, it has over999 elements in the infoblock, when unloading XML'ka weighs almost 2 GB, how to process such a monster. It is clear that it is stupid to throw the whole thing with SimpleXMLElement and enjoy life will not work. I wrote a class to make life easier for the machine, the bottom line is:
The file is read line by line, the location of the blocks is determined, for example, by the <Product> tag, this block is saved to a line, and then it is thrown into SimpleXMLElement. thus it is possible to go through each product. But the problem is that the elements themselves are very fat, and on the test machine they are processed incredibly long, it takes 1-3 seconds to process one element.
Are there any tricks for working with such large files, or is it just, test, check, and then take a normal server and enjoy the power.

Answer the question

In order to leave comments, you need to log in

2 answer(s)
B
Boris Korobkov, 2019-02-15
@BorisKorobkov

If the structure is simple, then it is better to change it to csv and use php.net/manual/en/function.fgets.php
If you need xml, then php.net/manual/en/refs.xml.php

G
grinat, 2019-02-16
@grinat

Through xmlreader, it reads by node, you can throw each node into SimpleXML and continue to enjoy life, example: biostall.com/import-frickin-huge-xml-files-with-xm...
If you use regexps instead of SimpleXML, you will get very strong acceleration.

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question