M
M
Maxim2016-04-10 15:53:54
Doctrine ORM
Maxim, 2016-04-10 15:53:54

Is there any way to speed up doctrine flush?

Hello!
There are 8000 products + categories + stock availability. Parse from csv.
In total, when inserting a doctrine, about 16,000 lines come out. It takes 1 minute + - 10 seconds.
Since the insertion occurs automatically in a loop - to avoid php memory leaks, the construction is used

foreach(.... $key->$value) {
....... создание объектов и persist
if($key % 20 === 0 ) {
  $em->flush();
  $em->clear();
}
}
$em->flush();
$em->clear();

, thanks to which flush eats not 120, but 50 mb. At the same time, the losses are not significant - it adds 10 seconds.
Interested in the question, is this a normal speed for such a volume of data or can it be accelerated?

Answer the question

In order to leave comments, you need to log in

2 answer(s)
A
Alexey Romanenko, 2016-04-10
@webmaxer

Here time and memory are spent not only on flush (). Creating an object will also take time.
The doctrine most likely will not be able to accelerate. It is possible to interpose through pdo.

G
gtr52, 2016-04-14
@gtr52

Try to dig aside:
When implementing one synchronization in an online store, I also encountered that the update took more and more time with each following product.
$em->clear(); did not help, as I understood with $em->flush(); - the doctrine goes through the list of entities in memory that were loaded into it and checks whether there was a change, and so the check took a huge amount of time.
Total when using $em->flush($object); - parsing XML for 4000 products and updating their data in the database - fits in 5-10 seconds.
Might help)

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question