How to optimize RAM consumption?

Question

symfony

Good day!
There is a task to parse values ​​of one table and add to another.
An entity is given that characterizes a many-to-many relationship between Product and Feature with product_id and feature_id filled in. In the "value" property of this entity, you need to add values ​​​​from another, erroneous, entity.
Here is the script I wrote

private function parseValuesForProductsFeatures()
    {
        $em = $this->getContainer()->get('doctrine')->getManager();
        for ($iteration = 1; $iteration < 1003884; $iteration += 1) {
            $productsFeaturesValue = $em->getRepository('MainCatalogBundle:ProductsFeaturesValues')->findOneBy(array(
                'id' => $iteration,
            ));
            $productId = $productsFeaturesValue->getProduct()->getId();
            $featureId = $productsFeaturesValue->getFeature()->getId();

            $fieldsValue = $em->getRepository('MainCatalogBundle:MistakeProductsFeatures')->findOneBy(array(
                'productId' => $productId,
                'featureId' => $featureId,
            ));

            $value = $fieldsValue->getValue()->getValue();
            $productsFeaturesValue->setValue($value);

            if ($iteration % 200 === 0) {
                $em->flush();
                $em->clear();
            }

        }
        $em->flush();
        $em->clear();
    }

Given that about a million rows of RAM is not enough. Doing unset() with all variables at the end of every 200 iterations doesn't help. Where to dig, what to solve the problem?

Answer 1

First, check that you are executing code in the prod environment. The dev environment eats significantly more resources.
And then you can switch to DQL without hydration into objects, or even to native queries that do not return anything in PHP, if you can issue a query with such logic.

Answer 2

Is this a console solution?
Why do everything impudently? 1 and 1003884 (the beginning of the end of the cycle) forget it as an argument for cli
, complete the task in smaller approaches.
1 -1000
1001 -2000

Answer 3

Or maybe add the magic of iterators?))

<?php
$batchSize = 20;
$i = 0;
$q = $em->createQuery('select u from MyProject\Model\User u');
$iterableResult = $q->iterate();
while (($row = $iterableResult->next()) !== false) {
    $em->remove($row[0]);
    if (($i % $batchSize) === 0) {
        $em->flush(); // Executes all deletions.
        $em->clear(); // Detaches all objects from Doctrine!
    }
    ++$i;
}
$em->flush();

Taken from here:
latest/reference/batch-processing.html#id1