How to process a request without getting a timeout error?

Question

1C-Bitrix

There is such a piece of code in one of the methods
getList($params)->fetchAll()that
gets all the necessary data from the custom table.
Accordingly, with a large number of elements, the execution of the script falls in memory.
How can this problem be solved, while getting all the elements in the same way in the end? Do not offer an option with an increase in memory_limit)

It is clear that some kind of iterator is needed or use limit / offset, but I can’t figure out how.

Answer 1

> which gets all necessary data from the custom table.
Error here
The number of required elements to display on the page cannot be such that a timeout occurs when receiving them. A person is able to perceive, well, a maximum of a thousand lines. And better - 50.
Therefore, 50 lines are necessary, and not "all".
If the data is retrieved for further processing, then you need to learn SQL and process the data on the database side, getting a ready-made result
.

Answer 2

You guessed correctly that an iterator is needed. PHP has such a thing as a generator, one of the varieties.
Now, if you use it, then you can, in principle, try to meet the memory limit.
On my projects, for a complete pass through the records in the database, I use something like this:

public function getEntitiesByBatch(int $limit): \Generator
    {
        $expr = $this->createQueryBuilder('s');
        $predicates = $expr->expr()->andX();
        $predicates->add($expr->expr()->isNotNull('s.field1'));
        $predicates->add($expr->expr()->isNotNull('s.field2'));
        $accountState = $expr->expr()->eq('a.field3', 'true');

        $qb = $this->createQueryBuilder('s')
            ->where($predicates)
            ->join('s.account', 'a')
            ->where($accountState)
        ;

        $batches = ceil($qb->select('count(s.id)')->getQuery()->getSingleScalarResult() / $limit);

        for ($batch = 1; $batch <= $batches; ++$batch) {
            yield $this->createQueryBuilder('s')
                ->where($predicates)
                ->join('s.account', 'a')
                ->where($accountState)
                ->setFirstResult($batch * $limit - $limit)
                ->setMaxResults($limit)
                ->getQuery()
                ->toIterable() // тут можно возращать просто массив, но мне нужен итерируемый объект
            ;
        }
    }

Upd 1 And sorry, you have Bitrix here, maybe he has some default methods xs. But the general meaning of generators does not change much from this :)
Upd 2 And yes, this code works in a console script.