How to parse large volumes in Yii2?

Question

Yii

Hi all.
There is a code that parses data on api prices for an item in different cities

foreach (Items::find()->where(['in_market' => 1])->all() as $item){
                foreach (Cityes::find()->all() as $city){

                    /**
                     * @var $city Cityes
                     */
                    $price = $this->getApiPrice($item->unique_name , $item->enchantment_level , $city->unique_name , $item->quality);

                    if ($price){
                        $model = new Prices();
                        $model->item_id = $item->id;
                        $model->city_name = $city->unique_name;
                        $model->sell_price_min = $price[0]['sell_price_min'];
                        $model->sell_price_max = $price[0]['sell_price_max'];
                        $model->sell_price_date = '';
                        $model->buy_price_min = $price[0]['buy_price_min'];
                        $model->buy_price_max = $price[0]['buy_price_max'];
                        $model->buy_price_date = '';
                        $model->save();
                    }

                }
            }

His problem is that he rests during the execution of the script. I tweaked it a bit.

foreach (Items::find()->where(['in_market' => 1])->limit(1)->offset(Yii::$app->request->get()['offset'])->all() as $item){
                foreach (Cityes::find()->all() as $city){

                    /**
                     * @var $city Cityes
                     */
                    $price = $this->getApiPrice($item->unique_name , $item->enchantment_level , $city->unique_name , $item->quality);

                    if ($price){
                        $model = new Prices();
                        $model->item_id = $item->id;
                        $model->city_name = $city->unique_name;
                        $model->sell_price_min = $price[0]['sell_price_min'];
                        $model->sell_price_max = $price[0]['sell_price_max'];
                        $model->sell_price_date = '';
                        $model->buy_price_min = $price[0]['buy_price_min'];
                        $model->buy_price_max = $price[0]['buy_price_max'];
                        $model->buy_price_date = '';
                        $model->save();
                    }

                }
            }

            if (Items::find()->where(['in_market' => 1])->count() > Yii::$app->request->get()['offset']){
                return Yii::$app->response->redirect(['site/import-data' , 'data' => 'price' , 'offset' => Yii::$app->request->get()['offset'] + 1]);
            }

But in this way I rest on the number of redirects.
Tell me if it is possible to somehow beat requests into groups, or some other options.
Extending the script execution time is a bad option, are there any other options.

Answer 1

1. Use console controllers instead of web and set the frequency of its operation.
2. Use queues. Put the task in the queue. The console command goes through all the queues and executes the task. Similar to the first option.
3. The most not relevant, but effective in your situation, is to use not objects and Active Query for queries, but “bare” SQL queries that return data in an array. AR consumes quite a bit of memory for objects. It will save you for a while. But it is better to use console commands. It is more suitable for such tasks.
4. Prepare denormalized data for a quick query into a separate DBMS table or use fast NOSQL databases. And from them to read your data.