How to organize enumeration and processing of a large number of data?

Question

PHP

It is necessary to sort, process and issue at least 2k goods as xml.
But the problem is that the server on which the module is running is very weak and cannot withstand such a load.
What is the best way to do this?

Answer 1

Don't write shitty code.
Debug what eats up all resources or time and fix it

Answer 2

And what kind of load is that?
Write in such a way that you do not store a large amount of data in memory at a time, write everything to disk at once, and you will not have problems with the "load", at most it will take a long time to execute.

Answer 3

It is necessary to refactor the scheme, bring everything into a divine form to 5-6 normal form.
80% of all projects with which I had to deal "like a store in PHP" index varchar and hedgehogs with it. Because of this, "sampling 2K products" is simply an impossible task. It is worth reading the book SQL Antipatterns and High Performance MySQL. Understand how InnoDB works and how optimization works in general .
Crap Schema Example
Table products
id unsigned int autoincrement (PK)
name varchar(255) NOT NULL
group varchar(255) NOT NULL
oneAttr varchar(255)
otherAttr varchar(255)
anotherAttr varchar(255)
description varchar(255)
dateCreated DATETIME
You need to break it into at least 4
tables ) NOT NULL Table products_attrs id unsigned int autoincrement (PK) name varchar(255) NOT NULL Table product_attrs_values ​​product_id unsigned int (PK <- FK) attr_id unsigned int (PK <- FK) attr_value varchar(255) NOT NULL Etc. P. This is the second year of uni :)
And a lot of the townsfolk do not even reach it, which makes me pretty sad.
And, of course, you need to be able to use EXPLAIN .

Answer 4

But the problem is that the server on which the module is running is very weak and cannot withstand such a load.

Do it via PHP CLI. Take goods from the database in batches, and not load everything at once into memory.