How to update a large amount of data?

Question

PHP

I have a user base for which some data needs to be updated once a week. This data is parsed from other sites. The problem is that the update script takes a long time to complete, and the hosting has a 30 second limit.
If you update the information of one user, and not all at once, the script will run for 10-20 seconds. You can try running it with a cron every minute, passing in the user ID, but you don't want to manually create hundreds of rules.
How can this problem be solved? I can't change the script execution time.

Answer 1

Add a column with the date of the last update to the users table. The script teaches you to find a victim yourself and run it on schedule every minute. In this case, only one rule is required, and the script itself will find users who need to be updated.

Answer 2

Run the script on another machine.

Answer 3

Are you sure that you cannot write the following code at the beginning of the script?

ini_set('max_execution_time', 0);

Answer 4

1. Make the task of parsing and updating data asynchronous. The parser creates a pool of tasks for updating, then the second script processes this pool, updating the necessary data.
2. Try to implement update by trigger. As soon as the parser pulled out the data and inserted it into the database, they will be automatically updated in the necessary tables. It must be remembered that this method is not always good, and it can lay the base on the shoulder blades due to constant indexing or vice versa, rare indexing.

Answer 5

I would do this (for example, if mysql is used to store users): I would collect all the data on another machine, make one single sql dump to insert / update data, and then upload it to the hosting.

Answer 6

On another machine, write a script with unlimited execution time that will connect to the database on your host and do all the work.
The database, accordingly, must be taught to listen not only to local calls (in MySQL, you can limit external calls to a list of IP addresses).
In a database for heavy operations within the database itself, write utility stored procedures so as not to drive data over the network.

Answer 7

You can, of course, remember the user id on which the script timed out and restart it in case of an error, starting from this ID, well, provided that the IDs are ordered. But it's kind of a perversion :)

Answer 8

When there was a restriction on the hosting, I implemented a call to the parser via ajax. That is, the queue is processed on the client. But this does not apply in all cases.

Answer 9

I get around the broken set_time_limit by creating a job queue.
Simply
select id,task from update_list limit 3 where active = 1
then when processing is done
update update_list limit set active=1 where id= $obj->id
The main thing is that one operation does not exceed 30 seconds, of course.
If it exceeds, it is necessary to cut the tasks into pieces, optimize the database.
For example, split one table into many by record id, for example, creating tables:
users_10k
users_20k