D
D
denis_druid2017-06-12 19:31:38
PHP
denis_druid, 2017-06-12 19:31:38

Why is the "heavy script" executed every other time?

Hello. Help me decide this:
There is a parser script, it collects and runs a lot of text through various filters, then it pours the whole thing into the database, cleans the old data from the database and stuff like that.
Runs for approximately 5 minutes.
The problem is that every second or third time it only processes half of the data.
Example:
1st run: 310 entries added
2nd run: 108 entries added
3rd run: 311 entries added
4th run: 327 entries added
5th run: 113 entries added
6th run: 324 entries added
etc.
300+ records is the norm, as it should be. But every time it "doesn't finish", then there are always about 100 records (106, 104, 108, etc.).
At first I sinned on cron, that somewhere there was a plugging going on, I was picking it for a day. But a couple of times today I also observed this when running the script manually through the browser. The feeling that some kind of cache is clogged somewhere, and then resets to zero after a while. But something else is possible.
Where to dig?

Answer the question

In order to leave comments, you need to log in

3 answer(s)
D
denis_druid, 2017-08-19
@denis_druid

So, the problem was solved as follows:
As I wrote above in the comments, the script flaw occurs only at certain hours in the morning. Most likely, the remote server from which the parsing is being performed is being loaded with service tasks at this time, which increases the response time.
There were no errors in my server logs. Apparently, the remote server did not answer as it should.
The solution came after changing this line:
With 60 seconds the problem went away completely.

S
Serezha, 2017-06-12
Ahen @Ahen

View logs, server graph from memory.
But usually it's all about the time limit of the script (again, there should be a corresponding error in the logs, like The process *** exceeded the timeout of 60 seconds), the limit is set so that hung scripts are killed automatically. Decided in php.ini, max_execution_time item in seconds.

W
Wexter, 2017-06-12
@Wexter

1) there was not enough memory, see the logs
2) there was not enough time, again, see the logs

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question