F
F
Flaker2014-03-22 19:11:55
PHP
Flaker, 2014-03-22 19:11:55

What is the best way to parse and broadcast a large number of RSS using PHP?

It is required to broadcast a large number of RSS feeds ( about 5k ) to several social networks.
PHP script runs for no more than 30 seconds. Runs on the crown.
One run is not enough to process all the tapes.
The first thing that came to mind:
Run the script every minute.
Process tapes within the available time.
When the available time comes to an end, memorize the number of the last processed tape.
At the next start processing from the memorized number.
In this case, if the tapes are large, then the processing of each one will not be fast, therefore, one tape can be parsed with very large time intervals.
Tell me, how do you solve this problem?

Answer the question

In order to leave comments, you need to log in

2 answer(s)
M
Melkij, 2014-03-22
@Flaker

Well, for example:
we put the queue server
By the crown or else how tasks are put to bypass certain addresses in the queue.
Workers parse tasks from the crawler's queue, knock on the address, parse the result.
Or we hang 2 queues:
the first is the tasks for the crawler, which works only with the network, and puts the received documents in the second queue.
Other workers disassemble the second queue and process the tapes.
In this case, the crawler can be made on a multicurl, it will be more efficient to utilize resources, instead of waiting for the network.
But if you have a problem with just the time limit?...

T
Timur Sergeevich, 2014-03-22
@MyAlesya

set_time_limit(0);

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question