A
A
Alex Murfy2017-04-15 16:32:28
Parsing
Alex Murfy, 2017-04-15 16:32:28

Public parser of goods by filters from different sites, technology stack?

Good day to all!
There are 3 portals with goods, you need to make it so that the user can create his own filter after registration (weight, price, volume, etc.).
The task is to parse for each user every minute the products according to these filters and send a link to a NEW product.
In general, the bottom line is that a user can have 60 filters (for example, 20 for each portal) and every minute you need to parse these 60 pages for a new product (i.e. make 60 requests).
And if there are at least 1000 such users, then this will be 60,000 requests every minute.
The project must be in php. Who wrote parsers, please share your experience. Because at this stage of implementation, everything works, I use multicurl. But as soon as I set 90+ filters and imitate the presence of 10 users, everything becomes very sad.
Thanks in advance :)

Answer the question

In order to leave comments, you need to log in

1 answer(s)
M
Mi7teR, 2017-04-15
@Mi7teR

Parse all products from all three portals, then only parse new products. And give the user products already from their own database according to their filters.

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question