Answer the question
In order to leave comments, you need to log in
Public parser of goods by filters from different sites, technology stack?
Good day to all!
There are 3 portals with goods, you need to make it so that the user can create his own filter after registration (weight, price, volume, etc.).
The task is to parse for each user every minute the products according to these filters and send a link to a NEW product.
In general, the bottom line is that a user can have 60 filters (for example, 20 for each portal) and every minute you need to parse these 60 pages for a new product (i.e. make 60 requests).
And if there are at least 1000 such users, then this will be 60,000 requests every minute.
The project must be in php. Who wrote parsers, please share your experience. Because at this stage of implementation, everything works, I use multicurl. But as soon as I set 90+ filters and imitate the presence of 10 users, everything becomes very sad.
Thanks in advance :)
Answer the question
In order to leave comments, you need to log in
Didn't find what you were looking for?
Ask your questionAsk a Question
731 491 924 answers to any question