How to parse with wildberries?

Question

Python

There was a task to parse goods from wildberries. You need to constantly monitor the first page of a certain category and, if there are new products, then send it to the client (in my case, this is telegram). The fact is that you need to check the link every 5 seconds, and there are more than 500 of them, how can this be done so that the server does not block requests, use a proxy? And is it really necessary?

Answer 1

Let's start with the fact that 500 pcs. every 5s is 100 requests per second. Proxies will be banned very quickly with such traffic.
Further, obtaining data mega-fast makes sense in automated trading systems on the stock exchange, and if this is sent to a person in telegrams, then in any case he will respond much slower than these 5 seconds. And why then ask so often? If you poll with an interval of 3 seconds, then the goods will arrive with a delay of 13 minutes on average, a maximum of 25 minutes, which is quite acceptable for many tasks.
I advise you to talk with the customer again on the topic of what exactly he needs.

Answer 2

selenium

Answer 3

I can process 500 pages in 3 seconds without proxy and blocking!!!
If you're interested, I can show you!

Answer 4

do the impudence on increasing
at first without a proxy,
then suddenly it will work wisely: there, go, the price does not change every 5 seconds
, and so on.

Answer 5

Not quite on the topic of the question, a parser for the number of purchases of goods for wildberries was required, if anyone needs to find this solution and use it .