B
B
BadCats2021-05-10 04:11:44
Computer networks
BadCats, 2021-05-10 04:11:44

Creating a proxy server for site parsing?

DISCLAMER: I'm new to networks and just started to understand this, so I can write / describe some things incorrectly.
Actually the problem:
There is an online store, the prices from which must be parsed. Of course, it has protection against bots - by ip and the number of requests and captcha. After n-th time, the parser stumbles over the captcha. As I understand it, I need an anonymous proxy/vpn to change my ip?
If so, then the following question arises:
Is it possible, having a dynamic ip from the provider - to configure a proxy server on mikrotik so that requests from the parser go through it? If you can, I will be grateful if you throw a suitable tutorial at me (I seem to have read somewhere that just OpenVPN is suitable for this?). If not, I'm happy to hear alternatives.
A couple of clarifications:
1) You need to set up a proxy yourself, if there is such an opportunity, and not rent it on the side - for two reasons:
- security
- the desire to learn how to do it (a more weighty argument)
2) To be honest, I'm horrified by the number vpn types I
started reading the documentation:
https://mikrotik.wiki/wiki/%D0%A2%D0%B5%D0%BE%D1%8...
- but, if possible - in a nutshell - not about HOW different protocols work, and WHY AND WHEN they are used.

Answer the question

In order to leave comments, you need to log in

1 answer(s)
C
CityCat4, 2021-05-10
@BadCats

Interesting. A person who understands that he is doing something "near-legal" comes and asks "uncles, teach me bad things" :)
Bad, dude, they learn on their own :)

Is it possible, having a dynamic ip from the provider, to configure a proxy server on mikrotik so that requests from the parser go through it?

Of course it is possible. In the event that the speaker is white. If it is gray - it does not make sense, because it "turns white" somewhere at the provider.
However, the ending of this story will be approximately the same, only if the dynamics are gray, it will come much faster if the store that is being parsed is fat.
The support will receive an application about a captcha that came from nowhere on the store's website, where "I've been going for a hundred years." One, two, hundredth... Support will find out (either from the store, or by itself) where the flow of requests comes from and will simply ban you for violating the contract. If the dynamics is white and the pool of the Provo is large, this will work for some time (depending on the speed of removing bans on the store's website).
(I kind of read somewhere that just OpenVPN - is it suitable for this?

And I read somewhere that in Australia people walk on their heads and are called "antipodes" :) VPN is a Virtual Private Network. Which of these words can be applied to parsing - I don't know :)
and WHY AND WHEN they are used.

When there is a need - then apply. This is a topic for a solid book.

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question