How profitable is doing web scrap / data mining on freelance now?

Question

Freelance

Interested in the experience of those who work in this direction - does it make sense to meddle? Is it hard to parse? Do many sites block parsers? What are the pitfalls? I took a quick look at the upwork, they seem to offer normal prices for projects.

Answer 1

Anything can be parsed if there are resources for it.
Normal prices are usually because normal overheads. Even a good pool of proxies to raise or buy can cost a lot of money. And it’s good if you have it constantly involved and there is a flow of work, otherwise it may be economically unprofitable.
As an example, we were recently sold a base of crunchbase companies (about 800k) for 5k (although they wanted 10k). If I parsed myself, <1k would be enough for me for the proxy pool and anti-captcha. Profit for a freelancer is obvious.
Well, now it’s not fashionable to just parse pages, everyone wants data-science

Answer 2

It makes sense to meddle if there is an interest in such work and not just the goal of earning. On the upwork you can find more or less interesting projects, but there are not so many of them with a sane amount of payment. The complexity of parsing is different, and, accordingly, the cost will be very different. Some things will not be worth parsing at all, because solving the challenges will not cover the amount that people are willing to pay. In order to quickly and efficiently bypass protections, with minimal time and resource losses, you need to have a developed framework with proxy rotation, captcha solving (OCR and 3rd Party), bypassing challenges of modern anti-scraping protection systems, etc. That is, to have a tool that reduces the cost of development and collection and speeds up processes, without which it will be very difficult to maintain the quality level, meet deadlines and increase volumes.

Answer 3

And before it was not profitable, but now it has not become more profitable.

Answer 4

Do many sites block parsers?

And why parzit from one address and everything in a row without pauses?
Yes, just like everything else, like any other job: only pros make good money.
It's only good if you specialize in it.
Otherwise, the overhead costs for each project will eat up all the income.
If you are constantly engaged, then you can keep a pool of web spiders and you already know everything about technology - therefore, fulfilling a specific order will take you nothing at all.
If you ask such questions - then for you personally: you should not do this.
Of course, it is always easy for the performer. Much easier than for the customer. Otherwise, the customer would not pay the contractor for this.