Web scraping, parsing in Python?

Question

Python

Hello, I'm learning python, I'm interested in the web. I want to try freelancing, I found out about parsing (web scrapping)
, I'm interested.
How relevant is web scraping now?
How good is Python for this?
Is it possible to make money on this now?
Please tell us about this, and if it’s not difficult, advise some kind of platform, a freelance site for a beginner, so that you can take your first order, even if it’s free, now the experience itself is interesting.

Answer 1

I'll tell you about my experience - I came to programming 1.5 years ago, before that I had nothing to do with it at all. I started learning Python when I got tired of solving problems from textbooks - I registered on Upwork and Freelancer.com. I was looking for orders specifically for parsing, because. For me, this seemed like the easiest thing to do at the time. Because I didn’t have any experience and I didn’t have a portfolio either - I couldn’t get my first order for a long time and for this I was banned from Upwork, but I slowly started taking orders on freelancer.com. There are many orders for parsing / scraping and for different budgets - from 20 USD to hundreds and thousands per project (in such projects, in addition to parsing, something else usually needs to be done, usually something related to ML and Data Science). With the growth of experience, the number of orders made and the number of positive reviews, I began to raise my bar with the cost of orders. The other day I calculated that in 2018, purely on parsing / scraping, I earned an average of about 500 USD per month (+ - 50). I'm currently learning Django, because it became more interesting for me
PySy: if you drop skype or telegram, I can drop real projects for practice) or maybe we will agree on cooperation ..

Answer 2

An area for newbies that pays poorly due to the large supply in this low-skilled job market. Python is great for this kind of work.

Answer 3

I don't know python, I write in php, I like to take orders for scraping for a change. Full of orders. There are also quite fatty ones with a good price. Upwork platform.
For scraping, you need to know: ideally, how http works (headers, cookies, sessions, etc.), multithreading / queues, RE (regular expressions) is a must.
You need to understand that often scraping is on the verge of legality. And there are many cases when scavengers were caught and judged, both in Russia and abroad, so you should always be aware of how legal your actions are and whether they can figure you out if you wish.

Answer 4

Scraping is probably one of the top 4 freelancing topics. (The rest is the creation of web applications, neural networks, higher mathematics or statistics - they come across a little less often. But you need to know certain frameworks. For mathematics and neural networks, you need a higher education.)
Earnings - an average of $ 30 per site, but you will need to sit on each site several full days. Those. Realistically get somewhere on 1-2 $ / hour. It is more difficult to find an hourly rate for scraping if the customer is not a beginner and has compared prices. This is provided that you understand scraping frameworks. Also keep in mind that many sites do not scrape like html, because they are dynamic in JS, you need to learn additional. frameworks, and any protection can stand. Most giant sites don't scrape together. Facebook is in a very difficult situation right now.
At the same time, there are crowds, really crowds, Indians, lowering prices.
There are bad customers, pulling deadlines. Those. You can work full time for a whole week. And even a month - for $ 20. Minus 20% that the freelance exchange will take (percentage may vary).
IMHO, freelancing is for those who do not have a job, for some life reasons. Or a lot of free time in which you can earn extra money. But it should be a lot, because problems require many hours to find solutions.
And scraping is for those who don't like messing around with the frontend. Which will take most of the time when developing web applications, on the same django.

Answer 5

By itself, parsing will be of little interest to you, because. offers are plentiful and competition is mainly on price. But if you also learn how to process the data that you pull out from the sites, then here you can already look for orders in a more interesting way.