How to extract article text from html page?

Question

Python

Hey!
There are news sites, blogs, etc., whose pages contain articles. But in addition to the texts of the articles, the pages also contain comments, advertising, navigation, and so on. The task is to extract from the pages of news sites, blogs, and so on. only the texts of the articles . Since the sources are different, the html markup is different for everyone. That is, you need to implement something like a read mode.

Answer 1

We take bs4 and write a parser for each site

Answer 2

If there is micro-markup on the site, it's easy.
If not - individually for each site.
Or try to guess automatically, but there the quality will be much worse.
Or display a query to the user.
See how this parser works https://evernote.com/intl/en/products/webclipper
Sometimes it guesses, sometimes it misses, but it offers several options. With the possibility of manual correction in some cases.
The developers who made it are pretty well-paid guys.
Evernote is worth a billion dollars and notes are their main service.
So rest assured - the parser is well done.