There is a news site parser. He collects articles, but how to make sure that he does not collect unnecessary paragraphs?

Question

Python

There is a parser, it goes through the navigation page, collects urls, follows the urls and copies news articles. The whole problem is that the articles have tags that are also parsed, but I don't need them. How can this be bypassed. The unwanted tag has an "insert" class, but the strip() method doesn't help, it removes all the tags.

Answer 1

You need either regular expressions or dom crawler. Look for an unnecessary tag and remove it along with the content.