Beautiful Soup, html5lib or lxml?

Question

Python

It is supposed to be used for parsing user-generated content. Accordingly, the main requirement is the correct processing of broken HTML. Speed ​​is not critical. The lxml

documentation has this:

1. BeautifulSoup Parser
2. html5lib Parser

Those. it can parse with these libraries and return an lxml tree . The html5lib docs say:

Support for minidom, ElementTree (including cElementTree and lxml.etree ), BeautifulSoup (deprecated) and custom simpletree output formats

I will most likely need to go over the entire DOM, I think SAX will be convenient. Even so: run through SAX and build a new tree using certain filter-transforming rules and text typography.

Here in thought. Tell me what to choose?

Answer 1

html5lib has the most correct and reliable parser (according to the spec), but it is slow. lxml is the fastest and parses quite well. You can use iterparse instead of SAX, it's often more convenient, and often faster.

Answer 2

lxml for sure.

Answer 3

in fact, beautifulsoup was developed for broken html, I don’t know why they dissuade it.

Answer 4

@ur001, I looked at the lxml code again, there are a lot of regexps scattered across modules ( github.com/lxml/lxml/blob/master/src/lxml/html/clean.py#L62 ), but I was completely wrong : html parsing in lxml is based mostly on xml parsing.
Those. lxml thinks html is just invalid xml that can be fixed. From a theoretical point of view, the assumption is incorrect, for parsing html5 and xml, completely different parsers are needed, but in practice it often works.