Is it possible to pull all external links from a website using Python?

M

Mark Adams2016-10-01 23:31:44

Python

Mark Adams, 2016-10-01 23:31:44

What tools in Python can quickly pull out all external links from all pages?

Reply

Answer the question

In order to leave comments, you need to log in

2 answer(s)

R

Rou1997, 2016-10-02
@Rou1997

If it’s everything from any site, then you need an embedded browser, that is, Selenium or Phantom.js, because it’s 2016, and JavaScript still exists, and there is Flash, which still exists, and with it's even more difficult for him.
Well, if there are enough simple ones <a href>, and only those that are not loaded by AJAX, then you can use RoboBrowser (BeautifulSoup), etc., that is, libraries that only work with DOM (HTML).