How to crawl a wget site for external links?

D

De YURII2016-01-27 16:03:07

Wget

De YURII, 2016-01-27 16:03:07

There is a need to crawl about 1000 pages for external links on the page, where to dig and is wget suitable for this?

Reply

Answer the question

In order to leave comments, you need to log in

1 answer(s)

D

Dimonchik, 2016-01-27
@dimonchik2013

in general, take a look at this
home.snafu.de/tilman/xenulink.html
as for wget - configure mirroring, download only HTML pages and the ability to follow external links - then on the local you will see which domain folders it will create
this is the best dutn dock dbserv.pnpi.spb.ru/~shevel/Book/node103.html