How to get all site documents if the site doesn't have a map?

M

M

Murad Nurmagomedov2015-10-16 15:12:33

Parsing

Murad Nurmagomedov, 2015-10-16 15:12:33

How to get all site documents if the site doesn't have a map?

Hi all.
For educational purposes, I want to write a script that will bypass all site documents and check for internal optimization for serp.
How does a search robot find all the pages? And if the site does not have a sitemap?

Reply

Answer the question

Answer the question

In order to leave comments, you need to log in

2 answer(s)

D

Dimonchik, 2015-10-16
@murad1986

scrapy
or
grablib

E

evnuh, 2015-10-16
@evnuh

by links

Similar questions

D

Dmitriy_pythonshik2020-06-21 16:39:38

How to click on several buttons with different classes on a website? 1Reply

U

uzi_no_uzi2021-09-14 22:11:00

How to wait for YouTube selenium + phantom.js (c#) to load? 1Reply

A

aqwe1092016-01-31 17:01:22

How to parse a site if two encodings iso-8859-1 and utf-8 are specified? 2Reply

D

dabiankolins12018-06-14 19:17:51

Web scraping, parsing in Python? 5Reply

S

Sergey Afanasiev2020-06-25 09:07:39

How to parse sites like saavefromnet or Getvideo? 1Reply

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question