How to get all site documents if the site doesn't have a map?

M

M

Murad Nurmagomedov2015-10-16 15:12:33

Parsing

Murad Nurmagomedov, 2015-10-16 15:12:33

How to get all site documents if the site doesn't have a map?

Hi all.
For educational purposes, I want to write a script that will bypass all site documents and check for internal optimization for serp.
How does a search robot find all the pages? And if the site does not have a sitemap?

Reply

Answer the question

Answer the question

In order to leave comments, you need to log in

2 answer(s)

D

Dimonchik, 2015-10-16
@murad1986

scrapy
or
grablib

E

evnuh, 2015-10-16
@evnuh

by links

Similar questions

B

Bobur Bakhritdinov2016-02-02 19:56:18

PHP parser with support for proxy functions is it real? 3Reply

A

AlbertS0012018-06-17 12:20:03

How does the parser work (example inside)? 2Reply

K

kupurim2018-06-17 17:41:21

What are the libraries for parsing with the ability to execute js? 1Reply

V

Vlad Grinevich2020-06-25 17:15:05

Where can I find a working plugin for captcha puzzle solution in browser automation studio (BAS)? 0Reply

K

Kirill Kiselev2018-06-28 13:29:47

How does the site besposrednika.ru work? 2Reply

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question