How do spiders find sites?

M

M

microbot2017-04-18 06:27:40

Crawling

microbot, 2017-04-18 06:27:40

How do spiders find sites?

Are you interested in the technology or script by which web-crawler finds sites? Exactly where does he get the addresses of the sites he visits from? Or a general question: how to find out the names of sites located on the same ip address. Implementation in php is desirable, if there is such.

Reply

Answer the question

Answer the question

In order to leave comments, you need to log in

1 answer(s)

D

Dimonchik, 2017-04-18
@dimonchik2013

spyonweb.com has an API here

Similar questions

B

beginer1232016-09-03 18:47:30

Will Python scrapy work? 2Reply

A

Artem2019-03-17 00:37:10

How to extract JSON object from script tag content? 2Reply

S

skver912017-03-09 13:14:36

How can I get all site page links from google.com? 0Reply

R

ruplace2015-07-10 09:32:09

httrack. How to configure it to download _only_ html pages from a list of sites with a depth of 2-3? 0Reply

K

Kizzeon2020-01-12 06:25:21

Crawlera: 1 request equals one scrambled web page or equals one tag with text? 1Reply

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question