How to get all links from a site?

Question

PHP

There is a site, I need to parse (grab) all the links. But only the first 10 are shown on the page, and the remaining 10 pieces are loaded by clicking on the button. There are a little more than 600,000 links.. =)
Question: if what scripts/programs to get all the links?

Answer 1

Simple Html Dom
An example of using it to parse all links from google.com
$html = file_get_html(' www.google.com ');
echo $html->find('a');
If you only need a link (href), then you can specify the attribute
echo $html->find('a')->href;

Answer 2

How to parse dynamic web sites using Python 3?
How to parse ajax web page?
What is the best way to parse the results of an ajax request?
How to access Ajax content while parsing?
How to parse dynamic content where different request data (AJAX POST)?

Answer 3

See the request that is sent by ajax. And then do it yourself.