B
B
Bjornie2016-11-21 15:02:21
PHP
Bjornie, 2016-11-21 15:02:21

Use phantomjs or loop through a million pages?

I want to save data from a site that contains elements that do not have a direct address in the attributes, for example, pagers, the "More" button (popup). Just cURL and htmlsimpledomparser won't be enough to simulate a click. At the same time, I know the addresses of the final pages
that look like /index.php?id=1362
there is a valid html-title (unfortunately all pages return 200 code, and there is no other way to distinguish them).
Do you think it's a good idea not to be smart with phantomjs and go through a million pages like a tank?
I will run the script from the desktop from the console.

Answer the question

In order to leave comments, you need to log in

1 answer(s)
_
_ _, 2016-11-21
@AMar4enko

Well, if you are not banned on the tenth request, then why not?

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question