B
B
Bagobor2012-11-16 02:19:53
data mining
Bagobor, 2012-11-16 02:19:53

Ready solutions for web crawler with https/authorization[social]?

Are there ready-made solutions for web crawler with https/authorization[social]?
Ready to consider options for frameworks/scripts/environments/services — enough to get a set of pages/files that match the given templates.
Faced with the task of compiling tables based on data from several sites, data can be obtained after authorization through social networks (Facebook), and the site itself is only accessible via https - the current solution is to enter “by hand” and automate downloading through a browser addon (Chrome \ FF).

Answer the question

In order to leave comments, you need to log in

2 answer(s)
B
Bagobor, 2012-11-16
@Bagobor

The main problem is that at the moment it is not possible to imagine how OpenID authorization works like authorization. And the fact that it was not possible to find crawlers that would explicitly support this.
Of course there is a chance that I'm not looking correctly. I would be grateful for advice.

P
petervolkov, 2016-01-12
@petervolkov

I would take CasperJS (a layer on top of PhantomJs / SlimmerJS)
Everything implemented in a regular browser is easily automated

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question