How to download tens of thousands of pages from a list?

Question

Parsing

Hello!
There was a task to download many pages from the site, simply parse, there is a list of pages, but not one download manager came up (too many and they don’t know how to download from a file with a list).
I have been Google for more than a week and it seems that this topic is of little interest to anyone, or is deliberately silent, I also do not exclude that I am looking in the wrong place and in the wrong way ...
Prompt a program for downloading files from the list (win or osx) with a graphical interface.
Also, a question for experienced parsers, if I download tens of thousands of pages from one site, does it (the site) somehow display it, do I need a proxy?
Thank you all in advance!

Answer 1

Download Master can do it for sure.
As for the proxy - if the admin looks at the logs, and is surprised that from one address dofigischa pages "viewed" - he will already ask himself a question. Well, in any analysts can see. But while he sees it, you will already have another dynamic IP. And even if it's the same - so what? Are you afraid of being banned? Then put a proxy - who knows what kind of site, and what kind of admin is there, and what exactly he monitors.

Answer 2

bash/sh
wget
curl

Answer 3

Content downloader
Naturally displayed, but not everyone is watching.
If no proxy protection is needed.

Answer 4

winwget.sourceforge.net