How to quickly upload a large number of small images to a debian server?

Question

linux

Hello, I
have a remote VPS with Debian on board. And you need to periodically download a large number of small pictures by their URL with renaming, then compress. And the problem is like garbage, but the question is speed.
What are the options? I will be glad to any links.
At first I thought to use "wget ​​-i pupkinlist.txt", but it is impossible to specify the final name of each file this way.
Now I'm doing it probably by the lamer method. I get a list of files in php and run exec('wget.. then exec(' convert.. so several php cli instances (hoping that they work in parallel and do not interfere with each other)..
Actually, I really want to know a better option?
(plus there is an adjacent a task in the same vein - a mass check for the existence of files in a folder, such as file_exists).
Maybe somehow you need to combine many execs into one with a bunch of arguments? How to do it right and will it speed up the process? Feel? that you need to use console commands, but I don’t know how to parallelize them correctly and make sure that there are not too many of them.
What do you advise?

Answer 1

$ awk '{ ext=gensub(/.*(\.[^.]*$)/,"\\1",1);
         print "wget " $2 " -O " $1 ext}' images.txt

wget image.jpg -O some_id1.jpg
wget image2.jpg -O some_id2.jpg
wget image3.jpg -O some_id3.jpg

Answer 2

If you are already using PHP, then I would try to do this:
1) read a bunch of URLs from the list (10,20 ... )
2) initialize via php.net/manual/ru/function.curl-multi-init.php and start the threads
3) wait until all the threads complete their work
4) rename the files
5) compress them
6) repeat step 1

Answer 3

Put in a file or slip directly a list of urls and do something like:
Where P is the number of wget threads.
You can add some variable that would name the files in the way you need (wget's -O option).
PS - you can check the existence of a file by reference with a construction like:

if ; then
...

Answer 4

Of course, I'm a web developer and not an admin, and therefore the advice will be regarding the web, but I would use sockets and transfer a pack at once with one stream.

Answer 5

yes, finally easy, tell the whole algorithm in detail, why download is there access, what for the file? updated? etc.

Answer 6

rsync