M
M
matveyvarg2016-01-28 20:11:54
Server optimization
matveyvarg, 2016-01-28 20:11:54

What's the fastest way to save images by URL?

There is an array with links to pictures. It is necessary to shove them into the archive as quickly as possible.
At the moment it is done like this:

for item in datas:
        iitem = item.replace('\\','')
        resource = urllib2.urlopen(item)
        out = open('img+'+str(iterat)+'.jpg', 'wb+')
        out.write(resource.read())
        out.close()
        zf.write('img+'+str(iterat)+'.jpg')
        os.remove('img+'+str(iterat)+'.jpg')
        iterat+=1
        zf.close

Reworked via socket, the gain is quite small.
What is the best way to speed up the process in this case?

Answer the question

In order to leave comments, you need to log in

2 answer(s)
D
Dimonchik, 2016-01-28
@dimonchik2013

here
https://habrahabr.ru/post/275595/
start with
from multiprocessing import Pool

R
Roman Mirilaczvili, 2016-01-28
@2ord

You can speed up by downloading in multiple streams.

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question