S
S
skomoroh2012-05-20 21:08:57
Python
skomoroh, 2012-05-20 21:08:57

pycurl.CurlMulti

Tell me where to dig with CurlMulti.

Such a problem: when you run curl in single-threaded mode, everything works fine and fast

for url in url_list:
    curl = get_curl(url)
    try:
        curl.perform()
        callback_func(curl)
    except pycurl.error, e:
        logging.error("%s:%s" % (e[0],e[1]))

timeouts are worth
    c.setopt(pycurl.CONNECTTIMEOUT, 10)
    c.setopt(pycurl.TIMEOUT, 15)


but on startup
m = pycurl.CurlMulti()
m.handles = []
for url in url_list[:10]:
    c = get_curl(url)
    m.handles.append(c)    
    m.add_handle(c)
ret, num_handles = m.perform()

m.perform() thinks 10-15 minutes and all curl returns with
28: Connection time-out
or
28: Operation timed out after 11201 milliseconds with 0 out of -1 bytes received

Answer the question

In order to leave comments, you need to log in

1 answer(s)
S
skomoroh, 2012-05-20
@skomoroh

Thanks everyone, I solved the problem:
1 took the system libcurl
2 downloaded the source of the latest curl from curl.haxx.se/download.html
3 ./configure --enable-ares --with-ssl --enable-ipv6 --with-libidn
make
sudo checkinstall -D
4 sudo pip install -U --force-reinstall pycurl

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question