Python: Stuck streams with HTTP requests

Question

Python

Curl multi + Threading
Total 10 connections. Successfully managed to carry out first 70,000 requests, then 300,000. At some point, everything stops without error messages. Connection timeouts are set. There is a process instance in the Windows Task Manager, but the script just freezes without any sign of a problem. What to read?

Answer 1

it can be anything, from channel sagging, to memory junk and your IP ban ... you have to look, it's
generally more convenient to debug such things through this library: code.google.com/p/workerpool/

Answer 2

I faced the same problem, I solved it trivially by setting a timeout for unconditional nailing the thread. I didn’t delve into the root causes in detail, sometimes it seems that the socket just falls off, and the library wants to read something from it.