Why can't I create a moderate number of threads in python?

Question

Python

There is a code that creates 900 threads to download a file. But I am getting this error

Traceback (most recent call last):
  File "C:/Users/bb/Desktop/wp/Новая папка/upload.py", line 132, in <module>
    thread.start()
  File "C:\Users\bb\AppData\Local\Programs\Python\Python35-32\lib\threading.py", line 844, in start
    _start_new_thread(self._bootstrap, ())
RuntimeError: can't start new thread

Although there is enough free memory. Why is this happening and how to solve it?

Answer 1

Depending on which operating system you are using, the OS kernel allocates a fixed stack size for each of the processes. Usually this is not a very large value, and so for example in my Ubuntu it is 8 MB.
It seems to me that you simply run out of address space (since there is still a lot of metadata on the stack), and therefore the interpreter forbids you to create additional threads. If you are confident in what you are doing and you really need 900 threads, you can control this process manually:

>>> import threading
>>> threading.stack_size()
0
>>> threading.stack_size(64*1024) # 64 KiB

And in the case of Python and crawlers, if the crawler itself is small enough (which is probably true, depending on your code), most of the time you will spend roundtrip to the server and back for HTML content. Those. 80-99% of the time the processor will be idle, waiting for a response from the network driver. In such cases, I use asynchronous calls and get a performance increase of 4-8 times compared to a synchronous program and 2 times - compared to a multi-threaded one.