How to deal with multi-core processors in Python?

Question

Python

Actually, do I understand correctly that this is generally impossible, since due to the GIL in Python, all threads will still be done sequentially by one processor? Is this relevant for Python 3.4? How to get around (write code in C)?

Answer 1

https://docs.python.org/2/library/multiprocessing.html is the only way I know of to recycle multiple cores in python. GIL is in both branches of python, there is no way around it (like, if I'm not mistaken, writing an extension in C, you will still be forced to run it under the same conditions).

Answer 2

from time import time
from threading import Thread
from multiprocessing import Process
def count(n):
while n > 0:
n -= 1
startTime = time()
count(100000000)
count(100000000)
print('\nSequential execution time : %3.2 f s.'%(time() - startTime))
startTime = time()
t1 = Thread(target=count, args=(100000000,))
t2 = Thread(target=count, args=(100000000,))
t1. start(); t2.start()
t1.join(); t2.join()
print('\nThreaded execution time : %3.2f s.'%(time() - startTime))
startTime = time()
p1 = Process(target=count, args=(100000000,))
p2 = Process(target=count, args=(100000000,))
p1.start(); p2.start()
p1.join(); p2.join()
print('\nMultiprocessed execution time : %3.2f s.'%(time() - startTime))
Gives on a 4-core process:
Sequential execution time : 6.83 s.
Threaded execution time : 11.37 s.
Multiprocessed execution time : 6.30 s.
But let's say parallelization of requests to the http server and in the thread version will give a huge gain.
Those. without taking into account the specifics of the task - in multithreading / multiprocessing - it's better not to meddle.

Answer 3

kazmiruk is absolutely right. We write a project in python2, use multiprocessing (not threading!) and everything works fine.

Answer 4

I watched a video from a conference on Python. In general: GIL only affects Python code, i.e. if Python acts only as a wrapper, and resource-intensive operations are done in C, then the C code will be distributed by the operating system to different processors and, as a result, a performance gain will be obtained.

Answer 5

GIL is not in Jython and IronPython, you can dig in their direction.

Answer 6

In general, I join the previous answers - because of the GIL, there are only a few processes in pure python. If you either write extensions or use ready-made libraries (for example, NumPy), then you can release the GIL inside the extension code.
By the way, as a rule, NumPy compiled with MKL will load all the processor cores on its tasks;)
If you need more advice, it's better to briefly describe what the task is.

Answer 7

There is a fork of Python with support for multi-core architecture. It seems even possible to merge with one of the future versions of Python. I just can't remember the name right off the bat. It doesn't google at all.
There is also PyPy-STM, a Python interpreter with support for multi-core systems:
PyPy-STM (PyPy Software Transactional Memory), which develops an implementation of the Python language that can parallelize the execution of different threads of one multi-threaded application on several CPU cores. The development of PyPy-STM is aimed at eliminating one of the main problems of Python - the presence of a global interpreter lock (GIL, global interpreter lock), which does not allow parallel execution of several threads of Python code.