Is it possible to share the load between threads/processes/cores?

N

Nodar2014-07-11 20:59:23

linux

Nodar, 2014-07-11 20:59:23

Hello.
There was a question of this nature:
Let's say there is some process that requires a lot of resources for implementation.
Is it possible to distribute its work between / threads / processes / cores? That is, to speed up the execution of the program. For example, we are archiving a large file, is it possible to speed up the archiving process by distributing the load?
Googling about this, I came across Parallel processing. This is it?

Reply

Answer the question

In order to leave comments, you need to log in

6 answer(s)

R

radioxoma, 2014-07-11
@radioxoma

Python in its purest form will never be the fastest. If you want fast, use compiled languages. Want even faster? Use the GPU. And yet, python is quite handy for creating a high-level interface and controlling modules in other languages.
As for CPU-demanding tasks (I emphasize, not input / output ), this is a rather painful topic. Due to the GIL, threads will not run on different cores at the same time, but this can be bypassed. So the options are:
Here's a great article on said methods. Of course, all of the above makes sense if the problem can be parallelized.

C

Calc, 2014-07-11
@Calc

Also look in the direction of "divide and conquer" algorithms.
If a task can be divided into several independent subtasks, then they (subtasks) can be run in separate threads.
Also read towards forkjoin algorithms

Y

yttrium, 2014-07-11
@yttrium

if you use a third-party library .. then most likely everything is already parallelized there. if your own, then study threading
to start

I

Igor, 2014-07-11
@merryjane

It can be done using cgroup so that the application does not take away resources or their distribution is in a certain ratio, which will allow the application to work faster.

E

Eugene, 2014-07-11
@Jekel

Python can't parallelize in cores.
it is necessary to run a number of processes equal to the number of cores and drive data between them for processing.

A

Alexander Borisovich, 2014-07-12
@Alexufo

Under Windows I get out so
processhacker.sourceforge.net
ON the server there is a service which is engaged in processing. Server two cores.
If you do not set a priority for a service, processing is slow, the web server is stupid, because everyone has the same priorities.
Made affinity (affinity core) process with processing only the first core + realtime priority, assigned only the second core to the web server. It seems to be normal)
For good, you need to take out the load on another web server and its CPU.