J
J
Joysi2021-12-29 11:47:19
Python
Joysi, 2021-12-29 11:47:19

How to effectively use multi-core/multithreading in Jupyter Notebook?

Good afternoon! I write and run small script requests in Jupyter-Notebook, all are different:
- API calls over the network
- algorithms (combining array elements + evaluation and choosing the best combination
- search in files
- mathematical calculations (Numpy)
If it is possible to speed up some BLAS calculations on a video card, then the algorithms are already harder. But when running heavy CPU tasks, it is often seen that the load of a modern multi-core processor (I have an i5-10400) is far from optimal. Is there a version of Notebook that allows you to run MULTIPLE cells in parallel (well, and scatter them over the cores, etc. to optimize by process). I understand that it is possible to run several Jupyter servers + several notebooks in parallel. But this is inconvenient, and it makes it impossible for multiple running cells to share common data. I understand about the collisions of sharing the same data from different threads, but I need read-only data, ideally it would be:
1) Got the data from outside
2) Preprocessing
3) 4-5 cells were launched in parallel (which take data from the previous paragraph that will not change in the future as input).

If Jupyter can't do this, maybe another PL + its notebook can (Go, Kotlin ... ) ? Or is there something else?

Answer the question

In order to leave comments, you need to log in

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question