Answer the question
In order to leave comments, you need to log in
How to effectively use multi-core/multithreading in Jupyter Notebook?
Good afternoon! I write and run small script requests in Jupyter-Notebook, all are different:
- API calls over the network
- algorithms (combining array elements + evaluation and choosing the best combination
- search in files
- mathematical calculations (Numpy)
If it is possible to speed up some BLAS calculations on a video card, then the algorithms are already harder. But when running heavy CPU tasks, it is often seen that the load of a modern multi-core processor (I have an i5-10400) is far from optimal. Is there a version of Notebook that allows you to run MULTIPLE cells in parallel (well, and scatter them over the cores, etc. to optimize by process). I understand that it is possible to run several Jupyter servers + several notebooks in parallel. But this is inconvenient, and it makes it impossible for multiple running cells to share common data. I understand about the collisions of sharing the same data from different threads, but I need read-only data, ideally it would be:
1) Got the data from outside
2) Preprocessing
3) 4-5 cells were launched in parallel (which take data from the previous paragraph that will not change in the future as input).
If Jupyter can't do this, maybe another PL + its notebook can (Go, Kotlin ... ) ? Or is there something else?
Answer the question
In order to leave comments, you need to log in
Didn't find what you were looking for?
Ask your questionAsk a Question
731 491 924 answers to any question