Will multithreading save me?

Question

Django

Good afternoon everyone.
I am writing one application in Django and Grab'e. It sends a LOT of requests to servers on the internet and stores the parsed response data in a database. The original data is updated almost continuously on the servers. Now, on the local server, the entire action is launched at my command, but in production I want to make it a continuous process (I uploaded the answer, parsed it in the background, saved it to the database, went to load and parse again - no one noticed).
The question is
: the number of http requests outgoing from me just rolls over, the speed of a single-threaded script is not very encouraging.
I would like to know if there is any way to significantly speed up my script? I'm not very good at streams, but I thought that maybe there is a way to delegate some of my requests to them ... After all, logically, those servers on the Internet where I send so many requests receive them hundreds of times more from different users, and manage to answer everything. Yes, they have better hardware and a wider communication channel, but is that the only thing?
PS If you, perhaps, did not understand my question or think that this topic cannot be covered within the framework of the Toaster, advise a book that explains how to build such complex web services.

Answer 1

a lot of threads is good, but with an increase in the number of threads, performance will fall, that is, 8 threads will not work twice as fast as 4.
In fact, most of your time is networking, which is slow. The output is an event loop. Within the framework of one thread, we launch many many non-blocking requests, and as the requests complete, we process the result.
The combination of several threads (by the number of processor cores) and an event loop will give the maximum utilization of the processor.

Answer 2

I'm not very good with threads, but I thought there might be a way to delegate some of my requests to them...

Many threads in Python are a sham ... due to GIT.
In Python, execution in multiple threads does not speed up, but slows down the final execution of the program ... no matter what the smart people are blowing here.

Answer 3

I would like to know if there is any way to significantly speed up my script?

Try using aiohttp for network requests . This is a library based on the event cycle that Sergey Protko wrote about . Whether the acceleration will be radical I do not know, but the fact that it will be almost 100% sure.

Answer 4

Solved the problem by rewriting everything with Grab::Spyder.
Works great, judging by the time measurements.