K
K
kkolorid2019-09-16 19:54:10
Python
kkolorid, 2019-09-16 19:54:10

How to use multithreading in Python for long http requests?

Good day. I am implementing a program in which a loop goes through an array, performs certain actions with data, sends an http request to a third-party api - receives a response, and then writes a new element to another array (conditionally).
The problem is that the http request takes a lot of time against the background of a large amount of data being processed - noticeably, the script runs several times longer than if I did not spend time on these requests. Accordingly, resources are idle at this time. I read that multithreading in Python is a little different and will not give an increase (unless it is specifically broken somehow) and is used only in cases where resources really stand idle. So the question is:
How can such a feature be implemented? So that the script in the loop iteration reaches the place where the request is sent and, without waiting for a response, goes further along the loop. More precisely, waiting, but in the background. Whatever the result, as if "catching up" with the cycle.
It is desirable to have examples of conditional code, or at least in which direction to think, how to organize it all ... Thank you.

Answer the question

In order to leave comments, you need to log in

1 answer(s)
D
Dmitry Shitskov, 2019-09-16
@kkolorid

Dig towards asyncio and/or aiohttp.

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question