F
F
frendri2020-08-09 19:01:28
Python
frendri, 2020-08-09 19:01:28

How to make requests in a multithread with a certain period of time?

Good afternoon. I ran into a problem, googled, but did not find a solution to my particular problem.
You need to write a script so that it makes requests every certain period of time and then, when the answer comes, it adds the request response to the array. Let's say I have 10 sessions. And the required time interval between requests to me: 0.2 seconds. So I need 1 of the sessions to send a request every 0.2 seconds (If no response was received, and 0.2 seconds had already passed, a new request for another session began, but the answer in this thread was still expected). And as a result, the script sends 10 requests in 2 seconds from different sessions and then the cycle starts again
Read it here: Python multithreading Requests, what blocks processing?, but as I understand it, again, here, until the response to the request comes, a new request will not start.
I tried like this:

def multi_parse(self):
    self.ticker = 1
    self.parse_info = []
    with concurrent.futures.ThreadPoolExecutor(max_workers=1) as executor:
      executor.map(self.thread_parse, self.multisession)

  def thread_parse(self, session):
    print(f'Делаю запрос #{self.ticker} {t.time()}')
    thread_local.session = session
    multi_session = thread_local.session
    req = multi_session.get("", timeout=5).json()
    
    self.parse_info.append(req)
    t.sleep(0.2)
    print(f'Завершил запрос #{self.ticker}')
    self.ticker += 1

But here all requests are sent at the same time.

Answer the question

In order to leave comments, you need to log in

1 answer(s)
S
Sergey Tikhonov, 2020-08-10
@tumbler

I would do this: I would
create a pool of queues (10 pieces), for each I would launch a thread that would wait for a message from the queue and run the desired request. And in the main thread, once every 0.2 seconds, I would fire 1 request into one queue in a cycle, at each iteration moving on to the next queue. In total, all requests are coordinated through a common "writer", and "readers" process responses each in their own thread.

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question