K
K
kuzmenko_alex2014-08-16 02:47:51
Python
kuzmenko_alex, 2014-08-16 02:47:51

What is suitable for my task: multithreading or asynchrony?

Good afternoon!
It is necessary to perform the following actions: create an http session, take links from the database, send a certain number of requests to them, write the answers back to the database. There should be a lot of such sessions and it takes a very long time to process them one by one. We need to speed this up somehow.
The first thing that comes to mind is to use streams. I would spawn many threads, each would enter its session, send a couple of requests, write the result to the database (all threads should work correctly from one database) and end. At least that's the clear way for me.
Now on the Internet I often see references to asyncio and any kind of asynchrony, and I wondered if it was possible to solve my problem using asynchrony.
And so, what is better for solving my problem: parallel execution (threads, forks, multiprocessing) or asynchrony (asyncio, aiohttp, twister, tornado)?
I would be very grateful for any advice!

Answer the question

In order to leave comments, you need to log in

3 answer(s)
_
_ _, 2014-08-16
@AMar4enko

It is better to use streams, it will be easier for you to debug them.

R
Rrooom, 2014-08-16
@Rrooom

You have a question. You expect smart answers to it.
Or maybe it would be better to implement one option first, and then another? What is the final thing to understand, what is easier and better for your task?
The main job is sending http requests? Take twisted or async http client tornadoes - it will be easier and faster.

M
Matvey Kukuy, 2014-08-16
@Matvey-Kuk

The main thing when choosing parallelism is not to forget about habrahabr.ru/post/84629

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question