S
S
stayHARD2015-09-29 14:57:23
Python
stayHARD, 2015-09-29 14:57:23

Multi-threaded writing to the database. How?

Good afternoon, %username% .
I'm trying to get the script parser to work multi-threaded.
Now it works sequentially:
1. Selects links to sites from the database.
2. Cycle through each link by calling the get_info()
function 3. The get_info() function searches the site for the information I need and at the end makes an SQL query to UPDATE the record with this link.
How to speed it all up?
I tried a bunch of examples using thread, threading, multithreading, multiprocessing modules, everything only aggravates the situation (processing takes longer (!) Than sequentially).
It was not possible to find a suitable and good example, please poke your nose where to look, because it turns out that I'm missing something. Otherwise, why all this bunch of modules that work longer :)
Many thanks in advance.

Answer the question

In order to leave comments, you need to log in

1 answer(s)
A
Andrew K, 2015-09-29
@mututunus

https://github.com/aio-libs/aiopg

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question