Z
Z
zlodiak2019-05-24 11:58:51
Python
zlodiak, 2019-05-24 11:58:51

Why are queries executed sequentially?

There is a class that parses some content. I initialize it with a specific link to the page, it works and outputs the result to the console.
since there are a lot of links to pages, I tried to parallelize this process using a loop:

from multiprocessing import Process

for link in links:
    img_recognizer = ImgRecognizer(someArgs)

    proc = Process(target=img_recognizer.get_phone())
    procs.append(proc)
    proc.start()

for proc in procs:
    proc.join()

Please tell me why the following code does not execute requests in parallel, but sequentially. This is expressed in the fact that after starting the program, selenium opens the browser and parses the content (this is visible to me as a user). As soon as the parse is completed, selenium closes the browser window and opens a new one. The same is repeated further.
Using the multiprocessing package, I expected that several windows would open almost simultaneously. However, this did not happen and I do not understand why
UPD: at the same time, if I do the same through the threading package, then the described problem does not arise

Answer the question

In order to leave comments, you need to log in

1 answer(s)
I
Ivan, 2019-05-24
@Kurku

Read documentation:

>>> help(Process.join)
join(self, timeout=None)
    Wait until child process terminates

This means that the interpreter will wait until the process ends.
That is, in your code, you actually do everything sequentially.
And you need to write something like this:
from multiprocessing import Process
from time import sleep
     
def worker(n):
    print("the worker went to the office")  
    sleep(n)
    print("the worker has been walking for %d sec" % n)
     
if __name__ == "__main__":
    plist = [] 
    for i in range(4, 0, -1):
        proc = Process(target=worker, args=(i,))
        plist.append(proc)
     
    for proc in plist:
        proc.start()

This will return:
the worker went to the office
the worker went to the office
the worker went to the office
the worker went to the office
the worker has been walking for 1 sec
the worker has been walking for 2 sec
the worker has been walking for 3 sec
the worker has been walking for 4 sec

And yes, processes and threads are completely different things. Streams are much more convenient to work with. So why not use them.

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question