S
S
stayHARD2015-10-25 21:43:47
Python
stayHARD, 2015-10-25 21:43:47

Why are extra threads being created at the end of the queue?

Goodnight.
Interested in a question related to Queue and Threading in Python.
I am writing a small multi-threaded application.

from queue import Queue
from threading import Thread
from grab import Grab

def do_work(link):
    g = Grab()
    g.setup(timeout=30, connect_timeout=5)
    g.go(link)
    print(q.qsize())

def worker():
    while True:
        link = q.get()
        do_work(link)
        q.task_done()

q = Queue()
num_worker_threads = 50

for i in range(num_worker_threads):
    t = Thread(target=worker)
    t.daemon = True
    t.start()
list_urls = [
# big list with urls
]
for item in list_urls:
    q.put(item)

q.join()

As a result, in the console I get:
...
38
37
36
35
34
33
32
31
30
29
28
27
26
25
24
23
22
21
20
19
18
17
16
15
14
13
12
11
10
9
8
7
6
5
4
3
2
1
0
0
0
0
0
0
0
0
0
0
0
0
...

At the end, I get such a number of zeros as how many threads are set. And yes, if I have 500 links in my list, then the queue will be 450. Why is that?

Answer the question

In order to leave comments, you need to log in

1 answer(s)
N
nirvimel, 2015-10-25
@nirvimel

From the looks of it, everything is in order.
We have 50 PARALLEL threads that feed from a queue of 500 elements.
1. By the time the first thread finished executing its task, which it spent some time to complete, there were 450 elements left in the queue for a long time, because the threads pick up new tasks instantly and in PARALLEL.
2. By the time the 450th task is completed, the remaining 50 are already in operation, they have already been snapped up by threads, and there are 0 elements left in the queue. After that, the remaining 49 threads finish executing their tasks and each one reports that it observes zero elements in the queue, after which the workers fly out on task_done () and are selected by join in the main thread.

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question