Re: Thread Question

Ritesh Raj Sarraf wrote:

I'm planning to implement threads in my application so that multiple
items can be downloaded concurrently. I want the thread option to be
user-defined.

Looking at the documentation of threads (Core Python Programming), I've
noticed that all threads are executed a once. Depending upon what they
are doing, some finish early and some later.

But I want to implement something like:

for item in list_items:
for num in thread_args:
thread[num].start()
thread[num].start()

Is this the correct way of threading applications ?
This is the first time I'd be doing threading. So was looking for
comments and suggestions.


What you want is to use a pool of threads so that you can configure how
many requests are issued at a time (you don't want to try to issue 100
requests all in parallel). You can communicate with the threads through a
Queue.

So if the code for a thread looks like:

def run(request, response):
while 1:
item = request.get()
if item is None:
break
response.put(download_from_web(item))

# your main loop can be something like:

requestQueue = Queue()
responseQueue = Queue()
thread_pool = [
Thread(target=run, args=(requestQueue, responseQueue)
for i in range(numthreads)]
for t in thread_pool: t.start()

for item in list_items:
requestQueue.put(item)

for i in range(len(list_items)):
response = responseQueue.get()
handle_response(response)

# and then to shut down the threads when you've finished:
for t in thread_pool:
requestQueue.put(None)
for t in thread_pool:
t.join()

.

Relevant Pages

  • Re: Thread Question
    ... I added some dummy code to simulate long-running requests, ... from Queue import Queue ... Thread's name into the response Queue. ...
    (comp.lang.python)
  • Re: Is Message passing good approach for multi-threaded applications?
    ... application there are 16 serial ports and 2 Ethernet ports.There are ... passing.There are 2 queues.One for requests and one for responses. ... into response queue. ... requirement.Each request and subsequent response should take only 200 ...
    (comp.os.linux.misc)
  • Re: Thread Question
    ... Thread's name into the response Queue. ...     Get items from the request Queue, ... # Queue up the requests. ...
    (comp.lang.python)
  • Re: Thread Question
    ... going to only download three requests at a time and if one of ... the requests takes a long time it will hold up all the others. ... def run(request, response): ... requestQueue = Queue ...
    (comp.lang.python)
  • [rfc][patch 2.6.18-rc7] block: explicit plugging
    ... This is a patch to perform block device plugging explicitly in the submitting ... the device idle when it is known more requests will be submitted. ... Explicit plugging keeps a process-private queue of requests being held. ... struct rb_root cic_root; ...
    (Linux-Kernel)