Re: Thread Question

Ritesh Raj Sarraf wrote:
Duncan,

I couldn't make out much from the code.

Please, try again to understand Duncan's code. It's much better than
what you did.

Instead this is what I did.

threads = []
nloops = range(len(lRawData))
for i in nloops:
(sUrl, sFile, download_size, checksum) =
stripper(lRawData[i])
t = threading.Thread(target=download_from_web, args=(sUrl,
sFile, sSourceDir, None))
# = pypt_thread(download_from_web, i,
stripper(lRawData[i]))
threads.append(t)

i = 0
join_i = 0
while i < nloops:
counter = 0
while counter < 3:
threads[i].start()
counter += 1
i += 1
counter = 0
join_i = i - 3
while counter < 3:
threads[join_i].join()
counter += 1
join_i += 1

Is this correct ? Comments!!

This is just painful. It's not exactly "incorrect", but I think (I
*think*, it's hard to tell even after reading it 3 times) that it's
going to only download three requests at a time and if one (or more) of
the requests takes a long time it will hold up all the others. Also,
you're creating one thread per item in lRawData even though you only
need/use three at a time.

if you set numthreads = 3 in Duncan's code, you would only be
downloading 3 things at a time, but you'd only be using three threads
and long running requests would not affect the others.

If you need help understanding it please ask questions. I, for one,
would be happy to comment it for you to explain how it works. It's so
nice and elegant that I've already cut-and-pasted it into my own
"notebook" of cool useful python "patterns" to use in the future.

Reread it slowly, think about what it's doing, if questions arise write
them down and ask them. Duncan's code is beautiful. It's well worth
your time to understand it.

Peace,
~Simon


Ritesh


Duncan Booth wrote:
Ritesh Raj Sarraf wrote:

I'm planning to implement threads in my application so that multiple
items can be downloaded concurrently. I want the thread option to be
user-defined.

Looking at the documentation of threads (Core Python Programming), I've
noticed that all threads are executed a once. Depending upon what they
are doing, some finish early and some later.

But I want to implement something like:

for item in list_items:
for num in thread_args:
thread[num].start()
thread[num].start()

Is this the correct way of threading applications ?
This is the first time I'd be doing threading. So was looking for
comments and suggestions.


What you want is to use a pool of threads so that you can configure how
many requests are issued at a time (you don't want to try to issue 100
requests all in parallel). You can communicate with the threads through a
Queue.

So if the code for a thread looks like:

def run(request, response):
while 1:
item = request.get()
if item is None:
break
response.put(download_from_web(item))

# your main loop can be something like:

requestQueue = Queue()
responseQueue = Queue()
thread_pool = [
Thread(target=run, args=(requestQueue, responseQueue)
for i in range(numthreads)]
for t in thread_pool: t.start()

for item in list_items:
requestQueue.put(item)

for i in range(len(list_items)):
response = responseQueue.get()
handle_response(response)

# and then to shut down the threads when you've finished:
for t in thread_pool:
requestQueue.put(None)
for t in thread_pool:
t.join()

.

Relevant Pages

  • Re: Thread Question
    ... I added some dummy code to simulate long-running requests, ... from Queue import Queue ... Thread's name into the response Queue. ...
    (comp.lang.python)
  • Re: Is Message passing good approach for multi-threaded applications?
    ... application there are 16 serial ports and 2 Ethernet ports.There are ... passing.There are 2 queues.One for requests and one for responses. ... into response queue. ... requirement.Each request and subsequent response should take only 200 ...
    (comp.os.linux.misc)
  • Re: Thread Question
    ... Thread's name into the response Queue. ...     Get items from the request Queue, ... # Queue up the requests. ...
    (comp.lang.python)
  • Re: Thread Question
    ... This is the first time I'd be doing threading. ... requests all in parallel). ... def run(request, response): ... requestQueue = Queue ...
    (comp.lang.python)
  • Start download in separate Response
    ... in my ASP.NET App, I have a list of files which the user can download ... the response is sending the file and so no new requests can be made. ...
    (microsoft.public.dotnet.framework.aspnet)