Re: Producer-consumer threading problem

George Sakkis wrote:
On Jun 10, 11:47 pm, Larry Bates <larry.ba...@xxxxxxxxxxx`> wrote:
I had a little trouble understanding what exact problem it is that you are
trying to solve but I'm pretty sure that you can do it with one of two methods:

Ok, let me try again with a different example: I want to do what can
be easily done with 2.5 Queues using Queue.task_done()/Queue.join()
(see example at http://docs.python.org/lib/QueueObjects.html), but
instead of having to first put all items and then wait until all are
done, get each item as soon as it is done.

1) Write the producer as a generator using yield method that yields a result
every time it is called (something like os.walk does). I guess you could yield
None if there wasn't anything to consume to prevent blocking.

Actually the way items are generated is not part of the problem; it
can be abstracted away as an arbitrary iterable input. As with all
iterables, "there are no more items" is communicated simply by a
StopIteration.

2) Usw somethink like Twisted insted that uses callbacks instead to handle
multiple asynchronous calls to produce. You could have callbacks that don't do
anything if there is nothing to consume (sort of null objects I guess).

Twisted is interesting and very powerful but requires a different way
of thinking about the problem and designing a solution. More to the
point, callbacks often provide a less flexible and simple API than an
iterator that yields results (consumed items). For example, say that
you want to store the results to a dictionary. Using callbacks, you
would have to explicitly synchronize each access to the dictionary
since they may fire independently. OTOH an iterator by definition
yields items sequentially, so the client doesn't have to bother with
synchronization. Note that with "client" I mean the user of an API/
framework/library; the implementation of the library itself may of
course use callbacks under the hood (e.g. to put incoming results to a
Queue) but expose the API as a simple iterator.

George

If you use a queue and the producer/collector are running in different threads you don't have to wait for "to first put all items and then wait until all are
done". Producer can push items on the queue and while the collector asynchronously pops them off.

I'm virtually certain that I read on this forum that dictionary access is atomic if that helps.

-Larry
.

Relevant Pages

  • Re: Producer-consumer threading problem
    ... None if there wasn't anything to consume to prevent blocking. ...  You could have callbacks that don't do ... iterator that yields results. ...
    (comp.lang.python)
  • Re: Producer-consumer threading problem
    ... The Queue class does all ... produce and consume arguments for now--I'll cover that below). ... if item is sentinel: ... notify producer to wrap things up ...
    (comp.lang.python)
  • Re: A community Windows binding
    ... > GUI training vehicle, I don't think you can beat it (unless you ... simple windowing on Windows. ... Callbacks are a pain, but there aren't ... Between callbacks and the single queue of events for all windows of JEWL ...
    (comp.lang.ada)
  • Re: Event-Queuing
    ... Die CallBacks ... Dim Queue as Collection ... Dim Job as CJob ... If Queue Is Nothing Then Set Queue = New Collection ...
    (microsoft.public.de.vb)
  • Re: Synchronizing dialogs from different threads
    ... But the first base function could send a notification before my main ... So I can not synchronize the notification to the UI ... You can emulate the same behavior yourself, but that would involve setting up a queue and a loop where you consume the elements of the queue, and then the thread that was executing that loop would never get around to starting up the UI. ...
    (microsoft.public.dotnet.languages.csharp)