Re: Producer-consumer threading problem

Sounds like a sentinel would work for this. The producer puts a
specific object (say, None) in the queue and the consumer checks for
this object and stops consuming when it sees it. But that seems so
obvious I suspect there's something else up.

There's a decent implementation of this in the Python Cookbook,
Second Edition (9.4: Working with a Thread Pool), available from
Safari as a preview:
http://my.safaribooksonline.com/0596007973/pythoncook2-CHP-9-SECT-4

Basically, there's a request_work function that adds (command,
data) pairs to the input Queue. The command 'stop' is used to
terminate each worker thread (there's the sentinel).
stop_and_free_thread_pool() just puts N ('stop', None) pairs and
join()s each thread.

The threadpool put()s the consumed items in an output Queue; they
can be retrieved concurrently using get(). You don't call join()
until you want to stop producing; you can get() at any time.

Geoff Gilmour-Taylor

(I ended up using this recipe in my own code, but with a completely
different stopping mechanism---I'm using the worker threads to control
subprocesses; I want to terminate the subprocesses but keep the worker
threads running---and a callback rather than an output queue.)
.

Relevant Pages

  • Re: ThreadPool Questions
    ... If you are running this code on a computer with only one CPU, and your code is truly CPU-bound, then more threads are only going to make things worse. ... You _should_ see a benefit even if you queue one worker thread item per record, but your overhead will start to reduce your efficiency in a noticeable way. ... There is a slight inefficiency in that the consumer thread will occasionally have to wait at an unprocessed record even though another record later in the sequence has already been done, but assuming that the processing itself is really the bottleneck, this shouldn't really be a problem. ...
    (microsoft.public.dotnet.languages.csharp)
  • Re: Is this group dying?
    ... queue which are registered with a single consumer. ... The producer uses this index to ... know what consumer to register its spsc queue with. ... interlocks in the producer _and_ in the consumer. ...
    (comp.programming.threads)
  • Re: Is this group dying?
    ... consumer queue toward the dispatcher thread and a single-producer/ ... that producer has put a new "next" node in place. ... I want to implement the wait-free factory as a multi-threaded daemon process ...
    (comp.programming.threads)
  • Re: Thread question
    ... is the number of externally-caused awakenings. ... queue holding the items produced but not yet consumed. ... If the loop above is part of the consumer, ... Thing from the queue supplied by the producer. ...
    (comp.lang.java.programmer)
  • Re: Producer/Consumer
    ... but the consumer is only prepared to dequeue one at a time. ... even though there remain items in the queue (because the number of thread ... _newItemEvent is set by the producer after enqueueing an item, ...
    (microsoft.public.dotnet.general)