Queue can result in nested monitor deadlock

I think there's a slight design flaw in the Queue class that makes it
hard to avoid nested monitor deadlock. The problem is that the mutex
used by the Queue is not easy to change. You can then easily get
yourself into the following situation (nested monitor deadlock):

Say we have a class that contains a Queue and some other things. The
class's internals are protected by a mutex M. Initially the Queue is
empty. The only access to the queue is through this class.

Thread 1 locks M, then calls Queue.get(). It blocks. At this point the
Queue's mutex is released, but M is still held.

Thread 2 tries to put something in the Queue by calling the enclosing
class's methods, but it blocks on M. Deadlock.

The solution would be to expose the Queue's mutex via a constructor
argument and/or method, so that, to take the above example, one could
create M and give it to the Queue as its mutex. Of course this is
doable now, as the code below demonstrates. But it should be documented
and exposed.

As I'm new to the Python community, I'm not sure that this is the right
forum for this suggestion. Is it the sort of thing one would put on the
SourceForge bug list? Advice appreciated.

"Fixed" Queue class:

class LQueue(Queue.Queue):
"""
Fixes a problem with the standard Queue implementation: you can't
use your own mutex,
so you are subject to nested monitor deadlock. This version lets
you pass in your
own lock.
"""

def __init__(self, maxsize=0, lock=None):
"Optionally takes a lock (threading.Lock or threading.RLock) to
use for the queue's lock."
Queue.Queue.__init__(self, maxsize)
if lock:
self.mutex = lock
# Re-create the condition objects with the new mutex.
self.not_empty = threading.Condition(self.mutex)
self.not_full = threading.Condition(self.mutex)

.

Relevant Pages

  • ktr/alq/witness failure.
    ... ref4# sysctl debug.ktr.mask=0x2fffffff ... 14 vm page queue mutex -- ... ATA disk bioqueue lock -- ...
    (freebsd-current)
  • Re: Recursive mutex that can be waited upon (pthread)
    ... While you can use a mutex to avoid that data is changed, for me having a mutex does not mean that data is not changed, it only means that data is not changed by a different thread. ... My own thread may of course change the data, hence functions I call may want to change the data and if they do so, they must be sure that these changes are atomically, hence they must lock the object and they simply can't rely that I locked the object before -> thus I need recursive locks. ... Then I could as well throw out threads of my code and just use a single thread going through an event queue. ... you have a predicate condition on an invariant. ...
    (comp.programming.threads)
  • Re: IOCP critical sections and mutexes
    ... those are pulled from the queue via GetQueuedCompletionStatus ... BTW, for the record, both the user mode accessible mutex and critical ... My packets never go over 2k in size. ... then pushes it in a queue that will be later processed by my main app ...
    (microsoft.public.win32.programmer.kernel)
  • Re: Multithreaded queue with wait event
    ... if two threads release the mutex and are just about to ... See, if that happens, one thread will receive the event and dequeue. ... your queue can indeed return no item when there is no timeout ... I have two implementations, one using the semaphore, the other using ...
    (comp.programming.threads)
  • Re: crash via vm_page_sleep_if_busy() and contigmalloc
    ... and unlock its mutex. ... > page queue mutex was relinquished. ... I think callers need to either lock the vm_object in every case ...
    (freebsd-hackers)