Re: Python 2.6's multiprocessing lock not working on second use?

On Mon, Jan 19, 2009 at 8:16 AM, Frédéric Sagnes <speedup@xxxxxxxxx> wrote:
On Jan 19, 11:53 am, Frédéric Sagnes <spee...@xxxxxxxxx> wrote:
On Jan 17, 11:32 am, "Gabriel Genellina" <gagsl-...@xxxxxxxxxxxx>
wrote:



En Fri, 16 Jan 2009 14:41:21 -0200, escribiste en el grupo
gmane.comp.python.general

I ran a few tests on the new Python 2.6multiprocessingmodule before
migrating a threading code, and found out the locking code is not
working well. In this case, a pool of 5 processes is running, each
trying to get the lock and releasing it after waiting 0.2 seconds
(action is repeated twice). It looks like themultiprocessinglock
allows multiple locking after the second pass. Running the exact same
code with threads works correctly.

I've tested your code on Windows and I think the problem is on the Queue
class. If you replace the Queue with some print statements or write to a
log file, the sequence lock/release is OK.
You should file a bug report onhttp://bugs.python.org/

--
Gabriel Genellina

Thanks for your help gabriel, I just tested it without the queue and
it works! I'll file a bug about the queues.

Fred

For those interested, the code that works (well, it always did, but
this shows the real result):

class test_lock_process(object):
def __init__(self, lock):
self.lock = lock
self.read_lock()

def read_lock(self):
for i in xrange(5):
self.lock.acquire()
logging.info('Got lock')
time.sleep(.2)
logging.info('Released lock')
self.lock.release()

if __name__ == "__main__":
logging.basicConfig(format='[%(process)04d@%(relativeCreated)04d] %
(message)s', level=logging.DEBUG)

lock = Lock()

processes = []
for i in xrange(2):
processes.append(Process(target=test_lock_process, args=
(lock,)))

for t in processes:
t.start()

for t in processes:
t.join()

Opened issue #4999 [http://bugs.python.org/issue4999] on the matter,
referencing this thread.


Thanks, I've assigned it to myself. Hopefully I can get a fix put
together soonish, time permitting.
-jesse
.

Relevant Pages

  • Re: synchronizations of threads
    ... To answer your immediate question, it's not the lock using workLock that's superfluous, it's the lock using workQueue. ... As has been discussed elsewhere, including in Jon Skeet's web article on threading, using a privately allocated instance of Object for a lock is preferable to using the object that actually needs the synchronized access, because doing so ensures that only that particular code is locking on that particular object. ... I am surprised to see the non-generic Queue class being used, but more significantly the use of the "stop" variable to signal a shutdown rather than just communicating through the queue, which leads to the two-second timeout, another questionable design choice, and the fact that the workLock lock is held during the entire execution of the worker delegate, these are all bad problems IMHO. ...
    (microsoft.public.dotnet.languages.csharp)
  • Re: Queue object with strange behaviour??
    ... I have been reading Jon's page on threading. ... of the lock) ... , if a queue has built up, it will need to log ... we know that we only Waitif the queue is empty, ...
    (microsoft.public.dotnet.languages.csharp)
  • Python 2.6s multiprocessing lock not working on second use?
    ... I ran a few tests on the new Python 2.6 multiprocessing module before ... and found out the locking code is not ... It looks like the multiprocessing lock ... from multiprocessing import Process, Queue, Lock ...
    (comp.lang.python)
  • Re: Python 2.6s multiprocessing lock not working on second use?
    ... migrating a threading code, and found out the locking code is not ... trying to get the lock and releasing it after waiting 0.2 seconds ... I've tested your code on Windows and I think the problem is on the Queue ...
    (comp.lang.python)
  • Re: Python 2.6s multiprocessing lock not working on second use?
    ... migrating a threading code, and found out the locking code is not ... I just tested it without the queue and ...     lock = Lock ...
    (comp.lang.python)