Re: Get multiprocessing.Queue to do priorities

The Queue module, apparently, is thread safe, but *not* process safe. If you try to use an ordinary Queue, it appears inaccessible to the worker process. (Which, after all, is quite logical, since methods for moving items between the threads of the same process are quite different from inter-process communication.) It appears that creating a manager that holds a shared queue might be an option (http://stackoverflow.com/questions/342556/python-2-6-multiprocessing-queue-compatible-with-threads).

Just

for illustration: This shows that Queue.Queue doesn't work with processes:

------------------------
def worker(queue):
while True:
item = queue.get()
print item
queue.task_done()

queue_queue = Queue.Queue()

worker_thread = multiprocessing.Process(target=worker, args=(queue_queue,))
worker_thread.start()
for i in range(10):
queue_queue.put(str(i))
time.sleep(10)
while True:
try:
print 'still on queue: ' + queue_queue.get(False)
except Queue.Empty:
break
worker_thread.join()
------------------------

This yields:

still on queue: 0
still on queue: 1
still on queue: 2
still on queue: 3
still on queue: 4
still on queue: 5
still on queue: 6
still on queue: 7
still on queue: 8
still on queue: 9

So no queue item ever arrives at the worker process.



On 2009-05-09 22:00:36 +0200, Scott David Daniels <Scott.Daniels@xxxxxxx> said:

2.6 has a PriorityQueue in the Queue module.
If you aren't using 2.6, you could copy the code for your own version.


.

Relevant Pages

  • Re: filewatcher question
    ... I guess I can get the file watcher change event to just queue up the file ... Furthermore the file watcher changed event will do a quick look up if the ... I will also have to look up thread safe GUI as well ...
    (microsoft.public.dotnet.languages.csharp)
  • Re: Thread safe vector class
    ... > between worker threads and the main thread. ... producer/consumer queue. ... If that one doesn't fit your needs, Google for "thread safe producer ...
    (microsoft.public.vc.language)
  • Re: Thread safe queue in C
    ... within linux with C and i need a thread safe queue.How can i obtain a ... Anyway, creating a queue with POSIX Threads is rather trivial, I don't see why you can't quickly create one: ... typedef struct queue_s queue; ... queue* const _this, ...
    (comp.programming.threads)
  • Re: EventMachine.defer and ActiveRecord connection pool?
    ... up connections to the ActiveRecord connection pool and you are running ... request to a Queue (thread safe). ... its task it pops the next one off the queue. ... The whole purpose of @@running hash is to ensure that same task won't be ...
    (comp.lang.ruby)
  • Re: STL Queue
    ... > In the STL documentation, it says that a queue is not thread safe. ... > Thread A will "push" stuff onto a queue. ...
    (microsoft.public.win32.programmer.ui)