Re: 70% [* SPAM *] Re: multiprocessing.Queue blocks when sending large object



On Mon, 5 Dec 2011 09:02:08 +0100, DPalao <dpalao.python@xxxxxxxxx>
wrote:

El Martes Noviembre 29 2011, DPalao escribió:
Hello,
I'm trying to use multiprocessing to parallelize a code. There is a number
of tasks (usually 12) that can be run independently. Each task produces a
numpy array, and at the end, those arrays must be combined.
I implemented this using Queues (multiprocessing.Queue): one for input and
another for output.
But the code blocks. And it must be related to the size of the item I put
on the Queue: if I put a small array, the code works well; if the array is
realistically large (in my case if can vary from 160kB to 1MB), the code
blocks apparently forever.
I have tried this:
http://www.bryceboe.com/2011/01/28/the-python-multiprocessing-queue-and-lar
ge- objects/
but it didn't work (especifically I put a None sentinel at the end for each
worker).

Before I change the implementation,
is there a way to bypass this problem with multiprocessing.Queue?
Should I post the code (or a sketchy version of it)?

TIA,

David

Just for reference. The other day I found the explanation by "ryles" on
his/her mail of 27th aug 2009, with title "Re: Q: multiprocessing.Queue size
limitations or bug...". It is very clarifying.
After having read that I arranged the program such that the main process did
not need to know when the others have finished, so I changed the process join
call with a queue get call, until a None (one per process) is returned.

Best,

David


Why do people add character like [* SPAM *] to their subject
lines ?? Is it supposed to do something ?? I figured since
programmers hang out here, maybe one of you know this.

Thanks,
boB


.



Relevant Pages

  • 70% [* SPAM *] Re: Re: multiprocessing.Queue blocks when sending large object
    ... produces a numpy array, and at the end, those arrays must be combined. ... the code blocks apparently forever. ... process join call with a queue get call, until a None ... programmers hang out here, maybe one of you know this. ...
    (comp.lang.python)
  • 70% [* SPAM *] Re: multiprocessing.Queue blocks when sending large object
    ... I'm trying to use multiprocessing to parallelize a code. ... numpy array, and at the end, those arrays must be combined. ... blocks apparently forever. ... call with a queue get call, until a None is returned. ...
    (comp.lang.python)
  • Re: The huge amount response data problem
    ... characters is at least two megabytes of memory. ... Instead of first creating a huge array of names, ... I'd read the file adding tasks to a queue of the same size, after filling the queue I'd pause reading the file until the queue has a spare space. ... I've never tried to program something like this in Perl so I'd imagine someone has already solved this and added modules to CPAN to assist in this sort of task. ...
    (comp.lang.perl.misc)
  • Re: Choice of data structure for large event queues?
    ... > from the queue, it is checked against the set. ... A tree, skip list, or ... yet another memory structure which contains the zombie'd part along with the ... a circular and expandable array (an array consiting of multiple smaller - ...
    (comp.programming)
  • Re: Rogue threads found! The plot thickens.... (was: Re: ThreadPool(was: Re: Thread.list confusion))
    ... array which holds on to references to the threads created in line 40 ... definitely be killed in line 47, the array still holds a reference to ... (A Thread can only belong to one ThreadGroup, and Queue shouldn't reorganize threads it didn't create.) ... Somewhere around here I have a stream implementation for it. ...
    (comp.lang.ruby)