Re: Adding a Par construct to Python?

On 17 Mai, 14:05, jer...@xxxxxxxxxxxxxxxxxxxxxxxxxxxx wrote:
From a user point of view I think that adding a 'par' construct to
Python for parallel loops would add a lot of power and simplicity,
e.g.

par i in list:
    updatePartition(i)

You can do this right now with a small amount of work to make
updatePartition a callable which works in parallel, and without the
need for extra syntax. For example, with the pprocess module, you'd
use boilerplate like this:

import pprocess
queue = pprocess.Queue(limit=ncores)
updatePartition = queue.manage(pprocess.MakeParallel
(updatePartition))

(See http://www.boddie.org.uk/python/pprocess/tutorial.html#Map for
details.)

At this point, you could use a normal "for" loop, and you could then
"sync" for results by reading from the queue. I'm sure it's a similar
story with the multiprocessing/processing module.

There would be no locking and it would be the programmer's
responsibility to ensure that the loop was truly parallel and correct.

Yes, that's the idea.

The intention of this would be to speed up Python execution on multi-
core platforms. Within a few years we will see 100+ core processors as
standard and we need to be ready for that.

In what sense are we not ready? Perhaps the abstractions could be
better, but it's definitely possible to run Python code on multiple
cores today and get decent core utilisation.

There could also be parallel versions of map, filter and reduce
provided.

Yes, that's what pprocess.pmap is for, and I imagine that other
solutions offer similar facilities.

BUT...none of this would be possible with the current implementation
of Python with its Global Interpreter Lock, which effectively rules
out true parallel processing.

See:http://jessenoller.com/2009/02/01/python-threads-and-the-global-inter....

What do others think?

That your last statement is false: true parallel processing is
possible today. See the Wiki for a list of solutions:

http://wiki.python.org/moin/ParallelProcessing

In addition, Jython and IronPython don't have a global interpreter
lock, so you have the option of using threads with those
implementations, too.

Paul
.

Relevant Pages

  • Re: Adding a Par construct to Python?
    ... Python for parallel loops would add a lot of power and simplicity, ...     updatePartition ... responsibility to ensure that the loop was truly parallel and correct. ... processes in order to achieve the same - fine for expert programmers ...
    (comp.lang.python)
  • Re: flow control and nested loops
    ...     glortz; ... What's considered "best practice" in the Python world for this sort ... For the outermost loop, ... raise NextY ...
    (comp.lang.python)
  • Re: for-else
    ... So I may like something similar for Python 3.x (or the removal of the ...   else: ... keeps the same name as the corresponding clause on the while loop. ...
    (comp.lang.python)
  • Re: loops
    ...     print x ... What you wrote would appear to be an infinite loop so I'll assume you meant ... It seems weird to me their is no such thing in python. ... The large majority of use cases for iteration are iterating though ...
    (comp.lang.python)
  • Can you introduce some book about python?
    ... appending key-value pairs to a dict (Peter ... Is Python suitable for a huge, ... >> to the list on each loop... ...
    (comp.lang.python)