Re: multiprocessing vs thread performance

En Wed, 07 Jan 2009 23:05:53 -0200, James Mills <prologic@xxxxxxxxxxxxxxxxxxx> escribió:

Does anybody know any tutorial for python 2.6 multiprocessing? Or bunch of
good example for it? I am trying to break a loop to run it over multiple
core in a system. And I need to return an integer value as the result of the
process an accumulate all of them. the examples that I found there is no
return for the process.

You communicate with the process in one of several
ways:
* Semaphores
* Locks
* PIpes

The Pool class provides a more abstract view that may be better suited in this case. Just create a pool, and use map_async to collect and summarize the results.

import string
import multiprocessing

def count(args):
(lineno, line) = args
print "This is %s, processing line %d\n" % (
multiprocessing.current_process().name, lineno),
result = dict(letters=0, digits=0, other=0)
for c in line:
if c in string.letters: result['letters'] += 1
elif c in string.digits: result['digits'] += 1
else: result['other'] += 1
# just to make some "random" delay
import time; time.sleep(len(line)/100.0)
return result

if __name__ == '__main__':

summary = dict(letters=0, digits=0, other=0)

def summary_add(results):
# this is called with a list of results
for result in results:
summary['letters'] += result['letters']
summary['digits'] += result['digits']
summary['other'] += result['other']

# count letters on this same script
f = open(__file__, 'r')

pool = multiprocessing.Pool(processes=6)
# invoke count((lineno, line)) for each line in the file
pool.map_async(count, enumerate(f), 10, summary_add)
pool.close() # no more jobs
pool.join() # wait until done
print summary

--
Gabriel Genellina

.

Relevant Pages