# Re: NumPy arrays that use memory allocated from other libraries or tools

*From*: sturlamolden <sturlamolden@xxxxxxxx>*Date*: Wed, 10 Sep 2008 11:15:26 -0700 (PDT)

On Sep 10, 6:39 am, Travis Oliphant <oliphant.tra...@xxxxxxxx> wrote:

I wanted to point anybody interested to a blog post that describes a

useful pattern for having a NumPy array that points to the memory

created by a different memory manager than the standard one used by

NumPy.

Here is something similar I have found useful:

There will be a new module in the standard library called

'multiprocessing' (cf. the pyprocessing package in cheese shop). It

allows you to crerate multiple processes (as opposed to threads) for

concurrency on SMPs (cf. the dreaded GIL).

The 'multiprocessing' module let us put ctypes objects in shared

memory segments (processing.Array and processing.Value). It has it's

own malloc, so there is no 4k (one page) lower limit on object size.

Here is how we can make a NumPy ndarray view the shared memory

referencey be these objects:

try:

import processing

except:

import multiprocessing as processing

import numpy, ctypes

_ctypes_to_numpy = {

ctypes.c_char : numpy.int8,

ctypes.c_wchar : numpy.int16,

ctypes.c_byte : numpy.int8,

ctypes.c_ubyte : numpy.uint8,

ctypes.c_short : numpy.int16,

ctypes.c_ushort : numpy.uint16,

ctypes.c_int : numpy.int32,

ctypes.c_uint : numpy.int32,

ctypes.c_long : numpy.int32,

ctypes.c_ulong : numpy.int32,

ctypes.c_float : numpy.float32,

ctypes.c_double : numpy.float64

}

def shmem_as_ndarray( array_or_value ):

""" view processing.Array or processing.Value as ndarray """

obj = array_or_value._obj

buf = obj._wrapper.getView()

try:

t = _ctypes_to_numpy[type(obj)]

return numpy.frombuffer(buf, dtype=t, count=1)

except KeyError:

t = _ctypes_to_numpy[obj._type_]

return numpy.frombuffer(buf, dtype=t)

With this simple tool we can make processes created by multiprocessing

work with ndarrays that reference the same shared memory segment. I'm

doing some scalability testing on this. It looks promising :)

.

**Follow-Ups**:**Re: NumPy arrays that use memory allocated from other libraries or tools***From:*Travis Oliphant

**References**:**NumPy arrays that use memory allocated from other libraries or tools***From:*Travis Oliphant

- Prev by Date:
**Re: removing text string** - Next by Date:
**Re: subprocess.Popen hangs at times?** - Previous by thread:
**NumPy arrays that use memory allocated from other libraries or tools** - Next by thread:
**Re: NumPy arrays that use memory allocated from other libraries or tools** - Index(es):