custom allocation and custom type

Hi all,

I am trying to create a custom tuple type. I want it to live in a
custom memory region, which will be a memory-mapped file. Its
contents cannot be PyObject*. They have to be offsets into mapped
memory. GetItem( i ) would return:

(PyObject *)( t->ob_item[ i ]+ mmap_base_addr );

In the constructor, each item x would be stored as:

BYTE* x_offset= x- mmap_base_addr;
t->ob_item[ 0 ]= x_offset;

For starters, only ints, strings, and other tuples will be members.
I'll work out the list type and user-defined classes at a later time.
Each will be required to be already existing in the mmap; that is, be
of custom types too.

I had partial success with strings and ints just memcpy-ing the entire
PyObject structure. They are easier because they are immutable, and
don't contain references. That's why tuple is my next step: it's
immutable, but can contain references.

I want to back up a step and ask some questions about ints first.

What worries are there in memcpy-ing a PyIntObject? Would it be
better to compile my own MmapIntObject? I do Py_INCREF on the object
I return, to ensure it lives on, but I am getting an occasional crash
on exit. What did I miss? Do I need to readjust the ob_type pointer
every time I load the module (and lock for exclusive access)? Does
Python try to free the memory when it exits?

For the tuples, is there any way I can reuse the tuple code? The
constructor uses a custom pool for speed which I'd probably eliminate,
for the time. Slices would have to be MmapTupleObject instances too.
Should I also create MmapTupleIterObject?

PyObject_GC_New and PyObject_GC_NewVar are confusing. I can't follow
them to the actual call to 'malloc', since they themselves lead around
Python's custom memory pool. What steps do I need to be sure to take
in my own?

I'm having trouble building PerstTupleObject.c (persistent tuple
object) to try it out. I'm using MinGW on WinXP. I added #include
"persttupleobject.h" to the top of "persttupleobject.c" already, but
it's a duplicate otherwise, with the name replaced.

---
C>c:/programs/mingw/bin/g++ persttupleobject.c -c -Ic:/programs/
python25/include

persttupleobject.c:650: warning: 'PyPerstTuple_Type' defined locally
after being
referenced with dllimport linkage

C>c:/programs/mingw/bin/g++ -shared persttupleobject.o -o
persttupleobject.pyd -L
c:/programs/python25/libs -lpython25

persttupleobject.o(.text+0xce):persttupleobject.c: undefined reference
to `_imp_
_PyPerstTuple_Type'
persttupleobject.o(.text+0x15c):persttupleobject.c: undefined
reference to `_PyG
C_generation0'
[snip]

---
I get some of the same errors when compiling tupleobject.c from the
Python-2.5.2 distribution:

C>c:/programs/mingw/bin/g++ tupleobject.c -c -Ic:/programs/python25/
include

tupleobject.c:649: warning: 'PyTuple_Type' defined locally after being
reference
d with dllimport linkage

C>c:/programs/mingw/bin/g++ -shared tupleobject.o -o tupleobject.pyd -
Lc:/progra
ms/python25/libs -lpython25

c:/programs/python25/libs/libpython25.a(dcbbs00622.o)(.text+0x0):
multiple defin
ition of `PyTuple_Type'
tupleobject.o(.data+0x140):tupleobject.c: first defined here
tupleobject.o(.text+0x15c):tupleobject.c: undefined reference to
`_PyGC_generati
on0'
[snip]

---
Thanks in advance.
.

Relevant Pages

  • [RFC] page replacement requirements
    ... Submitting too much I/O at once can kill latency and even lead to deadlocks when bounce buffers are involved. ... Must be able to deal with multiple memory zones efficiently. ... When on completion of the write to their backing-store the reference bit is still unset a callback is invoked to place them so that they are immediate candidates for reclaim again. ... For traditional page replacement algorithms this is not a big issue since we just implement per zone page replacement; ...
    (Linux-Kernel)
  • Re: Java or C++?
    ... >> CTips wrote: ... GC can only kick in when the last reference has ... If f was already bound to a lot of memory when it enters do-it then ... > more than the extra memory leaked because of sloppy manual deallocation]. ...
    (comp.programming)
  • Re: 4-way Opteron vs. Xeon-IBM X3 architecture
    ... >>>The point here is that the issue concerns both speed AND capacity. ... >As to the reference, the message header points right back in this thread, ... >>>As you may suspect, I read plenty about memory systems, and I would ... >>>from the enthusiast market and assumed that it would work in the server ...
    (comp.sys.ibm.pc.hardware.chips)
  • Re: Garbage Collection Issues in long-standing services
    ... I would agree that I must be holding on to some references, ... store a reference to CS so that it can use it to send data back to the client. ... > By starting another process that allocates memory, ... it does not aggressively cleanup until the amount of physical memory ...
    (microsoft.public.dotnet.languages.csharp)
  • Re: A re-announce on GCs defects
    ... It's bad for CPU/Resource intensive but memory cheap objects. ... There may be more than one strong references and you don't know when and where to call Dispose. ... Instead of calling the destructor you call Dispose if the reference counter is 0. ... Note GC in java and C# is not really an addictive as someone would argue since there is no way to do real memory management like delete obj in C++. ...
    (microsoft.public.dotnet.languages.csharp)