Re: 64 bit Python
From: Jeff Epler (jepler_at_unpythonic.net)
Date: 02/15/05
- Next message: Jeff Epler: "Re: [Errno 18] Invalid cross-device link using os.rename"
- Previous message: Charles Stricklin: "How can I tell if my host supports Python?"
- In reply to: Mathias Waack: "64 bit Python"
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
Date: Mon, 14 Feb 2005 18:28:31 -0600 To: Mathias Waack <M.Waack@gmx.de>
There's not enough information to guess the "real problem", but it could
be this:
"variable size" objects (declared with PyObject_VAR_HEAD) are limited to
INT_MAX items since the ob_size field is declared as 'int'.
This means that a Python string, tuple, or list (among other types) may
be limited to about 2 billion items on ILP32 and LP64 architectures.
Dicts and probably sets are also limited to INT_MAX elements, because
the "ma_fill" and "ma_used" fields are ints.
If you don't mind recompiling all your extensions, you could change the
type of ob_size to long in the "#define PyObject_VAR_HEAD". I don't
know what breaks when you do this, but maybe a few google or google
groups searches could help you find others who have tried this.
Jeff
PS the limit of 500MB of "real data" in the 32-bit system may be because
a realloc may temporarily require (old size + new size) storage when it
does the equivalent of
new_ptr = malloc(new_size)
memcpy(new_ptr, old_ptr, old_size)
free(old_size)
which will temporarily use >900MB of data when resizing a ~450MB object.
Python frequently uses realloc() to grow structures like lists. If you
are working with strings, then
s = s + t
doesn't ever use realloc (in 2.3.x anyway) but always allocates and
fills the result 's+t', only freeing the old value of s later when it is
no longer reacahable (which could be as soon as the assignment statement
completes)
- application/pgp-signature attachment: stored
- Next message: Jeff Epler: "Re: [Errno 18] Invalid cross-device link using os.rename"
- Previous message: Charles Stricklin: "How can I tell if my host supports Python?"
- In reply to: Mathias Waack: "64 bit Python"
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
Relevant Pages
|
