Re: using mmap on large (> 2 Gig) files
- From: "Travis E. Oliphant" <oliphant.travis@xxxxxxxx>
- Date: Mon, 23 Oct 2006 17:57:31 -0600
Martin v. Löwis wrote:
myeates@xxxxxxxxxxxx schrieb:Anyone ever done this? It looks like Python2.4 won't take a length arg2 Gig since its not seen as an int.
What architecture are you on? On a 32-bit architecture, it's likely
impossible to map in 2GiB, anyway (since it likely won't fit into the
available address space).
On a 64-bit architecture, this is a known limitation of Python 2.4:
you can't have containers with more than 2Gi items. This limitation
was removed in Python 2.5, so I recommend to upgrade. Notice that
the code has seen little testing, due to lack of proper hardware,
NumPy uses the mmap object and I saw a paper at SciPy 2006 that used Python 2.5 + mmap + numpy to do some pretty nice and relatively fast manipulations of very large data sets.
So, the very useful changes by Martin have seen more testing than he is probably aware of.
-Travis
.
- References:
- using mmap on large (> 2 Gig) files
- From: myeates
- Re: using mmap on large (> 2 Gig) files
- From: "Martin v. Löwis"
- using mmap on large (> 2 Gig) files
- Prev by Date: Re: How to get each pixel value from a picture file!
- Next by Date: Re: numpy error
- Previous by thread: Re: using mmap on large (> 2 Gig) files
- Next by thread: Re: using mmap on large (> 2 Gig) files
- Index(es):
Relevant Pages
|
