Re: Python memory handling

Greets,

Sorry for my late answer, google groups lost my post...
First, thanks you for your explanations about memory handling in the
os and python.
I've tried with python 2.5 under linux :
For the parsing of a 66 Mb xml file with cElementTree :
When starting python : 2.1 Mb private memory used
import xml.etree.cElementTree as ElementTree #3.4 Mb used
et=ElementTree.parse('otherdata.xml') #218.6 Mb used
del et #43.3 Mb used
et=ElementTree.parse('otherdata.xml') #218.6 Mb used
del et #60.6 Mb used
et=ElementTree.parse('otherdata.xml') #218.6 Mb used
del et #54.1 Mb used
et=ElementTree.parse('otherdata.xml') #218.6 Mb used
del et #54.1 Mb used
et=ElementTree.parse('otherdata.xml') #218.6 Mb used
del et #54.1 Mb used

Why does I have a such erratic memory freeing ?
I've tried the same test many time with a new interpreter and I've got
43.3 Mb after the first free and 54.1 Mb after the others.
If there is a memory pool limit in list ans dict, why can't I goes
back to 43.3 or 54.1 Mb all the times ?

I've tried using readlines():
When starting python : 2.1 Mb private memory used
f=open('otherdata.xml') #2.2 Mb used
data=f.readlines() #113 Mb used
del data #2.7 Mb used
f.seek(0) #2.7 Mb used
data=f.readlines() #113 Mb used
del data #2.7 Mb used

That time I have a good memory handling (for my definition of memory
handling)

So is there a problem with cElementTree ?

I've done a last test with ElementTree :
When starting python : 2.1 Mb private memory used
import xml.etree.ElementTree as ElementTree #3.2 Mb used
et=ElementTree.parse('otherdata.xml') #211.4 Mb used (but very
slow :p)
del et #21.4 Mb used
et=ElementTree.parse('otherdata.xml') #211.4 Mb used
del et #29.8 Mb used

So why does I have such differences in memory freeing ? Only due to
fragmentation ?

Anyway, python 2.5 has a better memory handling than 2.4, but still
not perfect for me.
I think I've not really understood the problem with the use of malloc
(fragmentation,...)

Thanks for your help
Regards,
FP

.

Relevant Pages

  • Re: Memory Leak
    ... > Firstly I'm not using the latest upto date python releases so my first ... The script seems to work ok, it builds by setting processes off ... > are destroyed by using del to try and free up memory, ... I don't think you understand what 'del' does. ...
    (comp.lang.python)
  • Re: 2.6, 3.0, and truly independent intepreters
    ... just the GIL being in place, but of course it's there for a reason. ... Python faster on single core machines and more stable on multi core ... Other language designers think the same way. ... with languages that use memory pointers, have the potential to get out ...
    (comp.lang.python)
  • Re: why cannot assign to function call
    ... a porch the same if all its ... actual Python objects tend to be mutable only if they are ... In CPython, the id is given by the memory location of the object, which ... significant changes and replacements and add-ons that nevertheless don't ...
    (comp.lang.python)
  • Re: Writing huge Sets() to disk
    ... that this code takes a lot of extra memory. ... > I believe it's the references problem, ... It's a bit unfortunate that all those instance variables are global to ... it merely aggregates it for use in storing new Python objects. ...
    (comp.lang.python)
  • Re: Memory Leaks and Heapy
    ... I'm trying to plug some memory leaks in a TurboGears program. ... and simplejson for message passing between the server ... I'm using Heapy. ... means there are not many leaked objects in the python space. ...
    (comp.lang.python)