Re: optimizing memory utilization

From: Istvan Albert (ialbert_at_mailblocks.com)
Date: 09/15/04 

Date: Wed, 15 Sep 2004 11:39:22 -0400

> can't believe that python requires memory 200% "overhead" to store my
> data as lists of lists.

And it almost certainly doesn't.

One one hand there could be a sizebale overhead when running
your program even when loading a single data line.
So don't extrapolate from a small datasets.

But what I believe happens to you is that you keep references
alive and you end up with data duplication.

For example if you do:

lines = file('whatever').readlines()

then you say for example:

stripped_lines = [ line.strip() for line in lines ]

you've just made your program use twice the memory
at that point of the code.

You may also inadvertently keep your 'lines' variable
in the scope of the full program then
you'll effectively end up with a lot more memory consumption
overall.

Check and plug all these holes.

Istvan.

Relevant Pages

  • Re: how to find available classes in a file ?
    ... Is there a way of getting this list, without manually parsing the file? ... Bascially I am loading the module and then using diron the module ... attribute of the type classobj (for old style classes) and type type ... while the later also lists the imported ...
    (comp.lang.python)
  • Re: Data Validation
    ... You can't show more than 8 rows in a data validation drop down. ... but that should be enough and it makes loading the list very ... Excel and Word Function Lists available free to good homes. ...
    (microsoft.public.excel.worksheet.functions)
Loading