How to tell what is using memory

From: Nikki R (nikki5+_at_notspamar.com.au)
Date: 12/17/04 

Date: Sat, 18 Dec 2004 00:43:50 +1000

Hi,

I'm running a Perl script on Linux Red Hat 7.3 (in a company - I can't
upgrade).
My script uses up a lot of memory because it is reading large data files.
The largest file is about 10MB.

The data files were stored by using Data::Dumper, and my script reads each
data file one at a time:

foreach .... {
    my $var1;
        # Suck in the entire Data::Dumper'd file.
    my $filestr = join('', <SOMEFILE>);
        # The Data::Dumper file consists of a statement like:
        # $var1 = {....} ; # Large hash reference.
    eval $filestr;
    # That sets $var1 which is a hash reference. Now do something with
$var1.
    ....
}

As the script runs, memory grows fairly quickly. I don't think have a
memory leak anywhere - I think that Perl allocates new memory whenever
it sees a larger data file than previously.

That's my understanding of the way Perl's garbage collection works, anyway.
In the above loop even though all of the 'my' variables go out of scope,
Perl doesn't normally release any memory back to the operating system until
the script finishes running.

Is there anything I can do about this, e.g. change the way that the data
file is read in? I definitely need to use a hash and I don't have the
choice
of using anything else but Data::Dumper (this is part of a large project
that I cannot change).

Is there also a good way of finding out (via a tracer or profiler of
some sort perhaps) which variables are using how much memory in a Perl
script? I eventually found out the above culprits only by doing a fair
amount of analysis using the Perl debugger and using Unix's "top"
program to see how much memory my perl script was using in total. 

--
Nikki

Relevant Pages

  • Critique my code: using Lisp to parse lots of "ps -elf" files in AIX
    ... It's my first use of Lisp at work, ... and another ksh script to analyze the output. ... success using them to diagnose some of their own memory problems, ... an hour to process about an hour's worth of data (i.e. 60 data files). ...
    (comp.lang.lisp)
  • Re: Out of Memory!
    ... My script is running on WinXP, Dos shell, which is a server program. ... unless it will pops-up a "Out of Memory!" ... Could someone tell me this error message is coming from Perl? ... This sub will be called every time a user Send or Get a mesg. ...
    (perl.beginners)
  • Re: DBD::Oracle memory leak
    ... > a perl with multi-threading enabled, ... custom compiled perl 5.8.4 w/ ithreads. ... tried a simple test script with the ... I watched the memory use slowly ...
    (perl.dbi.users)
  • Re: Out of Memory!
    ... My script is running on WinXP, Dos shell, which is a server program. ... unless it will pops-up a "Out of Memory!" ... Could someone tell me this error message is coming from Perl? ...
    (perl.beginners)
  • Re: Perl script to monitor memory usage on unix box
    ... I could write the Perl script to run a system command like "ps aux", ... parse that information and kill a job if>90% of the total system memory ... But since that isn't a Perl ... I don't know if there's any CPAN modules, but you should use the /proc ...
    (perl.beginners)