Re: Memory issues

jm <jm@xxxxxxxxx> writes:

Modifying a little bit again the script, and checking execution with
ltrace, I observed malloc is called 1871 times when free is just called
922 times.
Isn't it an issue?

Please keep in mind that perl's memory allocation strategy in general is
optimized for longer running programs, not for one-off scripts (which
makes sense, since one-off scripts don't usually need the performance
gains). This means that for instance subroutines will get memory
allocated on the assumption that they'll be called again, and will take
about as much memory the next time.

This is NOT a memory leak per se, but it does mean that if you have a
subroutine that takes 100Mb to complete, your program will take that
memory and probably not give it back until the program ends. IOW, if you
have a long-running program that only means you need 100Mb for it to
run, it does NOT mean it takes a 100Mb for each call.

In your test case, don't assume that just because the regular expression
replacements don't in theory *need* to use any additional RAM, they
won't. Especially not if you're using UTF-8 encoded strings (which you
are). Perl algorithms tend to exchange RAM for speed in most cases
anyway, and replacing a match with a new string of exactly the same
length in bytes in a unicode string is a pretty uncommon use-case, so
it's likely not optimized.

Anyway, I've not seen a serious memory leak in perl itself in ages, and
I run perl processes that use up to 8 Gb of RAM and run for months
without issues.

--
Joost Diepenmaat | blog: http://joost.zeekat.nl/ | work: http://zeekat.nl/
.

Relevant Pages

  • Re: How to reinvent grep in perl?
    ... that you want to grep into it, and pipe its output into grep. ... Now I could do this with pipes and 3 different instances of perl ... So am I concerned about memory problems? ... string (that becomes a 3MB string after I prepend the file name and ...
    (perl.beginners)
  • Re: You thoughts/philosphies on manual garbage collection
    ... reconfiguring parameter files. ... There is a ton of string manipulation ... pointers being left laying around in the ether of RAM? ... If you do encounter excessive memory usage then you should ...
    (comp.lang.ruby)
  • Re: Max size of an array used in perl.
    ... Oses these days aren't limited by RAM. ... Disk is slower than RAM so the machine will go to a crawl. ... And, unfortunately, there are some things that are easy to do in Perl ... structures in memory instead of reverting to using a database. ...
    (comp.lang.perl.misc)
  • Re: Extract Numeric values from string
    ... I dont want to use split caue ... The string apparently contains 112M values. ... perl has a policy of trading memory for speed. ...
    (comp.lang.perl.misc)
  • Re: Perl garbage collector
    ... Does your OS have a function that allows processes to return memory to ... As you see some RAM is given back to the OS ... I would expect that the Perl virtual machine ... memory allocation and garbage collection. ...
    (perl.beginners)