Re: Out of memory! When running perl script on windows

Hi
i used to do scripting for Bioinformatics and still doing it. Most of
the task is to swallow a huge chunk of text, parse it, then tranfer or
convert into another form (SQL database, other text format etc...). I
had met this kind of problem several times.

Normally what i do is :
1- check for infinite loop, which caused by
+ unchanged variable values (missing $i++ etc.)
+ never fulfilled condition (-1 > 0 etc.)
2- undef not-used variable in the middle of script (sometimes these
variables hold huge data from file, and used only before parsing). So
undef after parsing, and before doing other task, to save memory.
3- optimize (reading file chunk-by-chunk can save memory than slurping
a whole file etc.)

etc.




Peter J. Holzer wrote:
On 2006-08-27 15:30, felad <felad@xxxxxxxxxxx> wrote:
I have found that every single loop the commit memory grow by 6000 K so
I guess this is why the program crash
But I really can't find that cause of it, most of my variable are local
and I can't find any infinite loop

Any ideas ?

First, check that you aren't creating cyclic data structures: The perl
garbage collector cannot free cyclic data structures. If you really need
them, use weak references or destroy them explicitely after you are done
with them.

Second, lexical variables aren't freed immediately after they go out of
scope but only after the function returns. So if you create large
temporary variables, use them inside functions, not in the main program.

Third, modules which may be helpful: Devel::Cycle, Devel::Leak.

Fourth, perl has quite a bit overhead for each object: If you must keep
lots of data in memory, try to keep few large objects instead of many
little ones.

hp

--
_ | Peter J. Holzer | > Wieso sollte man etwas erfinden was nicht
|_|_) | Sysadmin WSR | > ist?
| | | hjp@xxxxxx | Was sonst wäre der Sinn des Erfindens?
__/ | http://www.hjp.at/ | -- P. Einstein u. V. Gringmuth in desd

.

Relevant Pages

  • Re: Stoping IE and removing it from memory
    ... > way to stop IE and remove it from memory. ... > IE stops and the visible window is removed from the screen. ... Set oWmi = GetObject("winmgmts:" _ ... -- torgeir, Microsoft MVP Scripting and WMI, Porsgrunn Norway Administration scripting examples and an ONLINE version of the 1328 page Scripting Guide: ...
    (microsoft.public.scripting.vbscript)
  • Re: Stoping IE and removing it from memory
    ... > way to stop IE and remove it from memory. ... > IE stops and the visible window is removed from the screen. ... Set oWmi = GetObject("winmgmts:" _ ... -- torgeir, Microsoft MVP Scripting and WMI, Porsgrunn Norway Administration scripting examples and an ONLINE version of the 1328 page Scripting Guide: ...
    (microsoft.public.scripting.wsh)
  • Re: Scripting Performance after SP2 applied?
    ... I agree that there is a lot of time spent setting up the scripting in the ... As far as memory configuration, I have always set my max server memory set ... now with forced parameterization, ... I then restarted SQL Server, and set the database to forced ...
    (microsoft.public.sqlserver.tools)
  • Re: Scriptable dynamic forms applications
    ... Well, best part of the story -don't laugh- is that DWS doesn't use Delphi strings, but actually uses variant strings. ... Speed wasn't directly an ansi vs unicode affair, but more an issue with how much memory was used, at some point the .Net GC becomes a slug and instead of using f.i. ... MB of memory for 1 MB of string data, it started using closer to 100 MB of RAM. ... This is the kind of things that scares me about using .Net for scripting. ...
    (borland.public.delphi.non-technical)