Re: Looping-related Memory Leak

Tom Davis wrote:

I am having a problem where a long-running function will cause a
memory leak / balloon for reasons I cannot figure out. Essentially, I
loop through a directory of pickled files, load them, and run some
other functions on them. In every case, each function uses only local
variables and I even made sure to use `del` on each variable at the
end of the loop. However, as the loop progresses the amount of memory
used steadily increases.

I had a related problem before where I would loop through a very large
data-set of files and cache objects that were used to parse or
otherwise operate on different files in the data-set. Once again,
only local variables were used in the cached object's methods. After
a while it got to the point where simply running these methods on the
data took so long that I had to terminate the process (think, first
iteration .01sec, 1000th iteration 10sec). The solution I found was
to cause the cached objects to become "stale" after a certain number
of uses and be deleted and re-instantiated.

Here the alleged "memory leak" is clearly the cache, and the slowdown is
caused by garbage collector. The solution is to turn it off with
gc.disable() during phases where your programm allocates huge amounts of
objects with the intent of keeping them for a longer time.

However, in the current case, there is no caching being done at all.
Only local variables are involved. It would seem that over time
objects take up more memory even when there are no attributes being
added to them or altered. Has anyone experienced similar anomalies?
Is this behavior to be expected for some other reason? If not, is
there a common fix for it, i.e. manual GC or something?

Unless you post a script demonstrating the leak I will assume you are
overlooking a reference that keeps your data alive -- whether it's a true
global or within a long-running function doesn't really matter.

Peter
.

Relevant Pages

  • Looping-related Memory Leak
    ... memory leak / balloon for reasons I cannot figure out. ... end of the loop. ... only local variables were used in the cached object's methods. ...
    (comp.lang.python)
  • Re: Garbage collection question?
    ... In the example you present, the Object created at the top of each loop iteration becomes eligible for GC as soon as a new reference is assigned to variable o, *provided that* the old Object's reference has not been assigned to some object that persists across loop iterations. ... Even if different slots were used, the stack frame size is fixed at compile time, so an infinite loop will quickly have to start reusing variable slots, so you will not get a memory leak from this direction. ... When a thread returns from a method, its stack frame for that method invocation is popped, freeing all the memory occupied by the local variables of that method for that invocation. ...
    (comp.lang.java.programmer)
  • Re: Looping-related Memory Leak
    ... memory leak / balloon for reasons I cannot figure out. ... end of the loop. ... reference count never reaches zero, and they remain alive until the ... the generational collector doesn't break cycles that involve ...
    (comp.lang.python)
  • Re: How does "new" work in a loop?
    ... freeware program that does this, but it requires all input and output ... My program seems to work fine, but I'm wondering about this loop: ... fsOut.Write(inputBuffer, 0, (int) inputBuffer.Length); ... Unlikely that you are creating a memory leak. ...
    (microsoft.public.dotnet.languages.csharp)
  • Re: Go To Statement Considered Harmful
    ... -> variables on a stack. ... including the local variables of an outer or ... In this case, it's not the FOR loop that has its own local variables, ... If you use a GOSUB instead: ...
    (comp.lang.basic.misc)