Re: Program Dies after 60 or so Iterations of Loop
- From: Hal Vaughan <hal@xxxxxxxxxxxxxxxxxxxx>
- Date: Sun, 26 Aug 2007 20:14:53 -0400
I hit the wrong key and sent by mistake. Response continued below:
Hal Vaughan wrote:
Peter J. Holzer wrote:
On 2007-08-26 01:07, Hal Vaughan <hal@xxxxxxxxxxxxxxxxxxxx> wrote:
I found the old time radio shows a the Interet Archives, so I wrote a
program to scan the shows, then the episodes and let me list the ones I
wanted to download. The program then goes through the list of files and
downloads each file (most are MP3, some are Ogg). Files are at least a
few
megabytes in length. The problem is that after about 60 or so downloads
(it varies), my computer (running Linux, Ubuntu Fiesty Fawn) slows down
for about 30 minutes (not an exaggeration), then the program stops with
a "Killed!" statement.
That sounds like you are exhausting virtual memory. When the total
memory used by all processes approaches the sum of RAM and swap space,
the system spends more and more time to finding yet another piece of
memory it doesn't need immediately and could reuse. If you are unlucky,
the system becomes completely unresponsive. However, most of the time
the system just gives up at some point and kills a process (hopefully
the one which caused this sorry state).
Run "top" in a second window to confirm this.
I figured it was a virtual memory and swap issue as well, but I figured
I'm best just saying the symptoms instead of assuming because I find that
there is often an aspect I don't know about.
I had problems with running top. There was a slight slow down, then
suddenly a big one and it took forever to change to the Konsole window, so
I switched to my server, which always has an ssh connection open to my
workstation for cases like this and tried running top. By the the time it
came up in Konsole or ssh, the Perl program was already terminated and not
showing and the system was cleaning up.
#Download the file with WWW::Mechanize
#Isn't $data reinitialized each loop? Could old copies not be cleared
#by garbage collector?
$data = getpage($url);
$data is reinitialized, but WWW::Mechanize may cache downloaded
documents. I vaguely remember that this has been discussed here before
and that there is a way to turn this off even though perldoc
WWW::Mechanize doesn't seem to mention it. Google should find it.
I found it by Googling for "Perl mechanize memory." I tried several
different phrases, but they didn't help until I tried that one. Here's
the
trick. I was creating Mech like this:
$mech = WWW::Mechanize->new;
By default, Mech caches all the pages (or binary files) it downloads in
memory. I changed it to this:
$mech = WWW::Mechanize->new(stack_depth => 3);
If the stack_depth is set to 0, it'll cache all pages (the default). I
experimented and got VERY slow downloads with a stack_depth of 1 (it
doesn't make sense why, but I don't know what else that effects), so I
experimented. Maybe it was just chance that when I changed it from 1 to 3
it worked, but I'm going to leave it there until it finishes it's current
batch of files before testing it to be sure the speed is the same with a
value of 1.
Thanks! I had never imagined it was an issue with the module. I still tend
to think I've missed something. I'm self-taught and I eem to constantly
find new holes in what I've learned as I keep going.
Hal
.
- References:
- Program Dies after 60 or so Iterations of Loop
- From: Hal Vaughan
- Re: Program Dies after 60 or so Iterations of Loop
- From: Peter J. Holzer
- Re: Program Dies after 60 or so Iterations of Loop
- From: Hal Vaughan
- Program Dies after 60 or so Iterations of Loop
- Prev by Date: Re: Program Dies after 60 or so Iterations of Loop
- Next by Date: Re: Program Dies after 60 or so Iterations of Loop
- Previous by thread: Re: Program Dies after 60 or so Iterations of Loop
- Next by thread: Searching within the hash values of an array of hashes
- Index(es):
Relevant Pages
|
