Re: Working with very large text files
- From: "Paul Lalli" <mritty@xxxxxxxxx>
- Date: 30 Dec 2005 05:25:39 -0800
ytoledano@xxxxxxxxx wrote:
> Hi,
> I'm working on a server which writes text to large log files. The
> problem is that these files are 1 GB and when I open them with perl,
> perl tries to load the entire file into the memory and fails when XP is
> out of memory.
>
> I only need to use the text written in the last 5 minutes, which
> shouldn't take more then a couple of megs, but since I'm using XP, I
> can't use `tail`.
>
> What code doesn't make perl load the entire file into the memory?
Perl will only load the entire file into memory if you tell it to, by
slurping the entire thing, like:
my @lines = <FILE>;
foreach (@lines) {
process_line($_);
}
If you instead read the file line by line, and discard each one after
it's read, no more than one line will ever be in memory at once:
while (<FILE>) {
process_line($_);
}
Note that you *must* use while in the above, not foreach. Even though
they look the same, they do *not* operate the same internally.
If you're looking for just the last five lines of the file, you may
also be interested in modules such as Tie::File and File::Tail. (The
former is standard, the latter is available on CPAN)
Paul Lalli
.
- References:
- Working with very large text files
- From: ytoledano
- Working with very large text files
- Prev by Date: Date formatting
- Next by Date: Re: checking gzip files
- Previous by thread: Working with very large text files
- Next by thread: Date formatting
- Index(es):
Relevant Pages
|
