Re: HowTo parse huge Files

cadetg@xxxxxxxxxxxxxx wrote:
Dear Perl Monks, I am developing at the moment a script which has to
parse 20GB files. The files I have to parse are some logfiles. My
problem is that it takes ages to parse the files. I am doing something
like this:

my %lookingFor;
# keys => different name of one subset
# values => array of one subset

my $fh = new FileHandle "< largeLogFile.log";
while (<$fh>) {
foreach my $subset (keys %lookingFor) {
foreach my $item (@{$subset}) {
if (<$fh> =~ m/$item/) {
my $writeFh = new FileHandle ">> myout.log"; print $writeFh <
$fh>;
}
}
}

How many key-value pairs does %lookingFor (typically?) have?

BugBear
.

Relevant Pages

  • Re: forth for log parsing
    ... gavino writes: ... Does anyone here ever use gforth or another open forth to parse ...
    (comp.lang.forth)
  • Re: HowTo parse huge Files
    ... Mumia W. wrote: ... The files I have to parse are some logfiles. ... You are aware that line 2 reads in a new chunk from $fh, ...
    (comp.lang.perl.misc)
  • HowTo parse huge Files
    ... Dear Perl Monks, I am developing at the moment a script which has to ... The files I have to parse are some logfiles. ...
    (comp.lang.perl.misc)
  • Re: HowTo parse huge Files
    ... The files I have to parse are some logfiles. ... # values => array of one subset ... foreach my $item { ...
    (comp.lang.perl.misc)
  • Parsing an HTML file - skipping to a line
    ... I got an HTML file I want to parse. ... I have it stored in an array or whatever. ... If I was reading from file, I could use a foreach and break when I ...
    (perl.beginners)