Re: huge file : parsing problem

On Wed, 2006-29-03 at 15:53 +0000, Saurabh Singhvi wrote:
HI there,

I have a huge file (~7GB) of text data. I need to parse it to extract
something : basically minima and maxima, and then i need to quantify
the data into an output file based on ranges.

Now the problem is that the traditional method of opening and foreach
which is as follows:

open(FILE,"data.txt");
$min = 100;
$max = 0;

foreach $data (<FILE>) {

while( $data = <FILE> ){

if ($min > $data ) {
$min = $data ;
}
elsif ($max < $data) {
$max = $data ;
}
}
print $max, $min;
close FILE;

fails to work, as perl is trying to load the complete file into memory!

Now i also tried changing it to STDIN instead, by catting the data from
the text file.

./script.pl < data.txt

but this too gave the same error

Out of memory!

So kindly help me on how to do this thing.

thanks
Saurabh
--
__END__

Just my 0.00000002 million dollars worth,
--- Shawn

"For the things we have to learn before we can do them,
we learn by doing them."
Aristotle

* Perl tutorials at http://perlmonks.org/?node=Tutorials
* A searchable perldoc is at http://perldoc.perl.org/



.

Relevant Pages

  • huge file : parsing problem
    ... basically minima and maxima, and then i need to quantify ... Now the problem is that the traditional method of opening and foreach ... as perl is trying to load the complete file into memory! ...
    (perl.beginners)
  • Re: "Out of memory!" ???
    ... if there is enough memory to ... process the array in a loop by using "foreach". ... You make heavy use symbolic references. ... each iteration of the outer most foreach loop, ...
    (comp.lang.perl.misc)
  • Re: Frequency in large datasets
    ... runs out of memory. ... than using foreach. ... down into smaller chunks of about 5 gigs each. ...
    (comp.lang.perl.misc)
  • Re: "Out of memory!" ???
    ... my navie question is that if there is enough memory to handle the ... "foreach" is probably not the problem here. ... arrays and hashesfrom symbols just read by ... push @allSamples, $sample_ID; ...
    (comp.lang.perl.misc)