Comparing fields in 2 large files

I am new to perl and would appreciate any suggestions as to how to do
the following.

I have two files, one with 3.5 million records, the other with almost a
million records. Basically here's what I need to do.

See if field_1 in file_a is part of field_1 in file_b
If so, see if field_2 in file_a is part of field_1 in file_b
If so, see if field_3 in file_a is equal to field_2 in file_b
If not equal, write out field_4 in file_a plus field_3 in file_b

I have written a script that will do this, but it runs literally for
days. I'm guessing my method is not the most efficient. I do not
have enough memory to read the files at one time into an array.

Thanks in advance for ideas.

Craig
.

Relevant Pages

  • Re: XS: Free memory
    ... Do I need to free the memory? ... I use the XPUSHs to pass an array of real to a perl sub, ... That will probably depend on how you allocated your array of reals in ...
    (perl.beginners)
  • Re: Conserving memory
    ... Is the memory that was used for that array now freed? ... Is the memory that was used for that anonymous array now freed? ... OS-memory allocated by perl gets detached ... from a container when the container itself is ...
    (perl.beginners)
  • Re: The huge amount response data problem
    ... characters is at least two megabytes of memory. ... Instead of first creating a huge array of names, ... I'd read the file adding tasks to a queue of the same size, after filling the queue I'd pause reading the file until the queue has a spare space. ... I've never tried to program something like this in Perl so I'd imagine someone has already solved this and added modules to CPAN to assist in this sort of task. ...
    (comp.lang.perl.misc)
  • Re: why the perl docs suck
    ... from a Perl specific higher level ... Does "push" create an array of ... > the granularity of a Perl push statment. ... > you don't know that memory allocation is not contiguous! ...
    (comp.lang.perl.misc)
  • Re: Fast Linear Hex Search
    ... I've had this happen many times and someone on the Perl ... Perl allocates memory but does not free up memory as quickly ... > simply because Perl's garbage collector hasn't realized that some of the ... > you can make the original array global versus local. ...
    (comp.lang.perl.moderated)