Re: Scalable method for searching in relatively big files

KaZ wrote:
Hi,

I want to do this:

I have a file A (the file size can be up to around 40MB, I know it
isn't that big), and I have to read it one line after the other (I use
a while loop for this), and possibly print something into file B,
depending on what is in the line. In each iteration, I have to search
the file A for pattern and the file B to check what has been already
written in it.

Right now, I made the searches with arrays and grep. I put the whole
file A in an array (array @a), file B is an empty array (array @b) at
the beginning, and at the end I write the array @b into file B. I read
directly in the file A each line for the while loop, because I feared
it would interfer with the search if I used the array @a for the while
loop also.

The file A is for the moment only 4MB, and the script take 10 minutes
to complete. So it isn't a scalable solution in my opinion.

I was told to use database or possibly tied hashes, in order to get
more scalability.

Can anybody tell me a bit more about this two methods? I read what a
hash is, but fail to see how it could help me.

Show us your script, and someone here will be able to help. Without knowing exactly what kind of search (exact match? regex? whole line?) you are doing, it is impossible to recommend much.


DS
.

Relevant Pages

  • Re: trouble backup with perl.
    ... >> I'm experiencing to write a perl script for to make backup with afio ... This is a very inefficient loop, ... > whole file to an array and then loop over the array. ... >> Pleeese help I need backup. ...
    (perl.beginners)
  • Re: Scalable method for searching in relatively big files
    ... KaZ wrote: ... directly in the file A each line for the while loop, ... it would interfer with the search if I used the array @a for the while ... So it isn't a scalable solution in my opinion. ...
    (comp.lang.perl.misc)
  • Scalable method for searching in relatively big files
    ... directly in the file A each line for the while loop, ... it would interfer with the search if I used the array @a for the while ... So it isn't a scalable solution in my opinion. ... I was told to use database or possibly tied hashes, ...
    (comp.lang.perl.misc)
  • RE: hashes and loops
    ... Karyn Williams wrote: ... > trying to create a script to remove from a file the items in ... then could create an array which has the filenames: ... Now you could loop off this like: ...
    (perl.beginners)
  • Re: newbie problem - file revision & size
    ... Use of included script samples are subject to the terms specified at ... I would *like* to set up an array of the ... > filenames I want info on, then cycle thru them until the list is finished. ...
    (microsoft.public.windowsxp.wmi)