Re: Need some hints on speeding up

• Previous message: Gunnar Hjalmarsson: "Re: $_ in condition"
• In reply to: Spamtrap: "Need some hints on speeding up"
• Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

Date: Wed, 11 Aug 2004 16:25:55 -0700

In article <oeskh09h1t9tdp3dknsr2f2dl16emkodvf@4ax.com>, Spamtrap
<ocdb3j002@sneakemail.com> wrote:

> I only do occasional Perl programming and most things I write are
> short processes. I have something I'm working on that is scanning a
> text file with about 15 million lines and trying to extract matches
> from another text file, which has about 170 entries. The second text
> file is read into an array. The process then scans through the big
> file for certain possible patterns - it will find those in about 1 out
> of 25 lines,, when it finds one, it then loops through the array
> trying to find a match there, and then writes out a couple of lines
> into another text file.
>
> It also writes to the screen a summary line about every 25th record.
>
> When I run this it takes anywhere from 1.5 to 7.5 hours. It seems that
> to avoid the 7.5 hour time, I should fresh reboot, and ctrl+alt+del
> almost everything.
>
> But even 1.5 hours is too long since I need to be able to run this
> with different sets of data several times a day,
>
> One thought I had is that writting a summary progress to the screen
> [which helps me judge how far along it is, may be slowing things down]
>
> Running Windows 98 SE and the latest version of Active Perl.

People are going to need a little bit more information to help you.

First of all, comp.lang.perl is a defunct newsgroup. You would do
better to post to comp.lang.perl.misc.

What kind of hardware are you using? Your program may be limited by CPU
speed, disk I/O speed, or memory size. Can you upgrade your hardware if
that proves to be the limiting factor?

How long does it take to read through the 15M-line file? That gives you
a baseline for the minimum amount of time it will take to process the
file. If your full program takes considerably more than that, you may
be using a slow search algorithm or have other problems. Printing will
show down your program somewhat, but probably not a significant amount
unless you are really printing too much. Cut down the amount and see.

The best thing you can do is post a minimal, complete program (to
comp.lang.perl.misc) that people can inspect. Normally, you would want
a runnable program, but since you can't post the big text file or even
the 170 line pattern file, you may have to rely on code inspection
rather than profiling. Be sure and make your posted program readable.

You might want to profile your program. Check out 'perldoc -q profile'.

• Previous message: Gunnar Hjalmarsson: "Re: $_ in condition"
• In reply to: Spamtrap: "Need some hints on speeding up"
• Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]