Re: Why is my regex so slow?

try this:


print "Match\n" if($target =~ /^$regex$/*o*);

only compile regex only once



On Fri, Oct 31, 2008 at 2:54 PM, Mark Wagner <carnildo@xxxxxxxxx> wrote:

I've got a script I'm using to search through a list of Wikipedia
article titles to find ones that match certain patterns.

As-written, if you run it and supply '.*target.*' on standard input,
it will process my test file in 125 seconds. Make any of the changes
mentioned in the comments, and the time needed will drop to 1.8
seconds. Why the difference? Particularly interesting is that it
seems to matter where the regex pattern came from: if it's from
standard input, testing is slow; if it's assigned in the script,
testing is fast.

If it matters, I'm using Perl 5.8.8.

To see the problem I'm having, download

http://download.wikimedia.org/eswiki/20081018/eswiki-20081018-all-titles-in-ns0.gz
(a 4.1-MB file), unzip it, and run the program supplying the name of
the unzipped file.

Thanks,
Mark Wagner

--------------
binmode STDIN, ":utf8"; # Comment this out to speed things up

while(<STDIN>)
{
my $lines = 0;
my $lines2 = 0;
my $regex;
$regex = $_;
chomp $regex;

#$regex = '.*target.*'; # Or uncomment this to speed things up
open INFILE, "<", $ARGV[0];
binmode INFILE, ":utf8"; # Or comment this out to speed things up

while(<INFILE>)
{
my $target = $_;
chomp $target;
$target =~ s/_/ /g;

print "Match\n" if($target =~ /^$regex$/); # Or make
this case-insensitive to speed things up, or remove the start and end
anchors to speed things up

$lines = $lines + 1;
if($lines >= 10000)
{
$lines = 0;
$lines2 += 10000;
print STDERR "$lines2\r";
}
}
}

--
To unsubscribe, e-mail: beginners-unsubscribe@xxxxxxxx
For additional commands, e-mail: beginners-help@xxxxxxxx
http://learn.perl.org/



Relevant Pages

  • Re: Why is my regex so slow?
    ... article titles to find ones that match certain patterns. ... As-written, if you run it and supply '.*target.*' on standard input, ... seems to matter where the regex pattern came from: ... , unzip it, and run the program supplying the name of ...
    (perl.beginners)
  • Re: Print Section of file
    ... Specifically, FS *can* be a regex (if it's not a single character, ... then awk assumes it's a regex). ... regex (in fact it never is, unless the context requires a regex). ... a relational expression or a boolean combination of patterns... ...
    (comp.lang.awk)
  • Re: re module non-greedy matches broken
    ... Regex patterns are expressed positive by ... Modern regular expression engines (which are no longer regular ... there are some common patterns on how to write some specific ... AFAICS one needs non-greedy regexps very very rarely at all. ...
    (comp.lang.python)
  • Re: Isolate lines in a text file and perform replacement
    ... Perl and regular expressions to normalize the file names. ... applying these regex patterns to playlist and xml files. ... The script performs a recursive search and finds files with these ...
    (comp.unix.shell)
  • Why is my regex so slow?
    ... I've got a script I'm using to search through a list of Wikipedia ... article titles to find ones that match certain patterns. ... seems to matter where the regex pattern came from: ... chomp $target; ...
    (perl.beginners)