Can anyone explain PHP slowing down please ?


I'm using PHP 5 on Win-98 command line (ie no web server involved)

I'm processing a large csv file and when I loop through it I can process
around 275 records per second.

However at around 6,000 records this suddenly drops off to around 40
records per second.

This is a big problem as the "live" list is over 4 million records long.
I'd break it up but this is to be a regular test so that would be messy
to say the least -

Each record is 8 fields & total length tends to be below 200 characters
CSV is comma and ""

I was wondering if anyone with strong PHP knowledge has heard of this or
could help explain it please (As you probably know I'm very new to PHP)

I've trimmed the startup code to pseudocode to make it easier to read.
Otherwise my code is as below:

Sorry if line wrap is wrong - that would be my newsreader not the code

As you can see the code grabs a field from the database - spawns a
windows (msdos command line) .exe file to test it and writes the field
out to either a good or bad result file.

I dont do any file seeking or open and closing of files during the loop.

Tony

------------------------------ CODE START ------------------

<?php

//+++++++++++++++++++++++++++ PSeudocode start
open all new files for appending here (fopen($fin, 'a');)
open database for read-only here
Initialise all variables to 0 here

START;
get start-time
loop()
get end-time
write-statistics
close all files here
exit;
// +++++++++++++++++++++++++PSeudocode End

function loop() {
global
$fin,$fout,$fgood,$records,$fields,$good,$bad,$total,$dif_fcount,$nodata
;
while (($data = fgetcsv($fin, 1024, ",", "\"")) !== FALSE) {
if($data == '') { continue; }
$records++;
if (count($data) != $fields ) { $fields = count($data);
$dif_fcount++; }
if ($data[2] == '') { $data[2] = 'NO DATA' ; $nodata++; }
$raw = $data[7];
$star = "\"" . ($data[2]) . "\"";
$star = $raw ;
if (checkit($star) == false) {
fwrite($fout, $records . "," . $raw . "\r");
$bad += 1;
} else {
fwrite($fgood,$star . "\r");
$good += 1;
}
$total += 1;
echo("Total checked: " . $total . "\r" );
} //while
}


function checkit($star) {
exec("declination.exe " . $star , $aout, $returnval);
if ($aout[0][0] === "Y") {
return true;
} else {
return false;
}
}
?>
.

Relevant Pages

  • Re: Efficiency questions: combined ifs and looping 4 times
    ... Choice of algorithm always has a far bigger impact in performance than ... Use sizeof before the loop, store the result in a var and use ... speaks well of PHP. ... your order of complexity analysis is off. ...
    (comp.lang.php)
  • Re: PHP [win32] & CLI Testers needed.
    ... written, lets just say to create shell emulation for now, and have ... My previous attempts to create the source below, ended with the loop ... finally resolve any issues due to this problem in PHP compatibility. ... Debug Warning: testforLinuxUser.php line 24 - stream_select: supplied argument is not a valid stream resource ...
    (php.general)
  • Re: Can anyone explain PHP slowing down please ?
    ... I'm processing a large csv file and when I loop through it I can process around 275 records per second. ... I was wondering if anyone with strong PHP knowledge has heard of this or could help explain it please ... open database for read-only here ...
    (comp.lang.php)
  • Re: open a file once or repeatedly within a loop
    ... Should I worry about the file if my script crashes in the loop while the file is open? ... And wont even hand it over to the operating system at all if it crashes. ... I had extreme problems with one aspect of PHP's file handling - moving an uploaded file and then LOAD-ing it into a database simply failed: the file movement didn't happen at OS level until PHP exited. ... I cant answer for php, but C-wise, fclose really doesn't do more than fflush, except to release the internal memory structures. ...
    (comp.lang.php)
  • Re: [PHP] Performance: While or For loop
    ... >How often do you use a loop of any kind in PHP with enough iterations ... "while" loop. ... it's the same problem as energy savings. ... scalable system for accessing system services | ...
    (php.general)