Re: question about forked processes writing to the same file

it_says_BALLS_on_your forehead wrote:
Gunnar Hjalmarsson wrote:
it_says_BALLS_on_your forehead wrote: 

use strict; use warnings;
use Parallel::ForkManager;

my $pm = Parallel::ForkManager->new(10);

# assume @files contains 100 files that will be processed,
# and processing time could range from subseconds to hours

my $out = 'results.txt';
for my $file (@files) {
 $pm->start and next;

 # some code to process file
 # blah blah blah

 open( my $fh_out, '>>', $out ) or die "can't open $out: $!\n";
 print $fh_out "$file\n";
 close $fh_out;

 $pm->finish;
}
$pm->wait_all_children;


As long as you don't care about the order in which the output from
respective file is appended to $out, I can't see what the problem would be.


order is unimportant at this juncture. i was concerned if one process
would interrupt another that was writing, so that either both failed,
or a single entry became a garbled hybrid of two entries...something
along those lines. these, or other cases that may interfere with
writing one entry per line to $out, are what cause me apprehension.


Probably somebody more knowledgable than me should comment on your concerns.

Awaiting that, have you read

    perldoc -q append.+text

I can add that I'm using the above method in a widely used Perl program (run on various platforms), without any problems having been reported. However, I do use flock() to set an exclusive lock before printing to the file. Can't tell if locking is necessary.

--
Gunnar Hjalmarsson
Email: http://www.gunnar.cc/cgi-bin/contact.pl
.

Relevant Pages