Re: IO::Pipe and loss of data
- From: xhoster@xxxxxxxxx
- Date: 05 Jan 2006 21:51:22 GMT
"Genevieve S." <none@xxxxxxxx> wrote:
> I choped it as much as it was possible, but it did not yet show up the
> error. May be I cut too much, but as the last run of my big script was
> fine for 6 hours before the error appeared I think it is just not easy to
> reproduce the error.
Bummer. That always makes for hard to find problems.
> Anyway I put some code at the end of this message, which should run and
> show at least how I work to send a message and use the pipe.
In your sample code, the message is always short. Is it always about that
size in the real code?
....
> ******************************************************
> These are the remains of my code. But please use with care. It is an
> endless loop so you have to stop it manually and if it runs to long, it
> will have an enormously amount of data in the pipe
That isn't how pipes work. There is a fixed amount of buffer (probably 4K
on your system) for the pipe. If the reader can't keep up with the
writer(s), then that buffer gets full, and writers will block until the
reader has a chance to catch up, making more room in the buffer.
> (this is because in
> original script the line to be sent is assembled after a line of a
> logfile is read via tail. In this version there is always the same line
> to be send, which is very fast and does never have a break between
> lines).
This non-CPU boundness of the real code might be the problem. See comments
below near the eval.
>
> Sadly I must say I did not yet received the error with this...
> As logging of the script is nearly the same in my big one, I'd like to
> show you what I got yesterday:
> my_log>>>>>
> Wed Jan 4 13:22:17 2006 script started 24687
> [...]
> Wed Jan 4 13:49:56 2006 log reader started for 3070: 14508
> Wed Jan 4 19:40:27 2006 Server 3201: Missing message between 54 and 56
> Wed Jan 4 19:40:27 2006 Server 1665: Missing message between 38 and 40
> Wed Jan 4 19:40:30 2006 Server 4002: Missing message between 9 and 11
> Wed Jan 4 19:40:40 2006 Server 3022: Missing message between 52 and 61
> Wed Jan 4 19:55:25 2006 Server 3025: Missing message between 70 and 74
I would make it log the just-received line in it's entirety, not only the
serial number of that line. In fact, the main program should probably
store not just the previous message-id, but the entire previous message,
for each "server". Then, when there seems to be a gap, you can print out
the entire message before, and the entire message after, the gap. That
could help a lot in debugging.
> sub logProcessing{
> my $line = '';
> my ($msg, $server);
> eval {
> local $SIG{ALRM} = sub { die "lesedauer" };
> alarm(1);
> $line = <$pipe_log>;
> alarm(0);
> };
I don't understand the point of this time out. If the timeout occurs,
you never complain about it or even detect it. You merely start another
call of the same thing that just timed out. If all you do upon a timeout
is restart the same thing, why bother timing out?
Anyway, if the alarm goes off after the next line is read from $pipe_log,
but before the "alarm(0)" gets executed, then you die out of the eval{} and
the line you just read from $pipe_log gets ignored. That could account for
your missing data!
>
> return if ($line !~ /::/);
Wouldn't it be better to die upon malformed data?
> ($msg, $server, $line) = split(/::/, $line);
>
> if(!exists $mshash{$server}){
> $mshash{$server} = $msg;
> }else{
> if($msg != $mshash{$server} +1){
> writeLog("Server $server: Missing message between $mshash{$server} and
> $msg");
I would add $line, if not $mshash2{$server}, to the logged info.
> }
> $mshash{$server} = $msg;
$mshash2{$server} = $line; # so prior line can also be logged
> }
> } # end of sub: logProcessing
Xho
--
-------------------- http://NewsReader.Com/ --------------------
Usenet Newsgroup Service $9.95/Month 30GB
.
- Follow-Ups:
- Re: IO::Pipe and loss of data
- From: Ina Scherf
- Re: IO::Pipe and loss of data
- References:
- IO::Pipe and loss of data
- From: Genevieve S.
- Re: IO::Pipe and loss of data
- From: Genevieve S.
- IO::Pipe and loss of data
- Prev by Date: Re: CGI/sendmail help
- Next by Date: Re: Converting between Local and Unix time problems
- Previous by thread: Re: IO::Pipe and loss of data
- Next by thread: Re: IO::Pipe and loss of data
- Index(es):
Relevant Pages
|
