Re: non blocking read()

From: Donn Cave (donn_at_u.washington.edu)
Date: 12/02/04 

Date: Thu, 02 Dec 2004 11:18:09 -0800

In article <317ec2F388a2eU1@individual.net>,
 Greg Ewing <greg@cosc.canterbury.ac.nz> wrote:

> Donn Cave wrote:
> > If we are indeed talking about a pipe or something that
> > really can block, and you call fileobject.read(1024),
> > it will block until it gets 1024 bytes.
>
> No, it won't, it will block until *some* data is
> available, and then return that (up to a maximum of
> 1024).

You can test this on your platform, I will append
a 9 line program. For me, fileobject.read blocks
until all the requested data can be returned.

> If the fd has just been reported by select() as ready
> for reading, then something is there, so the first
> read() call won't block. But if there was exactly
> 1024 bytes there, the second read() call *will* block,
> because there are now 0 bytes available (which I think
> is what an earlier poster was hinting at).
>
> For this reason, if you have no way of knowing how
> much data to expect in advance, it's better to avoid
> making more than one read() call on a fd per select().
> If you don't get a whole line (or whatever chunk you're
> looking for), put what you've got into a buffer, and
> go back to select(). When you've built up a complete
> chunk in the buffer, process it. Keep in mind that
> part of the next chunk may be in the tail of the
> buffer, so be prepared to chop a chunk off the
> beginning of the buffer and leave the rest for later.

Yes, this looks right to me, but I think we're talking
about os.read(), not fileobject.read().

> Another possibility that's been suggested is putting
> the fd into non-blocking mode. I wouldn't recommend
> that; the last time I tried it (which was quite a long
> time ago) select() and non-blocking I/O didn't mix
> well. While it may be possible to get it to work, I
> don't think you'd gain much. You need to understand
> that there's no guaranteed relationship between the
> chunks of data written to one end of a pipe or socket
> and those returned by reading the other end. So you'd
> still need to be prepared to buffer and re-chunk the
> data. You'd end up doing all of what I outlined above,
> with the extra complication of non-blocking I/O thrown
> in. I don't see any advantage in it.

Exactly.

   Donn Cave, donn@u.washington.edu

Relevant Pages

  • Re: non blocking read()
    ... If you don't get a whole line (or whatever chunk you're ... chunk in the buffer, process it. ... chunks of data written to one end of a pipe or socket ... with the extra complication of non-blocking I/O thrown ...
    (comp.lang.python)
  • block on reading a half-filled buffer for ifstream
    ... Only when the buffer becomes full does ... Here's my test code for reading from a pipe: ... the program blocks until BUFF_SIZE chars have ...
    (comp.lang.cpp)
  • Re: IO::Pipe and loss of data
    ... >> It looks like you assume that writing to a pipe is atomic; ... > data from other processes doing writes on the same pipe. ... currently available in the pipe buffer. ... (with a chunk number and PID/TID embedded into the chunk if more than ...
    (comp.lang.perl.misc)
  • Re: Popen and EVFILT_WRITE question
    ... (It doesn't matter if I open the pipe with w+ or r+). ... cause it should fire since there's space in the buffer. ... The kernel knows that the fd at the end of the pipe is blocked for reading. ...
    (freebsd-hackers)
  • Re: writing to pipe
    ... number of bytes writable to the pipe? ... implimentation defined if nbytes> SSIZE_MAX, ... is sufficient space in the buffer, write all of the bytes and return ... more than PIPE_BUF at a time, the entire chunk will come through ...
    (comp.lang.c)