Re: Fixing socket.makefile()

From: Bryan Olson (fakeaddress_at_nowhere.org)
Date: 08/11/04 

Date: Wed, 11 Aug 2004 02:48:43 GMT

Donn Cave wrote:
[...]
> I don't get this. Has socket.py changed this much since 2.2?
> The readline I'm looking at says self._sock.recv(self._rbufsize),
> so you would only get this behavior if you specified a buffer
> size of 1 or less, and read() does the same - so you could do
> this to yourself, but not specially just with readline.

Hi Donn; yes, looks like I got confused on that one.

> At any rate, I think it would put this in better perspective
> to recall that pipes, terminals and in general any "slow"
> device has the same issues, and that they work out the same
> in Python as in the original C, with socket file descriptors
> in place of socket objects and stdio file pointers in place
> of file objects.

And it gets worse. I've seen layered handlers with buffers of
buffers of buffers.

> It's definitely a problem, and some kind of solution might be
> well received, but it needs to be portable (so forget MSG_PEEK
> unless you're really confident that it will be supported on
> every platform that now supports sockets to some useful degree),

Hold on ... Gooogle...Google...Google... Well, support for
MSG_PEEK seems to be universal except for a couple reported bugs
and versions of BeOS without BONE (BeOS Network Environment).
I've never used BeOS, but apparently BeOS'ers are used to the
idea that they need BONE to get network stuff working.

Actually testing against the wide range of platforms is beyond
my own capabilities.

> and it would be nice to apply to the problem in general and
> not just sockets.

Agreed, but for now I'd like to call that out-of-scope. I came
upon this particular problem when writing an HTTP/1.1 thingy.
The socket module works well, but I found the higher-level
library classes not-so-useful.

> I think the root of the problem really is that
> select() doesn't look at process buffers in fileobject instances,
> and it can't be made to do that because that information isn't
> available from the stdio file pointer underneath the fileobject.
> So, you need a replacement for fileobject, to start with.

Really we want a general, portable, extensible event-handler.
It should to be unified with all the asynchronous things, such
as socket/file activity, thread locks and semaphore, and GUI
event loops. 

-- 
--Bryan

Relevant Pages

  • Re: Sockets debugging tools
    ... This is perfectly possible by programming your peer not ... system buffers are fairly large ... and the socket just remains in a permanent would-block state? ... > returned by recv or send calls. ...
    (comp.unix.programmer)
  • Re: Socket switch delay
    ... buffers are queued and will be later sent. ... socket, not sure about a blocking socket. ... >>(and I noticed you don't do overlapped receive at the server!) ... > optional in the client too). ...
    (microsoft.public.win32.programmer.networks)
  • [PATCH 2.6.23-rc6 Resending] NETWORKING : Edge Triggered EPOLLOUT events get missed for TCP sock
    ... SOCK_NOSPACE bit is set in the socket flags. ... EPOLLOUT events to be missed for TCP sockets, ... No free buffers (all are dispatched to sendfile and are ...
    (Linux-Kernel)
  • Re: Automatic TCP send socker buffer sizing
    ... I've seen in production that some sockets get large very quickly during periods of high latency (eg: when sending to a user downloading from a cablemodem and their headend gets temporarily saturated and has large buffers, which raises the RTT under saturation, which increases the bandwidth delay product), but then as there isn't any code to shrink the buffers. ... This would probably need to be in the timers to notice the case of the sender temporarily stopping sending - eg in a keepalive http socket ... Perhaps we can have a new setsockopt() flag to disable automatic sizing on a socket if we have a specific size we want a buffer to be and don't want it dynamically sized. ...
    (freebsd-net)