CLISP + Araneida issues.

There is some buggy behavior in the interaction between Araneida and
CLISP, whose ultimate consequence is the failure of proper page flow
due to aborted TCP connections.

One way to sweep the buggy behavior under the carpet is to turn on the
use of buffered streams.

Bug 1: unbuffered I/O (Araneida issue)

For some strange reason, the clisp-compat.lisp module in Aaraneida
requests CLISP to make a :BUFFERED NIL socket stream, which is crazy:
all the reads are one byte at a time! Sucking bits through a straw.
Okay, this is more of a performance problem than an actual bug. But
thanks to this, other bugs are exposed.

Bug 2: line endings (CLISP issue)

Araneida also requests CLISP to make the stream with an ISO-8859-1
encoding, for 8 bit clean transport, and in the encoding it requests a
:UNIX handling for line endings.

However, this :UNIX mode does not work the way Araneida expects. It
expects to receive the individual CR and LF characters. The CLISP
socket stream stream still does some bogus CR-LF recognition in spite
of the UNIX mode. When READ-CHAR sees a CR, it returns a newline
character to the caller and leaves the following linefeed unread in the
socket.

A strange behavior is seen when the body of a HTTP POST is received
according to the Content-Length. Because CLISP has not read the newline
before the body, the READ-SEQUENCE results an off-by-one read. The
read() request is performed for the exact Content-Length. But it does
not begin at the start of the body, but rather at the newline which
precedes the body! Somehow the unbuffered CLISP stream realizes this,
and performs an extra one-byte-read to fetch the last byte of the body,
so the string which holds the entire body is filled correctly.

Even though the body is fetched properly, the way it's done is just too
strange. CLISP should just have the proper support for the requested
line endings and not play any silly games. So I regard this as a bug.
In a :UNIX line ending mode, CLISP should just return any carriage
return as a carriage return character to the caller.

Bug 2: Extra data from a browser is not handled properly. (Browser,
Araneida)

I see cases when a browser writes an extra newline after the HTTP POST
(in direct contravention of the HTTP RFC's!!!) That is to say, the body
is there to the exact content length, and then there is an extra
linefeed. Because the CLISP stream is in an unbuffered mode, it ignores
this newline, even though it probably arrived in the same TCP segment
as the previous bytes.

So what happens is that since Araneida is a HTTP 1.0 implementation
with no persistent connection support, it replies to the request and
then closes the connection. But because it has not read that one last
byte from the offending HTTP client, the underlying TCP stack turns the
close into an abort. A RST segment is sent to the browser, and the
ensuing behavior depends on race conditions. I find that over local
loopback TCP, the page is usually read properly. Between two machines,
the response page usually won't load. The browser either hangs and
times out, or reports an error.

If you turn on buffering for the CLISP stream, the problem goes away.
Or at least its probability becomes much smaller. The CLISP socket
snarfs all of the available data which is smaller than its 4096 byte
buffer. For the problem to reproduce with buffering, the extra byte
would have to arrive late, in its own TCP segment, so as not to be
included in the buffer fill operation.

There is no easy way to fix this, because a receiver has no control
over the timing of extra erroneous data on the connection. Checking for
it results in a race condition. Still, the server should at least try.
Before doing the close, poll the socket for any pending unreceived
data. Also, doing a half-close wouldn't be a bad idea: just shut down
the socket in the writing direction, and then read data from the client
until /it/ closes, or some reasonable time limit is exceeded. A
half-close won't abort the connection even if there is unread data. I'm
going to try implementing this fix.

I also started hacking Araneida to do HTTP 1.1 properly, with chunked
transfer-coding and persistent connections, so this won't be an issue
anyway. (If I finish this work, I will test it on various
implementations, with and without threads).

.

Relevant Pages

  • Re: Specific questions from Lisp newbie (Please dont taunt the fresh blood)
    ... ucw has recently added support for clisp with araneida in ... the development source. ...
    (comp.lang.lisp)
  • Re: Problems with Araneida
    ... >> I'm working on CLisp, with Emacs and the lsat version of Araneida. ... >> when I load the example.lisp and try to access the server throw IE, ...
    (comp.lang.lisp)
  • Re: Clisp Slime aborts
    ... slime loses the connection with Clisp. ... Lispbox doesn't do this, it just drops me back into the REPL. ... Maybe these versions of Slime are different? ... the EmacsCLISP socket connection is obviously closed. ...
    (comp.lang.lisp)
  • Re: Socket Programming
    ... Johannes Groedem wrote: ... or a way to do it within the confines of CLISP? ... whether a connection is available on a SOCKET:SOCKET-SERVER". ... operating under the impression that socket:socket-status would only ...
    (comp.lang.lisp)