The Hyperspec and portability between Common Lisp compilers (long)

From: Antonio Menezes Leitao (Antonio.Leitao_at_evaluator.pt)
Date: 04/28/04 

Date: Wed, 28 Apr 2004 22:51:38 +0100

Hi,

Porting Linj (my Lisp to Java compiler) to several Common Lisp compilers
has been an "interesting" experience. I'll use this post to report the
results I got so far.

Originaly developped in CMUCL, my first attempt was to port
it to OpenMCL. Unfortunately, there were some MOP changes between
the last two OpenMCL versions and I didn't know which version to use.
I tried to install the last version but couldn't run it (I guess I
didn't patch it properly) so I changed to the next to last version
which was OK. Unfortunately, I didn't had much time available to use the
Mac so, after some time trying to solve compilations errors, I had to gave
up. I'll try again latter.

CLISP was my second attempt. The first bug I found was the empty list on
macro lambda lists that I reported previously. CLISP doesn't accept them.
I know that there are conflicting points of view in this issue but I think
CLISP is wrong about this. I really would like to know what are they
arguments.

After adjusting my code to CLISP's taste, I found the second bug: I had a
macro character associated to a reader macro function that was returning
two values (not because I decided that way but because I was using a
function that returns two values (e.g., read-from-string)).

It happens that, according to the Hyperspec:

  Upon encountering a macro character, the Lisp reader calls its reader
  macro function, which parses one specially formatted object from the
  input stream. The function either returns the parsed object, or else it
  returns no values to indicate that the characters scanned by the
  function are being ignored (e.g., in the case of a comment).

Does this means that the reader macro function can _only_ return one or
zero values?

Or does this means that the reader macro function can return many (as long
as the first value is the parsed object) or zero values?

CMUCL prefers the second interpretation. CLISP only accepts the first and
gives an error if you dont follow it. What's the correct interpretation?

I decided to follow CLISP interpretation just to be compatible with both
compilers). After this, I found several problems but, this time, I'm sure
they are real bugs in CLISP as I couldn't find a way to reconcile CLISP
implementation with the Hyperspec. The bugs are related to pretty
printing streams and, unfortunately, they are severe enough to prevent the
Linj compiler of working correctly in CLISP.

As a result, I postponed the port to CLISP until the CLISP implementors
solve those bugs.

The next Common Lisp compiler in my list was Allegro. Apart from the
effort needed to translate operating system dependent calls (spawning
external processes, sockets, etc) no problems where found. At least I
don't remember them so if there were some, they weren't serious. I do
remember that I changed the compilation order of some classes and methods
but I don't remember if it was to avoid warnings or if there were
compilation problems. Nothing relevant to report here.

Then, I moved to Lispworks. Everything OK except that Lispworks
isn't affected by the readtable-case while printing with escaping
disabled (e.g., by princ-to-string). This is contrary to what is
mandated by section 22.1.3.3.2 of the Hyperspec. Xanalys agree with me
that this is a bug and they told me they will solve this problem in the
next version of Lispworks. Unfortunately, this issue is fundamental for
the Linj compiler so I can compile Linj in Lispworks but I can't run
it properly.

What's next? SBCL. Given the fact that SBCL originated from CMUCL, I
thought this would be easy. Unfortunately, I can't compile my code
(easily) in SBCL because of problems in defconstant forms. Let me explain
this carefully:

Let's assume a file containing the form:

(defconstant +foo+ (cons nil nil))

Due to some dependency problems (taken care in a defsystem-like
definition), this file must be compiled and loaded before other files can
be compiled. So SBCL compiles it, then proceeds to load it and it barfs
with an error when it sees that, between compilation and loading, the
value assigned to +foo+ changed.

Now, according to the Hyperspec (Macro defconstant):

  If a defconstant form appears as a top level form, the compiler must
  recognize that name names a constant variable. An implementation may
  choose to evaluate the value-form at compile time, load time, or both.
  Therefore, users must ensure that the initial-value can be evaluated at
  compile time (regardless of whether or not references to name appear in
  the file) and that it always evaluates to the same value.

So, it looks like the initial-value can be evaluated both at compile time
and at load time and both evaluations must produce the same value (if I'm
not mistaken, "same value" here means "eql"). If my reasoning is correct,
then SBCL is conforming to the Hyperspec.

Now, my question is: How can I use the defconstant form with an initial
value that is computed (at load time) on a file that is compiled and
loaded in the same Common Lisp image? It seems to me that the Hyperspec
is restricting the uses of defconstant in ways that simply prevents its
use for non-trivial initial values. Or I'm I misunderstanding the
Hyperspec?

It's also strange that neither CMUCL, nor CLISP, nor Allegro, nor
Lispworks have this behaviour.

Sorry for the long post and thanks for your comments.

Best regards,

Antonio Leitao

PS: A long time ago I got a lot of experience translating programs between
Lisps (including FranzLisp, LeLisp, ZetaLisp and Common Lisp). I was
quite happy when Common Lisp became a standard because the problems I had
making my code compatible with several Lisp implementations were over.

After my attempts to port Linj to different Common Lisp compilers, my
faith in the portability of Common Lisp programs is weaker now. I didn't
expect to find so many problems. I'm starting to believe that reference
implementations are better than written specifications.