Re: Reflections on a classic Lisp Paper

Kent M Pitman wrote:
[ Replying to comp.lang.lisp only.
http://www.nhplace.com/kent/PFAQ/cross-posting.html ]

1) Modern Common-Lisp style macros are not as clearly good
relative to other mechanisms as the rather different macros
he was writing about were, in that several advantages claimed
for the old formulation of macros do not apply to the new
formulations.


Such as...? I'm not being defensive here, just curious.
Claims like this do require evidence though.

Contentions first, arguments later. I'm being didactic,
like people who present abstracts before the body of their
papers. :-) Although, as you mentioned when you actually
got to the arguments, the arguments are weak in the absence
of a functioning fexpr-based lisp.

3) Fexprs in particular, retaining their acknowledged
advantages of applicability and first-class status at
runtime, can be made drastically better than the fexprs this
paper talked about by handling environments explicitly.


This is a possible but somewhat difficult claim to make. In order to
make it fully, there are probably some things you have to do which you
overlook here.

One thing only, which is to produce a working system that isn't
agonizingly dog-slow. As you note, I haven't done it yet.

In particular, you pretty much must count in the
positive effect of being able to do universal quantification across
all special forms, which was the result of the CL decision to follow
up on my paper's thesis and to limit special forms to a fixed set.
This allows the writing of portable code-walkers, and even though no
one has done that, I don't think it's impossible in CL--I just think
it's hard. I have one somewhere that I wrote, for example, and it worked pretty well.

I have one I wrote in scheme, and it works okay as long as it's
R4RS scheme - (shows how long ago I wrote it, I guess) But it
trips hard on implementation-specific extensions, which are a large
part of the game in Scheme dialects. Thing is, I haven't seen very
many compelling examples of utility derived from the ability to
code-walk. Environment access is both more and less tractable in
Scheme, due to the hygienic macros and absence of special variables.

Once you open the space to new
creation, people can't have code-walkers, even if the environment problem
is fixed, unless you create a protocol for understanding and code-walking
new special forms.

Actually, an idea I'm now toying with in that direction is one I
got from the very paper we're talking about. What I really *want* in
Fexprs is that they're applicable and have first-class runtime
existence so they can be stored in variables, returned from functions,
etc. I'm thinking of using your "conversion between" function as an
example and creating fexprs that mirror macros and vice versa, so that
a coexistence of the two can happen. User defines a macro, system
defines a fexpr derived from it, and then the macro source can be used
to drive a protocol for understanding and faux code-walking the fexpr
<insert slides of me waving my hands madly here>.
Both would exist in the system; the simple use cases handled as macros,
and anything that requires runtime existence handled by the
corresponding fexpr. I dunno yet whether this will actually work;
This is a handwave and I'm trying to figure it out still.


And I'm not sure that's possible without a general
purpose AI description language to fully describe the effect of the new special form, or without a reflective formal semantics that can be
dynamically understood and processed by codewalkers to assure that appropriate semantics are maintained. (And the problem there may well
be that once you can do that, it's possible that you've reduced the language
from a fixed set of special forms to some other fixed space dictated by
the semantics and that all you're offering is new syntax--which is all
that macros do. I don't know this for sure--I'd have to read up on
denotational semantics to find out--but it's a "point of due diligence"
I wouldn't let pass before declaring success if I were you).

Right. Don't get me wrong here, I'm not claiming that macros are
a bad or wrong thing -- only that the reasons presented in that paper
no longer support claims of their superiority. Anything else
definitely has to be tried and proven before huge claims can be made.

My arguments of these points are thus:

1) The Macros he was talking about in this paper were
run-time macros rather than compile-time macros. He
cites the ability to use the macro evaluation to alter
the text of the macro form into a non-macro form (expand
once, run many times) or not as the user requires as an
advantage of Macros.


Yes, we found through later experimentation that this wasn't very interesting
computationally and mostly addressed short-term small address space concerns
and so we got rid of the "memoization" kind of macros in CL. No one complained
until now. You need examples to put teeth in your remarks. Claims of this
sort are effectively assertions that there are expressions that cannot be
rewritten properly and efficiently without what you propose, and if you offer
no such challenge expressions, your argument is weak.

I think that is tantamount to saying, "Yes, that turned out not
to be an advantage of macros after all. We didn't miss it when
we got rid of it."

And I'm not complaining about its absence; only saying that the
capability, now abandoned, does not represent a compelling argument
in favor of macros as currently implemented (and apparently not
a very compelling argument in favor of macros as implemented then,
either).


I don't think those that were involved in this change would define it this
way. I think they'd mostly say that twe did two rounds of simplification:
one to remove fexprs and another to remove self-modifying macros since both seemed to add little power and mostly the opportunity to let users confuse
themselves. Your claim that this was ill-advised requires examples.

I'm sorry; that's not what I claimed. What I claimed was that the
reasoning presented in the paper is not an argument which supports
modern-style macros. To ask whether the removal of self-modifying
macrology was ill-advised or not, is a completely different thing,
and I have no real evidence of that one way or another.

But certainly it's wrong to suggest that the "round two" changes as I've
alled them here were done by people oblivious to the motivation of "round one",
and so saying "Lisp has changed so much that we should revamp it" is like
> saying "I decided to repaint my house and then I also repainted the trim
> on the house in a matching color, so I therefore have changed the house so
> much I should go back and reconsider the original change".


Ack! I'm not calling for a revamp; I believe that there should be
experimental dialects, yes, and that the question should be explored
again. But a revamp of existing dialects would be maniacal. Common
Lisp macros work, are well understood and happily used by thousands
of programmers, and are simple and powerful enough for all ordinary
(and most extraordinary) uses!

Every good Lisp dialect is a collection of design decisions that
*WORK* together. Scheme's hygienic macrology works better with
its Lisp-1 semantics than defmacro, but would not serve CL as well.
CL's lisp-2 semantics work better with its defmacro, but would
not serve as gracefully as lisp-1 semantics for the kind of
functional programming that Scheme excels at. And so on. It
would be counterproductive to mess up existing dialects by
introducing features not suited for them and not consistent with
the rest of their design decisions.

I believe that CL and Scheme, which are both excellent dialects,
would be made worse (or completely unrecognizable) by introducing
fexprs into those languages. But while they are self-consistent
and occupy "sweet spots" in the design space, I don't think that
they occupy the *ONLY* sweet spots in the design space, and I do
think that fexprs ought to be revisited as a valid choice by the
experimenters.

> I think these
changes were similarly motivated: to keep users from being caught in complex
code-analysis halting problems, and I think they succeeded. A claim that
they should be re-armed with foot-pointing weapons because users have gotten
smarter and less spazz-prone requires some elaboration.

:-) I don't believe that users have gotten smarter
or less spazz-prone. But I generally think that tools
should NEVER prevent people from doing anything. Maybe
you can just think of me as someone who *Likes* foot-
pointing weapons.

Right now, I'm annoyed that I can't apply or funcall
"and" and "if" and "or" and "lambda" and other important
forms, some of them user-defined. I can't do these
things because an applicable or funcallable or first-
class form of these first-order entities would
necessarily be a fexpr (of some sort) and modern Lisps
don't have, nor want, fexprs.

I think that there *is* a sweet-spot in the lispy design
space for a dialect where syntax (and all other routines)
are defined as fexprs rather than macros-and-functions.
The abolition of the division between macros and functions
is aesthetically tasty, in terms of producing a more
unified and seamless whole. If macros exist at all in
such a lisp, they would exist purely internally, derived
from fexpr definitions, as ways to do the equivalent of
function inlining. Other design decisions would have
to work with that, however, and those design decisions
would create a lisp so fundamentally different that it
could not be Common Lisp or Scheme, nor could it be a
successor to either of those languages.

I think you are underestimating the degree to which you might thwart
the ability to compile.

Well... no. I pretty much know that you would have to keep
a representation of the source code around and that any machine-
code vectors you do produce could be invalidated at the drop of
a hat by further execution of self-modifying code. I'm not
underestimating that. I might be underestimating the importance
of it, but I'm aware of the technical issues.

My reasoning is that the general public used Java 1.2 and found
its performance acceptable. They obviously don't *really* care
about compilation as such, nor execution speed. The fact that
they gripe about Lisp being slow has nothing to do with lisp
actually being slow; they just think it's alien and scary so
they spread long-outdated stories to give each other excuses
about why they don't like it. For most of them, it's a folklore
process, not a real technical requirement.

What I need to do, I think, is produce something that's not quite
as dog-slow as Java 1.2. Interpreted, compiled, or something in
the middle doesn't matter quite so much to me.

Some of what compilers do is to prove by exhaustive analysis that various
effects cannot happen, too. If you create a situation where there is a
proliferation of operators with the potential to do changes that the compiler
cannot know, such proofs can break. The burden is again on you to show
they weill not.

I think I regard that as being on a par with the burden that
static-typing fanatics always throw at Lispers' feet; to show
that type errors cannot happen at runtime and runtime type checks
will never be needed. It may be possible to prove true for
particular programs, and static typing and type inference are
good things for compiler optimizations. With declarations the
class of programs for which it is true may be expanded. But
just as modern Lisps retain the ability to emit code that does
runtime type checks and code that can encounter type errors at
runtime, I believe a fexpr-based Lisp would never be completely
free of the need for the capability to recompile functions on
the fly at runtime. Particular programs may be provably free
of this need (and may satisfy static type requirements too),
and the development tools should help you produce such programs
if that is your wish - but the language would need the ability
to do runtime partial recompilations, the same way other Lisps
need the ability to do runtime typechecks.

I don't think that's a showstopper, any more than most lispers
think runtime type checks are a showstopper. Perhaps I am a
maniac.

Bear

.

Relevant Pages

  • Re: Reflections on a classic Lisp Paper
    ... lisp dialects. ... user-defined special forms - Macros, Fexprs, and Nlambda. ... compilable because the compiler just couldn't do it. ...
    (comp.lang.lisp)
  • Re: A question (confusion) about closure
    ... If not in the first place, emacs lisp code could have evolved if elisp ... more modern tool underneath would not be difficult, because the design ... If you already have implemented some OO patterns manually, ... macros if needed. ...
    (comp.lang.lisp)
  • Re: New book about Common Lisp: Let Over Lambda
    ... I have read every lisp book I've been able to get my hands on ... Lisp macros are not given nearly enough ... I personally consider Let Over Lambda to be an unofficial sequel ... integral parts of the language. ...
    (comp.lang.lisp)
  • Re: LISPPA
    ... > which describes the unification algorithm in Scheme. ... But let's get back to why I think lisp is superior ... optimization macros or reader ... cases where a running program cannot continue running, ...
    (comp.lang.lisp)
  • Re: Macro functionality at runtime?
    ... we don't know what the subtree is yet. ... "Returns a list of tokens matched to subtree. ... > be optimized out by the lisp compiler. ... >> If we have first class macros, ...
    (comp.lang.lisp)
Loading