Re: Scheme macros

From: Anton van Straaten (anton_at_appsolutions.com)
Date: 02/23/04 

Date: Sun, 22 Feb 2004 23:03:46 GMT

Pascal Costanza wrote:
> Anton van Straaten wrote:
>
> > Pascal Costanza wrote:
> >
> >>To me, the conceptual simplicity of CL-style macros is striking: It's
> >>just a transformation of s-expression. That's it.
> >
> > That's what all of these macro systems are.
>
> R5RS doesn't say so. At least, I don't see where the term "macro
> transformer" is defined. It seems to me that the standard tries hard to
> hide that fact. (But I might simply not have found the relevant sections.)

If you look up "macro transformer" in the index, it points you to a page
which contains the following definition (Sec. 4.3, Macros):

"The set of rules that specifies how a use of a macro is transcribed into a
more primitive expression is called the 'transformer' of the macro."

I don't think it's hiding anything. Do you think otherwise?

> >>Once understood, it's clear that you can do anything
> >>with this conceptual model.
> >
> > The same is true of syntax-case.
>
> Of course, I will take your word for that. But I still don't understand
> what syntax-case does. I have browsed through the various links that are
> usually referred (mainly papers and a book by Dybvig), but I find it
> very hard to follow the contents. It would be good if there would exist
> some kind of high-level overview about syntax-case for people who
> already know DEFMACRO.

I agree the docs don't make it easy to get into at first. I learned
syntax-case (up to a point - I'm not an expert by any means) after first
learning and using syntax-rules for some time, and having previously been
familiar with defmacro. I think syntax-rules makes a good starting point,
because it teaches the hygienic pattern matching approach in a simpler
context. That same approach is used by syntax-case, but augmented with a
much more powerful procedural syntax manipulation capability.

But I'll ignore my own advice and take a stab at explaining syntax-case,
starting from a defmacro perspective. Perhaps the gentlest introduction to
syntax-case is Dybvig's paper, "Writing Hygenic Macros in Scheme with
Syntax-Case":
ftp://ftp.cs.indiana.edu/pub/scheme-repository/doc/pubs/iucstr356.ps.gz ,
and I'll use some of its examples below.

Start with defmacro, and imagine that instead of quoting syntax using
quasiquote, you use a special form, 'syntax', which instead of returning a
list as quasiquote does, returns a syntax object, i.e. an instance of an
abstract data type which represents a piece of syntax. This type has an API
which supports various code walking & manipulation capabilities. It can
also be converted to a list (or whatever value the original syntax
represented) via 'syntax-object->datum'.

An important thing to note here is that a syntax object "understands" the
syntax it represents - it's not just an undifferentiated list. It knows
which values are identifiers, it knows things about where those identifiers
are bound, and as we've touched on, it can track things like line numbers
(which may be implementation-dependent). If you're developing code
manipulation tools - editors, debuggers etc. - these syntax objects give you
a capability which defmacro doesn't even attempt to address. Syntax objects
are a richer way to represent program syntax than lists, and their uses go
beyond just macros.

Within a syntax expression of the form (syntax ...), any references to macro
variables are replaced with their values, i.e. there's no need to unquote
references to macro variables. So to steal an example from Dybvig, here's
an 'and2' macro which works like 'and', but for only two operands:

(define-syntax and2
  (lambda (x)
    (syntax-case x ()
      ((_ x y)
       (syntax (if x y #f))))))

The (lambda (x) ...) binds a syntax object to x, representing the syntax of
the expression which invoked the macro. In theory, you can do whatever you
want with that syntax object. Most commonly, you'll use syntax-case to do a
pattern match on it, which is what the above example does with the
expression (syntax-case x () ...). The () is for literals, like 'else' in
cond.

Within the above syntax-case expression, there's a single pattern and a
corresponding template:

      ((_ x y)
       (syntax (if x y #f))))))

The underscore in (_ x y) represents the name of the macro - you could also
write (and2 x y), it doesn't matter. This pattern will match any
invocations of AND2 with two parameters. After the pattern, is the
expression which will be executed when that pattern is matched. In this
case, it's simply a syntax expression which returns the expression (if x y
#f). The return value from syntax-case must be a syntax object, which
represents the actual syntax of the final program.

In the above, since x and y are macro variables (strictly speaking, "pattern
variables"), their values will be substituted when the syntax object is
created, so that (and2 4 5) becomes (if 4 5 #f).

I think I'll stop there for now, since I have other things to do! I've
touched on some of the more important points about syntax-case. There's a
lot more to it than the above - particularly, breaking hygiene, and
executing procedural code rather than simply applying a template. But all
of it fits into the above framework, and involves manipulating syntax
objects, in a way similar to what you would do in defmacro, but through the
syntax object API rather than manipulating lists. There are plenty more
examples in the Dybvig paper.

> > Syntax-rules is not hard to learn. If anything, it suffers from being
> > almost too simple; as well as from lacking good, short introductory
> > material. You specify a pattern, and specify the syntax that should
replace
> > that pattern. That's all there is to it.
>
> Examples like those given in
> http://okmij.org/ftp/Scheme/r5rs-macros-pitfalls.txt seem to indicate
> that syntax-rules just trade one set of possible pitfalls with a
> different set, but along that way the conceptual simplicity is lost.

I don't accept that "conceptual simplicity is lost" with syntax-rules. It's
a different approach, which in some ways is conceptually simpler than
defmacro, since it doesn't require the user to manually keep track of the
different levels at which the macro operates. The pitfalls you mention may
indeed be flaws in syntax-rules - I'm not familiar enough with them to
comment - but I find that syntax-rules works very well for many kinds of
macros, better than defmacro in fact.

Of course, the latter claim is hardly ever going to be accepted by someone
only familiar with defmacro. For the record, I learned and used defmacro
before ever using syntax-rules or syntax-case, and I still use defmacro from
time to time, so I think I have a good basis for comparison.

> Here are the examples from that reference implemented with DEFMACRO:
>
> (defun foo-f (x)
> (flet ((id (x) x))
> (id (1+ x))))
>
> (defmacro foo-m (x)
> `(macrolet ((id (x) x))
> (id (1+ ,x))))
>
> (defmacro bar-m2 (var &body body)
> `(macrolet ((helper (&body body)
> `(lambda (,',var) ,@body)))
> (helper ,@body)))
>
>
> I really don't see the problem. Seriously not.

I'm not sure what you mean about not seeing the problem. One of the
problems mentioned in the article is that syntax-rules pattern variables
don't shadow. I don't know if there's a justification for that, or it's
simply a bug in the design of syntax-rules. But you usually get an error if
you make this mistake, and it's easy to fix, and easy to avoid. It doesn't
mean that syntax-rules is not useful, and it's still better than defmacro,
which you can't dispute until you've learned syntax-rules. ;)

> > Syntax-case is more complex, and I do think that's a drawback when
compared
> > to defmacro. It increases the temptation to conclude the following:
> >
> >>And it immediately makes me wonder whether it is really worth it.
> >>After all, I know how to make things work with DEFMACRO.
> >
> > I might wonder something similar if I were a Python programmer looking
at
> > Lisp: Lisp seems hard to learn, and I would know how to make things work
> > with Python.
>
> Lisp and Scheme bring you metacircularity. As soon as Pythonistas write
> program generators, it's clear that their laguage is missing something
> important. Of course, they can write a Lisp interpreter in Python, but
> that's besides the point.
>
> Do you really think that syntax-case is an equally important step forward?

In some respects, yes, but that's not what I really meant. It's easy to
look at something from the outside and find reasons not to try it, and that
was my main point. But the points you've been picking on don't seem very
substantial to me - it seems as though you're looking for reasons to ignore
these systems, rather than looking for reasons you might want to learn them.
To conclude from the points you've raised that these systems can be ignored,
seems to me to throw a bunch of babies out with the bathwater. (They're
much cuter babies than that warty defmacro baby, too! ;)

Of course, a CL programmer who wants to write standard CL code, obviously
has little incentive to be interested in other macro systems. But if your
interests cross multiple languages, then there's value in the Scheme macro
systems, at the very least in the sense that learning more than one approach
to the same problem expands the horizons of your understanding of the
problem.

> >>BTW, what you really need to make something like DEFMACRO work is, on
> >>top of that, of course quasiquotation, GENSYM/MAKE-SYMBOL or
> >>string->uninterned-symbol and most probably a Lisp-2.
> >
> > I don't see that Lisp-2 is an issue.
>
> See http://citeseer.nj.nec.com/bawden88syntactic.html

I'm familiar with why people claim it's an issue, but in practice I think
it's not significantly worse than the issue of hygiene in defmacros in
general. As I've said and defended here once before, Lisp-1 can express any
Lisp-2 program, simply by changing any conflicting names so as not to
conflict - a conceptually trivial transformation, with consequences which
are primarily subjective. It would have an impact on porting Lisp-2 macros
to Lisp-1, but it doesn't limit what you can easily express in Lisp-1.

Put another way, having the ability to accidentally compensate for hygiene
violations in some cases - where multiple namespaces happen to prevent the
problem - isn't a solution to the general problem of not having hygiene.
Since you haven't solved the general problem, you still have to address
questions of hygiene, in various low-level ways. A single namespace doesn't
makes this problem worse in any significant way.

> Here it helps that a Lisp-2 seperates variables and functions by
> default. Variables are usually not important parts of an application's
> ontology. If they are, the convention in Common Lisp is to use proper
> naming schemes, like asterisks for special variables. Effectively, this
> creates a new namespace.

Mmm, asterisks. This, to me, is why the whole Lisp-1/2 debate is moot. The
solution is simply Lisp-N, where you can define namespaces, modules, etc.
and control how they're used. See PLT Scheme etc.

> 4. The fourth example can be solved with a proper GENSYM for "use" in
> the "contorted" macro.

The phrase "proper GENSYM" is an oxymoron. GENSYM operates at a strangely
low level of abstraction. Why don't you use GENSYM when declaring normal
lexical variables in a procedure? Rhetorical question, of course - the
point is, GENSYM is a kludge. It's not a particularly onerous one, but it's
part of what makes defmacro worth improving on.

> Some while ago, I wanted to experiment with continuations in Scheme.
> Apart from the fact that not all Schemes seem to implement continuations
> fully and/or correctly (see
> http://sisc.sourceforge.net/r5rspitresults.html ), the fact that the
> respective documentations make me feel uneasy about whether I have to
> relearn programming techniques for totally unrelated areas is a clear
> downside IMHO.

We're straying far afield here. ;) But I'll give my opinion about
continuations, too. Re the quality of implementations, once again you're
looking at edge cases. Forget about those, they're not important, except
in, well, edge cases. All of the major Schemes either support continuations
well, or tell you when they don't - e.g., some of the Scheme to C compilers
deliberately provide restricted continuations.

As far as relearning programming techniques goes, first and foremost,
continuations are a general conceptual model for control flow. If you only
write single-threaded code in a language with a traditional linear
stack-based control flow, you won't have much use for continuations -
they're far more powerful and general than is needed to deal with that case.
But for systems with more complex control flow, continuations can provide a
very useful model - web servers are just one example, but really any system
which involves multiple threads, distributed processing, etc. can benefit
from modeling via continuations.

Scheme is one of very few languages - along with SML/NJ, Stackless Python,
and the RhinoWithContinuations version of Javascript - which implements
first-class continuations. If you're developing tools in the spaces
mentioned above, this is a useful capability. Stackless Python uses
continuations to support microthreading; Scheme has a number of web server
solutions which use continuations to "invert control" so that the structure
of a web application's code can be decoupled from its web page structure;
and RhinoWithContinuations does something similar for web applications in
the Cocoon web framework. For applications which need them, continuations
are very useful, and have little competition. Their competition is mainly
OS-level threads, which really solve a different problem, and conceptually
are a stream of continuations anyway.

For ordinary programming, though, continuations are more or less
irrelevant - they should be dealt with under the hood, whether by tools like
those web server frameworks, or by language constructs like exception
handlers and microthreads. The only reason to learn about use of
first-class continuations as a programming construct is either for the sake
of learning, to deepen your understanding of programming; or if you are
interested in developing language or system tools that use them. If you are
interested in any of this, then yes, you're going to have to do some
learning and relearning - there's no way around that. But for most ordinary
applications, you can safely ignore continuations.

> [...]
> > The result is that it's actually easier to reason about syntax-rules
> > macros - which makes them easier to write, and easier to read. As a
result,
> > and also because of the enforced hygiene, they're less error-prone.
>
> I don't mind using DEFMACRO for simple things. I don't find them hard to
> write or read, and I don't know why they would be more error-prone.
> Sounds similar to some of the claims made by advocates of static type
> systems. Maybe this boils down to just a matter of taste.

Maybe - let's talk once you've tried syntax-rules. But you gave a clue to
your reading & writing process for DEFMACRO when you said that when reading
a syntax-rules macro, you were immediately worrying about which level the
various tokens were at. You've learned to look for, and expect something
that, with syntax-rules, you can simply forget about. You don't do these
things when writing ordinary functions - why do you put up with it when
writing macros? What would you think of Lisp if you had to use gensym to
initialize every variable you use? You've simply become very used to a
low-level technique, so that you don't believe there's any need for a higher
level technique.

> What stands in the way of implementing syntax-case on top of DEFMACRO?
> (This is not a rhetorical question.)

I don't think it would make much sense. The implementation of syntax
objects has little do with what defmacro does. The pattern matching forms
of syntax-case might be defined via DEFMACRO at the surface level, but their
definitions deal with syntax objects, so there'd be little for defmacro to
do once the syntax objects had been constructed. It wouldn't help much when
constructing the syntax objects, either, since the 'syntax' form doesn't use
defmacro syntax, and I can't see any point in converting it internally.

The reason that it's easy to implement DEFMACRO in syntax-case is that a
syntax object is a superset of the list representations of syntax used by
DEFMACRO. You can translate syntax-as-lists to syntax objects and back
again, without losing anything - it's part of the standard syntax object
API, so nothing additional is needed to do that, which is partly why a
syntax-case implementation of DEFMACRO is short. Going the other way is
more problematic, since syntax-as-lists has less information than a syntax
object.

Anton

Relevant Pages

  • Re: Why is this macro misbehaving?
    ... There's a similar macro in araneida ... So start by writing out an example of the syntax you want to be able to ... (defun %with-url-function (arglist seqexpr body) ... in defmacro definitions. ...
    (comp.lang.lisp)
  • SetText Method Confusion
    ... what is the correct syntax for the SetText Method? ... and neither will complile when I use them in my macro. ... before pasting the changed string to the new document. ... into the clipboard, from the clipboard into a DataObject, ...
    (microsoft.public.word.vba.beginners)
  • Re: [ Attn: Randy ] Ad-hoc Parsing?
    ... unless 'Agaga' is defined as a macro elsewhere. ... I recommended a different syntax for equates ... > I don't yet know enough about the final LuxAsm ...
    (alt.lang.asm)
  • Re: Why is LISP syntax superior?
    ... that lisp was the most powerful programming language ever invented, ... It is precisely the lack of syntax that I admire in Lisp ... (setf (elt seq i) ... I guess one could write a reader macro to transform foointo (elt ...
    (comp.lang.lisp)
  • Re: scheme seems neater
    ... Any macro facility that proposes to address this ... Some other Scheme macro systems don't, ... this, you need "hygiene", and this is the sense in which I consider hygiene ... Those constructs operate on a specific data type, the syntax object. ...
    (comp.lang.lisp)
Loading