Re: What's up with Scheme macros?

Ron Garret <rNOSPAMon@xxxxxxxxxxx> writes:

In article <uve4jvgew.fsf@xxxxxxxxxxx>,
Kent M Pitman <pitman@xxxxxxxxxxx> wrote:

Note that the spec for macrolet says the same thing, but it includes
the following exception, which is NOT part of the spec for defmacro:

"the consequences are undefined if the local macro definitions
reference any local variable or function bindings that are visible
in that lexical environment."

Actually, might be effectively a bug in the spec since it really
cannot work if you reference any block or tag either.

What is the bug? The fact that blocks and tags are not mentioned, or
that this exception is given for macrolet but not defmacro? (BTW, I
thought tags were part of the dynamic environment, not the lexical
environment.)

Ok, let's back up because I said a couple of things without checking
with the spec and ended up saying some incorrect things and I want to
correct what I said before I confuse anyone further. I'll have to go
back and re-analyze what I responded to see where I confused matters,
but for now let me just say some remarks that basically correct what I
said wrong.

And also, I apologize for doubting you here.

I'll come back to the remark above about consequences undefined. That
is easier to understand in context after I get to the other part.

It's easiest to understand both DEFUN and DEFMACRO if you think of them
as being defined approximately thus:

(defmacro defun (name bvl &body decls-doc-forms) ;ONLY APPROXIMATE!
(destructuring-bind (decls doc forms)
(parse-definition-body decls-doc-forms)
`(progn
(eval-when (:execute :load-toplevel)
(setf (symbol-function ',name) #'(lambda ,bvl ,@decls ,@forms))
(setf (documentation ',name 'function) ,doc))
(eval-when (:compile-toplevel)
;; Various implementation-dependent things compiler notices if form
;; is at toplevel...
(tell-compiler-function-definition-seen ',function ',bvl ',decls)
(tell-editor-how-to-indent ',function ',bvl)
;;...etc.
',name))))

The situation is similar for defmacro (albeit metacircular):

(defmacro defmacro (name bvl &body decls-doc-forms) ;ONLY APPROXIMATE!
(destructuring-bind (decls doc forms)
(parse-definition-body decls-doc-forms)
(let ((form (gensym "FORM"))
(env (or (cadr (member '&environment bvl))
(gensym "ENV")))
(macro-definition
`#'(lambda (,form ,env)
(declare (ignorable ,env))
(destructuring-bind ,bvl ,form ,@decls ,@forms))))
`(progn
(eval-when (:execute :load-toplevel)
(setf (macro-function ',name) ,macro-definition)
(setf (documentation ',name 'function) ,doc))
(eval-when (:compile-toplevel)
;; macro definition happens at compile-time too!
(tell-compiler-about-macro ',name ,macro-definition)
;; Various implementation-dependent things compiler notices if form
;; is at toplevel...
(tell-editor-how-to-indent ',function ',bvl)
;;...etc.
',name)))))

There are probably flaws in the above. I just made it up as a sketch
to illustrate that there are things that go on when you execute the
definition, and things that go on at compile time. So when you embed
one of these, it does capture the lexical environment, and it doesn't
execute until runtime.

So to undersand what these do when in a LET, you notice that this makes
them not at toplevel, and it means the :execute clause is done. At
toplevel, the :compile-toplevel stuff is done when compiling, and then
the :load-toplevel stuff is done when loading.

So you can see that's why here:

(let ((x (print ''a)))
(defmacro foo () x)
(foo))

there is an issue because (foo) is in a context where the definition inline
will be

(let ((x (print ''a)))
(progn
(eval-when (:execute :load-toplevel)
(setf (macro-function 'foo) ...) ...)
(eval-when (:compile-toplevel)
(tell-compiler-about-macro 'foo ...))
'foo)
(foo))

is being done, and since you're not at toplevel, this as if you did:

(let ((x (print ''a)))
(progn
(progn
(setf (macro-function 'foo) ...) ... nil)
nil
'foo)
(foo))

which you can reduce to:

(let ((x (print ''a)))
(setf (macro-function 'foo) ...)
(foo))

and it should be easy to see that (foo) is not going to win because its
macro definition is not set early enough.

Because you can't do the macro expansion without the value of x.
And you can't know the value of x without evaluating the print.
But the value of x is needed at compile time, since you have to
expand foo to compile it.

No, you don't. The defmacro is not a top-level form, and so it does not
get executed at compile time, and the call to FOO gets compiled as
whatever FOO was defined as before (most likely an call to an undefined
function). See section 3.2.3.1.1 of the spec.

Yes, I think that's right and that I was wrong.

I explained this before:

(Note that because the defmacro is not a top-level form, the call to
FOO will be compiled as a function call unless FOO has already been
defined as a macro before the file is compiled. And even if FOO has
been previously defined as a macro, the call to FOO will be compiled
as a call to the OLD foo, not the new one.)

Then, at minimum, you won't be able to use FOO as a macro right after
its definition. We thought that would confuse people.

That's right. You can't. And it does.

Yes.

Moreover, why would the compiler know to expand (foo) at all since
the definition of FOO as a macro would not be installed until
runtime, if it were to follow what DEFUN does.

Exactly. I don't see what any of this has to do with anything.

Well, it makes LET be a kind of secret DELAY operator for defmacro.
We thought that would be confusing.

That's right. It is.

Heh.

BTW, a top-level LET is a "secret delay" operator for DEFUN too, and
everything else that closes over the lexical environment. If this
doesn't work:

(let ((x ...))
(defmacro foo (...) (... x ...)))


On reflection, I this should work.

What won't work is to use FOO in the same file. But if you've done the
:execute [e.g., due to immediate evaluation], that works, too.

because you don't know what X is at compile time then this can't work
either:

(let ((x ...))
(defun my-macroexpander (...) ... x ...))

(defmacro foo (...) (my-macroexpander ...))

for exactly the same reason.

The problem here is that my-macroexpander isn't available until runtime.
The above will work as long as FOO isn't used in the same file.
It will fail if it is, and it needs
(eval-when (:execute :load-toplevel :compile-toplevel)
...)
around the LET if you want to use the definition of my-macroexpander later
in the same file that the defmacro foo occurs.

This becomes self-evident if you expand a top-level LET into its
constituent lambdaas:

(let ((x initval))
(defwhatever foo .... x ...))

expands into:

((lambda (x) (some-side-effecting-frob foo (lambda (...) ...)) initval)

You'll do better to understand it in terms of eval-when. That may be the
missing piece that helps you understand it.

Note, btw, that compiler version of the macro would have no access to the
lexical variables, so would be serious trouble in the case of a closure,
but since the :compile-toplevel will not activate in that case, it's a
non-issue.

Note further, finally coming back to the deferred topic about local macros,
that a local macro NEVER has a load time--it is always processed at what
amounts to compile time in the pre-evaluation environment, so it must be
possible to capture and use it immediately.

That is,

(defun foo (x)
(macrolet ((bar () x))
(bar)))

needs to be compiled into

(setf (symbol-function 'foo)
(lambda (x) <expansion-of-bar>))

but since x's value isn't known yet when that expansion is done, it's
like using the macro in the same file if it's other than at toplevel...
except there is no eval-when to rescue it in this case. So the situation
of seeing the variables is prohibited.

[...]
No, I don't care about the functionality one way or another. What I
care about (at least for the moment) is understanding why things are the
way they are in light of what the spec actually says. I don't care
*that* it doesn't work, I want to understand *why* it doesn't work. It
still seems to me that, according to the spec, it should.

Does this kind of analysis make any more sense? I THINK this is
closer to right. Though it's late and I might still have made some
mistake somewhere. At least this time I looked at the spec first,
though. Handy thing that. People always expect me to still remember
it, but the truth is that the day I finished writing it, I felt this
gigantic sigh of relief as I realized I could sometimes just look
things up like anyone else and not have to hold all that stuff in my
brain any more...

.

Relevant Pages

  • Re: Whats up with Scheme macros?
    ... which is NOT part of the spec for defmacro: ... might be effectively a bug in the spec since it really ... But the value of x is needed at compile time, ... expand foo to compile it. ...
    (comp.lang.lisp)
  • Re: Is DEFCONSTANT broken?
    ... (defconstant foo ...) ... If you compile a file containing defconstant which is used in that file, ... defconstant overcomes the externalizeability problem by externalizing the ...
    (comp.lang.lisp)
  • Re: HashMap get/put
    ... Class A defines method foo() which is final. ... Compile both classes into files A.class and B.class. ... public void write ... explicitly tagged with the 'virtual' keyword, and the override uses the ...
    (comp.lang.java.programmer)
  • Re: Is Double Dispatch really object-oriented?
    ... Compile error, in a language that supports MD properly => ... No dispatch ever fails ... Do you seriously claim that you can test Foo having just one B? ...
    (comp.object)
  • Re: Whats up with Scheme macros?
    ... Because the binding has to happen at a known time so the init form ... (defmacro foo () ... At compile time? ... it were to follow what DEFUN does. ...
    (comp.lang.lisp)