Re: A Proper Lexer

akopa wrote:

Does any one have any advice on doing fast-ish character I/O for a Lex
like scanner. I've read http://www.emmett.ca/~sabetts/slurp.html,
but I want to present single chars to the scanner.

....

I was a little surprised to find no equivalent of Lex available for
Common Lisp. There is http://www.geocities.com/mparker762/clawk but
it is string oriented, not stream oriented.


There is little incentive to use Lex in lisp, because the lisp reader *is* a
lexer (tokenizer), hence, lisp is the language least likely to have a
Lex-like library. I've built many small languages and just used what lisp
gives me, e.g. if you can design it so that the little language has a space
between each token, you don't have to write a lexer. I suppose you could
get fancier by employing read macros, et al.

When I write languages that cannot follow the above simplification, I use my
home-grown CL versions of extended S/SL (syntax/semantic language) and my
experimental packrat parser (which describes the construction of tokens and
parses - google for "Bryan Ford Packrat Parser"). I can share these with
you, but they are not Lex-like (nor are they well-documented, but there is
always email :-).

By definition, lexing something a character at a time is, by definition, a
state machine. If you design it with a state machine macro - code executed
on entry, transition and exit - you will not tie yourself up in knots. I
have a state machine macro which I could share.

Lexing is often a simple enough job that you can just roll it by hand using
a few simple primitives (I think I can list them, if you're interested).
Again, the state machine paradigm is the way to go.

As for speed, what are you doing? I've stopped worrying about speed in
lexing and parsing - desktop machines are so fast these days that I seem to
get away with fully-backtracking parsers and a packrat parser with no
packratting and don't even notice.

And if you REALLY want Lex, then I know from direct experience that you can
code one up directly from the algorithms in the Dragon book (I did it about
30 years ago, but my version is lost, or on an 8.5" floppy :-).

It ain't exactly what you asked for, but contact me if any of the above is
of interest.

pt



.

Relevant Pages

  • Re: A Proper Lexer
    ... Common Lisp. ... e.g. if you can design it so that the little language has a space ... experimental packrat parser (which describes the construction of tokens and ... If you design it with a state machine macro - code executed ...
    (comp.lang.lisp)
  • Re: memory usage in cmucl
    ... transform that language into Lisp code to implement it. ... in the state machine to be executed. ...
    (comp.lang.lisp)
  • Re: ILC2005: McCarthy denounces Common Lisp, "Lisp", XML, and Rahul
    ... >> the language should be available to users. ... In the design of Common Lisp, I asked Dave Moon (one of the architects ... Now, there are good kinds of low-level, like the way that floats are ...
    (comp.lang.lisp)
  • Re: CollabRx seeks brilliant engineers for an excellent e-science adventure
    ... belief that lisp programmers are smarter/better. ... Java or PHP programmers. ... a type of language that attracts a personality that meets my perceptions ...
    (comp.lang.lisp)
  • Re: Whats the best language to learn...
    ... on processors designed to run Lisp and Lisp OSes. ... byte-addressed memory, has native support for variable-sized value types, ... popular OO language, rather a language about like that of Delphi would have ... lisp, java, ruby, etc. ...
    (comp.programming)