OpenToken: Handling the empty word token

Hello all.

Very often grammars have so called epsilon-productions where one
alternative for a non-terminal symbol points to the empty word
(epsilon).

For example: Optional -> Something | epsilon

In OpenToken I modeled the epsilon token as an
OpenToken.Recognizer.Nothing.Instance and defined the production like
this:

Optional <= Something and
Optional <= epsilon

Now I realized that the lexer would actually never emit the epsilon
token, because of it's pure formal meaning, and thus the second
production would never be detected.

Is there a way to instrument the parser to silently accept the epsilon
token whenever it expects it without consuming a token from the lexer,
or is it a common convention to translate each grammar into a epsilon-
free representation?

Thanks in advance.

-- Martin
.

Relevant Pages

  • Re: OpenToken: Handling the empty word token
    ... but I haven't seen many grammars. ... token, while there is the epsilon as one of the expected tokens, then ... Another thing to be aware of; the lexer in OpenToken is horribly ...
    (comp.lang.ada)
  • Re: Purple Dragon Book: Newbie questions on Chapter 4 text.
    ... I will answer partially to the first question with a trivial grammar. ... At most one of alpha and beta can derive the empty string. ... If beta *=> epsilon, then alpha does not derive any string ... Now for Production rule 1 all tree condition are met. ...
    (comp.compilers)
  • Re: CFG for L(G) and L(G)
    ... to change a grammar G where epsilon is not part of Land you can get ... If a production has more than two symbols on the RHS, ... Now remove all empty productions. ...
    (comp.theory)