SQLite

Check-in [84d54eb3]
Login

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

Overview
Comment:Lemon updates: (1) include the #defines for all tokens in the generated C file, so that the C-file can be stand-alone. (2) If the grammar begins with a %include {...} directive on line one, make that directive the header for the generated C file. (3) Enhance the lemon.html documentation.
Downloads: Tarball | ZIP archive | SQL archive
Timelines: family | ancestors | descendants | both | trunk
Files: files | file ages | folders
SHA3-256: 84d54eb35716174195ee7e5ac846f47308e5dbb0056e8ff568daa133860bab74
User & Date: drh 2020-09-01 11:20:03
Context
2020-09-01
12:26
In the Lemon output, add a prefix comment that explains that the output file is automatically generated and shows the name of the source file. (check-in: d34caf3b user: drh tags: trunk)
11:20
Lemon updates: (1) include the #defines for all tokens in the generated C file, so that the C-file can be stand-alone. (2) If the grammar begins with a %include {...} directive on line one, make that directive the header for the generated C file. (3) Enhance the lemon.html documentation. (check-in: 84d54eb3 user: drh tags: trunk)
01:52
Improvements to the IN-early-out optimization so that it works more efficiently when there are two or more indexed IN clauses on a single table. (check-in: 35505c68 user: drh tags: trunk)
Changes
Unified Diff Ignore Whitespace Patch
Changes to doc/lemon.html. 
1
2
3
4
5

6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25






























26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45

46
47
48
49
50
51
52
53
54
55
56
57

58
59
60
61
62
63
64
65

66
67
68
69
70
71
72

<html>
<head>
<title>The Lemon Parser Generator</title>
</head>
<body bgcolor='white'>

<h1 align='center'>The Lemon Parser Generator</h1>

<p>Lemon is an LALR(1) parser generator for C.
It does the same job as "bison" and "yacc".
But Lemon is not a bison or yacc clone.  Lemon
uses a different grammar syntax which is designed to
reduce the number of coding errors.  Lemon also uses a
parsing engine that is faster than yacc and
bison and which is both reentrant and threadsafe.
(Update: Since the previous sentence was written, bison
has also been updated so that it too can generate a
reentrant and threadsafe parser.)
Lemon also implements features that can be used
to eliminate resource leaks, making it suitable for use
in long-running programs such as graphical user interfaces
or embedded controllers.</p>

<p>This document is an introduction to the Lemon
parser generator.</p>































<h2>Security Note</h2>

<p>The language parser code created by Lemon is very robust and
is well-suited for use in internet-facing applications that need to
safely process maliciously crafted inputs.</p>

<p>The "lemon.exe" command-line tool itself works great when given a valid
input grammar file and almost always gives helpful
error messages for malformed inputs.  However,  it is possible for
a malicious user to craft a grammar file that will cause 
lemon.exe to crash.
We do not see this as a problem, as lemon.exe is not intended to be used
with hostile inputs.
To summarize:</p>

<ul>
<li>Parser code generated by lemon &rarr; Robust and secure
<li>The "lemon.exe" command line tool itself &rarr; Not so much
</ul>


<h2>Theory of Operation</h2>

<p>The main goal of Lemon is to translate a context free grammar (CFG)
for a particular language into C code that implements a parser for
that language.
The program has two inputs:</p>
<ul>
<li>The grammar specification.
<li>A parser template file.
</ul>
<p>Typically, only the grammar specification is supplied by the programmer.
Lemon comes with a default parser template which works fine for most

applications.  But the user is free to substitute a different parser
template if desired.</p>

<p>Depending on command-line options, Lemon will generate up to
three output files.</p>
<ul>
<li>C code to implement the parser.
<li>A header file defining an integer ID for each terminal symbol.

<li>An information file that describes the states of the generated parser
    automaton.
</ul>
<p>By default, all three of these output files are generated.
The header file is suppressed if the "-m" command-line option is
used and the report file is omitted when "-q" is selected.</p>






|
>




















>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
|



















>
|

|


|





|
>
|
|




|
|
>








1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106

<html>
<head>
<title>The Lemon Parser Generator</title>
</head>
<body>
<a id="main"></a>
<h1 align='center'>The Lemon Parser Generator</h1>

<p>Lemon is an LALR(1) parser generator for C.
It does the same job as "bison" and "yacc".
But Lemon is not a bison or yacc clone.  Lemon
uses a different grammar syntax which is designed to
reduce the number of coding errors.  Lemon also uses a
parsing engine that is faster than yacc and
bison and which is both reentrant and threadsafe.
(Update: Since the previous sentence was written, bison
has also been updated so that it too can generate a
reentrant and threadsafe parser.)
Lemon also implements features that can be used
to eliminate resource leaks, making it suitable for use
in long-running programs such as graphical user interfaces
or embedded controllers.</p>

<p>This document is an introduction to the Lemon
parser generator.</p>

<a id="toc"></a>
<h2>1.0 Table of Contents</h2>
<ul>
<li><a href="#main">Introduction</a>
<li><a href="#toc">1.0 Table of Contents</a>
<li><a href="#secnot">2.0 Security Notes</a><br>
<li><a href="#optheory">3.0 Theory of Operation</a>
    <ul>
    <li><a href="#options">3.1 Command Line Options</a>
    <li><a href="#interface">3.2 The Parser Interface</a>
        <ul>
        <li><a href="#onstack">3.2.1 Allocating The Parse Object On Stack</a>
        <li><a href="#ifsum">3.2.2 Interface Summary</a>
        </ul>
    <li><a href="#yaccdiff">3.3 Differences With YACC and BISON</a>
    <li><a href="#build">3.4 Building The "lemon" Or "lemon.exe" Executable</a>
    </ul>
<li><a href="#syntax">4.0 Input File Syntax</a>
    <ul>
    <li><a href="#tnt">4.1 Terminals and Nonterminals</a>
    <li><a href="#rules">4.2 Grammar Rules</a>
    <li><a href="#precrules">4.3 Precedence Rules</a>
    <li><a href="#special">4.4 Special Directives</a>
    </ul>
<li><a href="#errors">5.0 Error Processing</a>
<li><a href="#history">6.0 History of Lemon</a>
<li><a href="#copyright">7.0 Copyright</a>
</ul>

<a id="secnot"></a>
<h2>2.0 Security Note</h2>

<p>The language parser code created by Lemon is very robust and
is well-suited for use in internet-facing applications that need to
safely process maliciously crafted inputs.</p>

<p>The "lemon.exe" command-line tool itself works great when given a valid
input grammar file and almost always gives helpful
error messages for malformed inputs.  However,  it is possible for
a malicious user to craft a grammar file that will cause 
lemon.exe to crash.
We do not see this as a problem, as lemon.exe is not intended to be used
with hostile inputs.
To summarize:</p>

<ul>
<li>Parser code generated by lemon &rarr; Robust and secure
<li>The "lemon.exe" command line tool itself &rarr; Not so much
</ul>

<a id="optheory"></a>
<h2>3.0 Theory of Operation</h2>

<p>Lemon is computer program that translates a context free grammar (CFG)
for a particular language into C code that implements a parser for
that language.
The Lemon program has two inputs:</p>
<ul>
<li>The grammar specification.
<li>A parser template file.
</ul>
<p>Typically, only the grammar specification is supplied by the programmer.
Lemon comes with a default parser template
("<a href="https://sqlite.org/src/file/tool/lempar.c">lempar.c</a>")
that works fine for most applications.  But the user is free to substitute
a different parser template if desired.</p>

<p>Depending on command-line options, Lemon will generate up to
three output files.</p>
<ul>
<li>C code to implement a parser for the input grammar.
<li>A header file defining an integer ID for each terminal symbol
    (or "token").
<li>An information file that describes the states of the generated parser
    automaton.
</ul>
<p>By default, all three of these output files are generated.
The header file is suppressed if the "-m" command-line option is
used and the report file is omitted when "-q" is selected.</p>
︙︙
80
81
82
83
84
85
86

87
88
89
90
91
92
93
94

<p>This command will generate three output files named "gram.c",
"gram.h" and "gram.out".
The first is C code to implement the parser.  The second
is the header file that defines numerical values for all
terminal symbols, and the last is the report that explains
the states used by the parser automaton.</p>


<h3>Command Line Options</h3>

<p>The behavior of Lemon can be modified using command-line options.
You can obtain a list of the available command-line options together
with a brief explanation of what each does by typing</p>
<pre>
   lemon "-?"
</pre>








>
|








114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129

<p>This command will generate three output files named "gram.c",
"gram.h" and "gram.out".
The first is C code to implement the parser.  The second
is the header file that defines numerical values for all
terminal symbols, and the last is the report that explains
the states used by the parser automaton.</p>

<a id="options"></a>
<h3>3.1 Command Line Options</h3>

<p>The behavior of Lemon can be modified using command-line options.
You can obtain a list of the available command-line options together
with a brief explanation of what each does by typing</p>
<pre>
   lemon "-?"
</pre>
︙︙
130
131
132
133
134
135
136

137
138
139
140
141
142
143
144

Show parser statistics before exiting.
<li><b>-T<i>file</i></b>
Use <i>file</i> as the template for the generated C-code parser implementation.
<li><b>-x</b>
Print the Lemon version number.
</ul>


<h3>The Parser Interface</h3>

<p>Lemon doesn't generate a complete, working program.  It only generates
a few subroutines that implement a parser.  This section describes
the interface to those subroutines.  It is up to the programmer to
call these subroutines in an appropriate way in order to produce a
complete system.</p>









>
|








165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180

Show parser statistics before exiting.
<li><b>-T<i>file</i></b>
Use <i>file</i> as the template for the generated C-code parser implementation.
<li><b>-x</b>
Print the Lemon version number.
</ul>

<a id="interface"></a>
<h3>3.2 The Parser Interface</h3>

<p>Lemon doesn't generate a complete, working program.  It only generates
a few subroutines that implement a parser.  This section describes
the interface to those subroutines.  It is up to the programmer to
call these subroutines in an appropriate way in order to produce a
complete system.</p>
︙︙
271
272
273
274
275
276
277






















































278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301





























302
303
304
305
306
307
308
309
310
311
312
313
314
315

316
317
318
319
320
321
322
323

</pre>
<p>After this routine is called, a short (one-line) message is written
to the designated output stream every time the parser changes states
or calls an action routine.  Each such message is prefaced using
the text given by zPrefix.  This debugging output can be turned off
by calling ParseTrace() again with a first argument of NULL (0).</p>























































<h3>Differences With YACC and BISON</h3>

<p>Programmers who have previously used the yacc or bison parser
generator will notice several important differences between yacc and/or
bison and Lemon.</p>
<ul>
<li>In yacc and bison, the parser calls the tokenizer.  In Lemon,
    the tokenizer calls the parser.
<li>Lemon uses no global variables.  Yacc and bison use global variables
    to pass information between the tokenizer and parser.
<li>Lemon allows multiple parsers to be running simultaneously.  Yacc
    and bison do not.
</ul>
<p>These differences may cause some initial confusion for programmers
with prior yacc and bison experience.
But after years of experience using Lemon, I firmly
believe that the Lemon way of doing things is better.</p>

<p><i>Updated as of 2016-02-16:</i>
The text above was written in the 1990s.
We are told that Bison has lately been enhanced to support the
tokenizer-calls-parser paradigm used by Lemon, and to obviate the
need for global variables.</p>






























<h2>Input File Syntax</h2>

<p>The main purpose of the grammar specification file for Lemon is
to define the grammar for the parser.  But the input file also
specifies additional information Lemon requires to do its job.
Most of the work in using Lemon is in writing an appropriate
grammar file.</p>

<p>The grammar file for Lemon is, for the most part, a free format.
It does not have sections or divisions like yacc or bison.  Any
declaration can occur at any point in the file.  Lemon ignores
whitespace (except where it is needed to separate tokens), and it
honors the same commenting conventions as C and C++.</p>


<h3>Terminals and Nonterminals</h3>

<p>A terminal symbol (token) is any string of alphanumeric
and/or underscore characters
that begins with an uppercase letter.
A terminal can contain lowercase letters after the first character,
but the usual convention is to make terminals all uppercase.
A nonterminal, on the other hand, is any string of alphanumeric








>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
|




















|


>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
|













>
|








307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443

</pre>
<p>After this routine is called, a short (one-line) message is written
to the designated output stream every time the parser changes states
or calls an action routine.  Each such message is prefaced using
the text given by zPrefix.  This debugging output can be turned off
by calling ParseTrace() again with a first argument of NULL (0).</p>

<a id="onstack"></a>
<h4>3.2.1 Allocating The Parse Object On Stack</h4>

<p>If all calls to the Parse() interface are made from within
<a href="#pcode"><tt>%code</tt> directives</a>, then the parse
object can be allocated from the stack rather than from the heap.
These are the steps:

<ul>
<li> Declare a local variable of type "yyParser"
<li> Initialize the variable using ParseInit()
<li> Pass a pointer to the variable in calls ot Parse()
<li> Deallocate substructure in the parse variable using ParseFinalize().
</ul>

<p>The following code illustrates how this is done:

<pre>
   ParseFile(){
      yyParser x;
      ParseInit( &x );
      while( GetNextToken(pTokenizer,&amp;hTokenId, &amp;sToken) ){
         Parse(&x, hTokenId, sToken);
      }
      Parse(&x, 0, sToken);
      ParseFinalize( &x );
   }
</pre>

<a id="ifsum"></a>
<h4>3.2.2 Interface Summary</h4>

<p>Here is a quick overview of the C-language interface to a
Lemon-generated parser:</p>

<blockquote><pre>
void *ParseAlloc( (void*(*malloc)(size_t) );
void ParseFree(void *pParser, (void(*free)(void*) );
void Parse(void *pParser, int tokenCode, ParseTOKENTYPE token, ...);
void ParseTrace(FILE *stream, char *zPrefix);
</pre></blockquote>

<p>Notes:</p>
<ul>
<li> Use the <a href="#pname"><tt>%name</tt> directive</a> to change
the "Parse" prefix names of the procedures in the interface.
<li> Use the <a href="#token_type"><tt>%token_type</tt> directive</a>
to define the "ParseTOKENTYPE" type.
<li> Use the <a href="#extraarg"><tt>%extra_argument</tt> directive</a>
to specify the type and name of the 4th parameter to the
Parse() function.
</ul>

<a id="yaccdiff"></a>
<h3>3.3 Differences With YACC and BISON</h3>

<p>Programmers who have previously used the yacc or bison parser
generator will notice several important differences between yacc and/or
bison and Lemon.</p>
<ul>
<li>In yacc and bison, the parser calls the tokenizer.  In Lemon,
    the tokenizer calls the parser.
<li>Lemon uses no global variables.  Yacc and bison use global variables
    to pass information between the tokenizer and parser.
<li>Lemon allows multiple parsers to be running simultaneously.  Yacc
    and bison do not.
</ul>
<p>These differences may cause some initial confusion for programmers
with prior yacc and bison experience.
But after years of experience using Lemon, I firmly
believe that the Lemon way of doing things is better.</p>

<p><i>Updated as of 2016-02-16:</i>
The text above was written in the 1990s.
We are told that Bison has lately been enhanced to support the
tokenizer-calls-parser paradigm used by Lemon, eliminating the
need for global variables.</p>

<a id="build"><a>
<h3>3.4 Building The "lemon" or "lemon.exe" Executable</h3>

<p>The "lemon" or "lemon.exe" program is built from a single file
of C-code named 
"<a href="https://sqlite.org/src/tool/lemon.c">lemon.c</a>". 
The Lemon source code is generic C89 code that uses
no unusual or non-standard libraries.  Any
reasonable C compiler should suffice to compile the lemon program.  
A command-line like the following will usually work:</p>

<blockquote><pre>
cc -o lemon lemon.c
</pre></blockquote

<p>On Windows machines with Visual C++ installed, bring up a
"VS20<i>NN</i> x64 Native Tools Command Prompt" window and enter:

<blockquote><pre>
cl lemon.c
</pre></blockquote>

<p>Compiling Lemon really is that simple. 
Additional compiler options such as
"-O2" or "-g" or "-Wall" can be added if desired, but they are not
necessary.</p>


<a id="syntax"></a>
<h2>4.0 Input File Syntax</h2>

<p>The main purpose of the grammar specification file for Lemon is
to define the grammar for the parser.  But the input file also
specifies additional information Lemon requires to do its job.
Most of the work in using Lemon is in writing an appropriate
grammar file.</p>

<p>The grammar file for Lemon is, for the most part, a free format.
It does not have sections or divisions like yacc or bison.  Any
declaration can occur at any point in the file.  Lemon ignores
whitespace (except where it is needed to separate tokens), and it
honors the same commenting conventions as C and C++.</p>

<a id="tnt"></a>
<h3>4.1 Terminals and Nonterminals</h3>

<p>A terminal symbol (token) is any string of alphanumeric
and/or underscore characters
that begins with an uppercase letter.
A terminal can contain lowercase letters after the first character,
but the usual convention is to make terminals all uppercase.
A nonterminal, on the other hand, is any string of alphanumeric
︙︙
334
335
336
337
338
339
340

341
342
343
344
345
346
347
348


<p>Yacc and bison allow terminal symbols to have either alphanumeric
names or to be individual characters included in single quotes, like
this: ')' or '$'.  Lemon does not allow this alternative form for
terminal symbols.  With Lemon, all symbols, terminals and nonterminals,
must have alphanumeric names.</p>


<h3>Grammar Rules</h3>

<p>The main component of a Lemon grammar file is a sequence of grammar
rules.
Each grammar rule consists of a nonterminal symbol followed by
the special symbol "::=" and then a list of terminals and/or nonterminals.
The rule is terminated by a period.
The list of terminals and nonterminals on the right-hand side of the








>
|








454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469


<p>Yacc and bison allow terminal symbols to have either alphanumeric
names or to be individual characters included in single quotes, like
this: ')' or '$'.  Lemon does not allow this alternative form for
terminal symbols.  With Lemon, all symbols, terminals and nonterminals,
must have alphanumeric names.</p>

<a id="rules"></a>
<h3>4.2 Grammar Rules</h3>

<p>The main component of a Lemon grammar file is a sequence of grammar
rules.
Each grammar rule consists of a nonterminal symbol followed by
the special symbol "::=" and then a list of terminals and/or nonterminals.
The rule is terminated by a period.
The list of terminals and nonterminals on the right-hand side of the
︙︙
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433


<p>The Lemon notation for linking grammar rules to reduce actions
also facilitates the use of destructors for reclaiming memory
allocated by the values of terminals and nonterminals on the
right-hand side of a rule.</p>

<a id='precrules'></a>
<h3>Precedence Rules</h3>

<p>Lemon resolves parsing ambiguities in exactly the same way as
yacc and bison.  A shift-reduce conflict is resolved in favor
of the shift, and a reduce-reduce conflict is resolved by reducing
whichever rule comes first in the grammar file.</p>

<p>Just like in








|








540
541
542
543
544
545
546
547
548
549
550
551
552
553
554


<p>The Lemon notation for linking grammar rules to reduce actions
also facilitates the use of destructors for reclaiming memory
allocated by the values of terminals and nonterminals on the
right-hand side of a rule.</p>

<a id='precrules'></a>
<h3>4.3 Precedence Rules</h3>

<p>Lemon resolves parsing ambiguities in exactly the same way as
yacc and bison.  A shift-reduce conflict is resolved in favor
of the shift, and a reduce-reduce conflict is resolved by reducing
whichever rule comes first in the grammar file.</p>

<p>Just like in
︙︙
535
536
537
538
539
540
541

542
543
544
545
546
547
548
549

<li> If both rules have precedence and the precedence is different,
     then resolve the dispute in favor of the rule with the highest
     precedence, and do not report a conflict.
<li> Otherwise, resolve the conflict by reducing by the rule that
     appears first in the grammar, and report a parsing conflict.
</ul>


<h3>Special Directives</h3>

<p>The input grammar to Lemon consists of grammar rules and special
directives.  We've described all the grammar rules, so now we'll
talk about the special directives.</p>

<p>Directives in Lemon can occur in any order.  You can put them before
the grammar rules, or after the grammar rules, or in the midst of the








>
|








656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671

<li> If both rules have precedence and the precedence is different,
     then resolve the dispute in favor of the rule with the highest
     precedence, and do not report a conflict.
<li> Otherwise, resolve the conflict by reducing by the rule that
     appears first in the grammar, and report a parsing conflict.
</ul>

<a id="special"></a>
<h3>4.4 Special Directives</h3>

<p>The input grammar to Lemon consists of grammar rules and special
directives.  We've described all the grammar rules, so now we'll
talk about the special directives.</p>

<p>Directives in Lemon can occur in any order.  You can put them before
the grammar rules, or after the grammar rules, or in the midst of the
︙︙
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599



600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629

<li><tt><a href='#ptype'>%type</a></tt>
<li><tt><a href='#pwildcard'>%wildcard</a></tt>
</ul>
<p>Each of these directives will be described separately in the
following sections:</p>

<a id='pcode'></a>
<h4>The <tt>%code</tt> directive</h4>

<p>The <tt>%code</tt> directive is used to specify additional C code that
is added to the end of the main output file.  This is similar to
the <tt><a href='#pinclude'>%include</a></tt> directive except that
<tt>%include</tt> is inserted at the beginning of the main output file.</p>

<p><tt>%code</tt> is typically used to include some action routines or perhaps
a tokenizer or even the "main()" function
as part of the output file.</p>




<a id='default_destructor'></a>
<h4>The <tt>%default_destructor</tt> directive</h4>

<p>The <tt>%default_destructor</tt> directive specifies a destructor to
use for non-terminals that do not have their own destructor
specified by a separate <tt>%destructor</tt> directive.  See the documentation
on the <tt><a href='#destructor'>%destructor</a></tt> directive below for
additional information.</p>

<p>In some grammars, many different non-terminal symbols have the
same data type and hence the same destructor.  This directive is
a convenient way to specify the same destructor for all those
non-terminals using a single statement.</p>

<a id='default_type'></a>
<h4>The <tt>%default_type</tt> directive</h4>

<p>The <tt>%default_type</tt> directive specifies the data type of non-terminal
symbols that do not have their own data type defined using a separate
<tt><a href='#ptype'>%type</a></tt> directive.</p>

<a id='destructor'></a>
<h4>The <tt>%destructor</tt> directive</h4>

<p>The <tt>%destructor</tt> directive is used to specify a destructor for
a non-terminal symbol.
(See also the <tt><a href='#token_destructor'>%token_destructor</a></tt>
directive which is used to specify a destructor for terminal symbols.)</p>

<p>A non-terminal's destructor is called to dispose of the








|










>
>
>

|













|






|








704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754

<li><tt><a href='#ptype'>%type</a></tt>
<li><tt><a href='#pwildcard'>%wildcard</a></tt>
</ul>
<p>Each of these directives will be described separately in the
following sections:</p>

<a id='pcode'></a>
<h4>4.4.1 The <tt>%code</tt> directive</h4>

<p>The <tt>%code</tt> directive is used to specify additional C code that
is added to the end of the main output file.  This is similar to
the <tt><a href='#pinclude'>%include</a></tt> directive except that
<tt>%include</tt> is inserted at the beginning of the main output file.</p>

<p><tt>%code</tt> is typically used to include some action routines or perhaps
a tokenizer or even the "main()" function
as part of the output file.</p>

<p>There can be multiple <tt>%code</tt> directives.  The arguments of
all <tt>%code</tt> directives are concatenated.</p>

<a id='default_destructor'></a>
<h4>4.4.2 The <tt>%default_destructor</tt> directive</h4>

<p>The <tt>%default_destructor</tt> directive specifies a destructor to
use for non-terminals that do not have their own destructor
specified by a separate <tt>%destructor</tt> directive.  See the documentation
on the <tt><a href='#destructor'>%destructor</a></tt> directive below for
additional information.</p>

<p>In some grammars, many different non-terminal symbols have the
same data type and hence the same destructor.  This directive is
a convenient way to specify the same destructor for all those
non-terminals using a single statement.</p>

<a id='default_type'></a>
<h4>4.4.3 The <tt>%default_type</tt> directive</h4>

<p>The <tt>%default_type</tt> directive specifies the data type of non-terminal
symbols that do not have their own data type defined using a separate
<tt><a href='#ptype'>%type</a></tt> directive.</p>

<a id='destructor'></a>
<h4>4.4.4 The <tt>%destructor</tt> directive</h4>

<p>The <tt>%destructor</tt> directive is used to specify a destructor for
a non-terminal symbol.
(See also the <tt><a href='#token_destructor'>%token_destructor</a></tt>
directive which is used to specify a destructor for terminal symbols.)</p>

<p>A non-terminal's destructor is called to dispose of the
︙︙
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679

the destructor is not called in this circumstance.</p>

<p>Destructors help avoid memory leaks by automatically freeing
allocated objects when they go out of scope.
To do the same using yacc or bison is much more difficult.</p>

<a id='extraarg'></a>
<h4>The <tt>%extra_argument</tt> directive</h4>

<p>The <tt>%extra_argument</tt> directive instructs Lemon to add a 4th parameter
to the parameter list of the Parse() function it generates.  Lemon
doesn't do anything itself with this extra argument, but it does
make the argument available to C-code action routines, destructors,
and so forth.  For example, if the grammar file contains:</p>









|








790
791
792
793
794
795
796
797
798
799
800
801
802
803
804

the destructor is not called in this circumstance.</p>

<p>Destructors help avoid memory leaks by automatically freeing
allocated objects when they go out of scope.
To do the same using yacc or bison is much more difficult.</p>

<a id='extraarg'></a>
<h4>4.4.5 The <tt>%extra_argument</tt> directive</h4>

<p>The <tt>%extra_argument</tt> directive instructs Lemon to add a 4th parameter
to the parameter list of the Parse() function it generates.  Lemon
doesn't do anything itself with this extra argument, but it does
make the argument available to C-code action routines, destructors,
and so forth.  For example, if the grammar file contains:</p>
︙︙
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721

in the most recent call to Parse().</p>

<p>The <tt>%extra_context</tt> directive works the same except that it
is passed in on the ParseAlloc() or ParseInit() routines instead of
on Parse().</p>

<a id='extractx'></a>
<h4>The <tt>%extra_context</tt> directive</h4>

<p>The <tt>%extra_context</tt> directive instructs Lemon to add a 2nd parameter
to the parameter list of the ParseAlloc() and ParseInit() functions.  Lemon
doesn't do anything itself with these extra argument, but it does
store the value make it available to C-code action routines, destructors,
and so forth.  For example, if the grammar file contains:</p>

<pre>
    %extra_context { MyStruct *pAbc }
</pre>

<p>Then the ParseAlloc() and ParseInit() functions will have an 2nd parameter
of type "MyStruct*" and all action routines will have access to
a variable named "pAbc" that is the value of that 2nd parameter.</p>

<p>The <tt>%extra_argument</tt> directive works the same except that it
is passed in on the Parse() routine instead of on ParseAlloc()/ParseInit().</p>

<a id='pfallback'></a>
<h4>The <tt>%fallback</tt> directive</h4>

<p>The <tt>%fallback</tt> directive specifies an alternative meaning for one
or more tokens.  The alternative meaning is tried if the original token
would have generated a syntax error.</p>

<p>The <tt>%fallback</tt> directive was added to support robust parsing of SQL
syntax in <a href='https://www.sqlite.org/'>SQLite</a>.








|



















|








812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846

in the most recent call to Parse().</p>

<p>The <tt>%extra_context</tt> directive works the same except that it
is passed in on the ParseAlloc() or ParseInit() routines instead of
on Parse().</p>

<a id='extractx'></a>
<h4>4.4.6 The <tt>%extra_context</tt> directive</h4>

<p>The <tt>%extra_context</tt> directive instructs Lemon to add a 2nd parameter
to the parameter list of the ParseAlloc() and ParseInit() functions.  Lemon
doesn't do anything itself with these extra argument, but it does
store the value make it available to C-code action routines, destructors,
and so forth.  For example, if the grammar file contains:</p>

<pre>
    %extra_context { MyStruct *pAbc }
</pre>

<p>Then the ParseAlloc() and ParseInit() functions will have an 2nd parameter
of type "MyStruct*" and all action routines will have access to
a variable named "pAbc" that is the value of that 2nd parameter.</p>

<p>The <tt>%extra_argument</tt> directive works the same except that it
is passed in on the Parse() routine instead of on ParseAlloc()/ParseInit().</p>

<a id='pfallback'></a>
<h4>4.4.7 The <tt>%fallback</tt> directive</h4>

<p>The <tt>%fallback</tt> directive specifies an alternative meaning for one
or more tokens.  The alternative meaning is tried if the original token
would have generated a syntax error.</p>

<p>The <tt>%fallback</tt> directive was added to support robust parsing of SQL
syntax in <a href='https://www.sqlite.org/'>SQLite</a>.
︙︙
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751

names terminated by a period.
The first token name is the fallback token &mdash; the
token to which all the other tokens fall back to.  The second and subsequent
arguments are tokens which fall back to the token identified by the first
argument.</p>

<a id='pifdef'></a>
<h4>The <tt>%if</tt> directive and its friends</h4>

<p>The <tt>%if</tt>, <tt>%ifdef</tt>, <tt>%ifndef</tt>, <tt>%else</tt>,
and <tt>%endif</tt> directives
are similar to #if, #ifdef, #ifndef, #else, and #endif in the C-preprocessor,
just not as general.
Each of these directives must begin at the left margin.  No whitespace
is allowed between the "%" and the directive name.</p>








|








862
863
864
865
866
867
868
869
870
871
872
873
874
875
876

names terminated by a period.
The first token name is the fallback token &mdash; the
token to which all the other tokens fall back to.  The second and subsequent
arguments are tokens which fall back to the token identified by the first
argument.</p>

<a id='pifdef'></a>
<h4>4.4.8 The <tt>%if</tt> directive and its friends</h4>

<p>The <tt>%if</tt>, <tt>%ifdef</tt>, <tt>%ifndef</tt>, <tt>%else</tt>,
and <tt>%endif</tt> directives
are similar to #if, #ifdef, #ifndef, #else, and #endif in the C-preprocessor,
just not as general.
Each of these directives must begin at the left margin.  No whitespace
is allowed between the "%" and the directive name.</p>
︙︙
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782

its corresponding <tt>%endif</tt>.</p>

<p>Note that the argument to <tt>%ifdef</tt> and <tt>%ifndef</tt> is
intended to be a single preprocessor symbol name, not a general expression.
Use the "<tt>%if</tt>" directive for general expressions.</p>

<a id='pinclude'></a>
<h4>The <tt>%include</tt> directive</h4>

<p>The <tt>%include</tt> directive specifies C code that is included at the
top of the generated parser.  You can include any text you want &mdash;
the Lemon parser generator copies it blindly.  If you have multiple
<tt>%include</tt> directives in your grammar file, their values are concatenated
so that all <tt>%include</tt> code ultimately appears near the top of the
generated parser, in the same order as it appeared in the grammar.</p>








|








893
894
895
896
897
898
899
900
901
902
903
904
905
906
907

its corresponding <tt>%endif</tt>.</p>

<p>Note that the argument to <tt>%ifdef</tt> and <tt>%ifndef</tt> is
intended to be a single preprocessor symbol name, not a general expression.
Use the "<tt>%if</tt>" directive for general expressions.</p>

<a id='pinclude'></a>
<h4>4.4.9 The <tt>%include</tt> directive</h4>

<p>The <tt>%include</tt> directive specifies C code that is included at the
top of the generated parser.  You can include any text you want &mdash;
the Lemon parser generator copies it blindly.  If you have multiple
<tt>%include</tt> directives in your grammar file, their values are concatenated
so that all <tt>%include</tt> code ultimately appears near the top of the
generated parser, in the same order as it appeared in the grammar.</p>
︙︙
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806

<p>This might be needed, for example, if some of the C actions in the
grammar call functions that are prototyped in unistd.h.</p>

<p>Use the <tt><a href="#pcode">%code</a></tt> directive to add code to
the end of the generated parser.</p>

<a id='pleft'></a>
<h4>The <tt>%left</tt> directive</h4>

The <tt>%left</tt> directive is used (along with the
<tt><a href='#pright'>%right</a></tt> and
<tt><a href='#pnonassoc'>%nonassoc</a></tt> directives) to declare
precedences of terminal symbols.
Every terminal symbol whose name appears after
a <tt>%left</tt> directive but before the next period (".") is








|








917
918
919
920
921
922
923
924
925
926
927
928
929
930
931

<p>This might be needed, for example, if some of the C actions in the
grammar call functions that are prototyped in unistd.h.</p>

<p>Use the <tt><a href="#pcode">%code</a></tt> directive to add code to
the end of the generated parser.</p>

<a id='pleft'></a>
<h4>4.4.10 The <tt>%left</tt> directive</h4>

The <tt>%left</tt> directive is used (along with the
<tt><a href='#pright'>%right</a></tt> and
<tt><a href='#pnonassoc'>%nonassoc</a></tt> directives) to declare
precedences of terminal symbols.
Every terminal symbol whose name appears after
a <tt>%left</tt> directive but before the next period (".") is
︙︙
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836


<p>LALR(1) grammars can get into a situation where they require
a large amount of stack space if you make heavy use or right-associative
operators.  For this reason, it is recommended that you use <tt>%left</tt>
rather than <tt>%right</tt> whenever possible.</p>

<a id='pname'></a>
<h4>The <tt>%name</tt> directive</h4>

<p>By default, the functions generated by Lemon all begin with the
five-character string "Parse".  You can change this string to something
different using the <tt>%name</tt> directive.  For instance:</p>

<pre>
   %name Abcde








|








947
948
949
950
951
952
953
954
955
956
957
958
959
960
961


<p>LALR(1) grammars can get into a situation where they require
a large amount of stack space if you make heavy use or right-associative
operators.  For this reason, it is recommended that you use <tt>%left</tt>
rather than <tt>%right</tt> whenever possible.</p>

<a id='pname'></a>
<h4>4.4.11 The <tt>%name</tt> directive</h4>

<p>By default, the functions generated by Lemon all begin with the
five-character string "Parse".  You can change this string to something
different using the <tt>%name</tt> directive.  For instance:</p>

<pre>
   %name Abcde
︙︙
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906

<li> AbcdeTrace(), and
<li> Abcde().
</ul>
</p>The <tt>%name</tt> directive allows you to generate two or more different
parsers and link them all into the same executable.</p>

<a id='pnonassoc'></a>
<h4>The <tt>%nonassoc</tt> directive</h4>

<p>This directive is used to assign non-associative precedence to
one or more terminal symbols.  See the section on
<a href='#precrules'>precedence rules</a>
or on the <tt><a href='#pleft'>%left</a></tt> directive
for additional information.</p>

<a id='parse_accept'></a>
<h4>The <tt>%parse_accept</tt> directive</h4>

<p>The <tt>%parse_accept</tt> directive specifies a block of C code that is
executed whenever the parser accepts its input string.  To "accept"
an input string means that the parser was able to process all tokens
without error.</p>

<p>For example:</p>

<pre>
   %parse_accept {
      printf("parsing complete!\n");
   }
</pre>

<a id='parse_failure'></a>
<h4>The <tt>%parse_failure</tt> directive</h4>

<p>The <tt>%parse_failure</tt> directive specifies a block of C code that
is executed whenever the parser fails complete.  This code is not
executed until the parser has tried and failed to resolve an input
error using is usual error recovery strategy.  The routine is
only invoked when parsing is unable to continue.</p>

<pre>
   %parse_failure {
     fprintf(stderr,"Giving up.  Parser is hopelessly lost...\n");
   }
</pre>

<a id='pright'></a>
<h4>The <tt>%right</tt> directive</h4>

<p>This directive is used to assign right-associative precedence to
one or more terminal symbols.  See the section on
<a href='#precrules'>precedence rules</a>
or on the <a href='#pleft'>%left</a> directive for additional information.</p>

<a id='stack_overflow'></a>
<h4>The <tt>%stack_overflow</tt> directive</h4>

<p>The <tt>%stack_overflow</tt> directive specifies a block of C code that
is executed if the parser's internal stack ever overflows.  Typically
this just prints an error message.  After a stack overflow, the parser
will be unable to continue and must be reset.</p>

<pre>








|








|















|














|







|








969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
1001
1002
1003
1004
1005
1006
1007
1008
1009
1010
1011
1012
1013
1014
1015
1016
1017
1018
1019
1020
1021
1022
1023
1024
1025
1026
1027
1028
1029
1030
1031

<li> AbcdeTrace(), and
<li> Abcde().
</ul>
</p>The <tt>%name</tt> directive allows you to generate two or more different
parsers and link them all into the same executable.</p>

<a id='pnonassoc'></a>
<h4>4.4.12 The <tt>%nonassoc</tt> directive</h4>

<p>This directive is used to assign non-associative precedence to
one or more terminal symbols.  See the section on
<a href='#precrules'>precedence rules</a>
or on the <tt><a href='#pleft'>%left</a></tt> directive
for additional information.</p>

<a id='parse_accept'></a>
<h4>4.4.13 The <tt>%parse_accept</tt> directive</h4>

<p>The <tt>%parse_accept</tt> directive specifies a block of C code that is
executed whenever the parser accepts its input string.  To "accept"
an input string means that the parser was able to process all tokens
without error.</p>

<p>For example:</p>

<pre>
   %parse_accept {
      printf("parsing complete!\n");
   }
</pre>

<a id='parse_failure'></a>
<h4>4.4.14 The <tt>%parse_failure</tt> directive</h4>

<p>The <tt>%parse_failure</tt> directive specifies a block of C code that
is executed whenever the parser fails complete.  This code is not
executed until the parser has tried and failed to resolve an input
error using is usual error recovery strategy.  The routine is
only invoked when parsing is unable to continue.</p>

<pre>
   %parse_failure {
     fprintf(stderr,"Giving up.  Parser is hopelessly lost...\n");
   }
</pre>

<a id='pright'></a>
<h4>4.4.15 The <tt>%right</tt> directive</h4>

<p>This directive is used to assign right-associative precedence to
one or more terminal symbols.  See the section on
<a href='#precrules'>precedence rules</a>
or on the <a href='#pleft'>%left</a> directive for additional information.</p>

<a id='stack_overflow'></a>
<h4>4.4.16 The <tt>%stack_overflow</tt> directive</h4>

<p>The <tt>%stack_overflow</tt> directive specifies a block of C code that
is executed if the parser's internal stack ever overflows.  Typically
this just prints an error message.  After a stack overflow, the parser
will be unable to continue and must be reset.</p>

<pre>
︙︙
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987

<p>Not like this:</p>
<pre>
   list ::= element list.      // right-recursion.  Bad!
   list ::= .
</pre>

<a id='stack_size'></a>
<h4>The <tt>%stack_size</tt> directive</h4>

<p>If stack overflow is a problem and you can't resolve the trouble
by using left-recursion, then you might want to increase the size
of the parser's stack using this directive.  Put an positive integer
after the <tt>%stack_size</tt> directive and Lemon will generate a parse
with a stack of the requested size.  The default value is 100.</p>

<pre>
   %stack_size 2000
</pre>

<a id='start_symbol'></a>
<h4>The <tt>%start_symbol</tt> directive</h4>

<p>By default, the start symbol for the grammar that Lemon generates
is the first non-terminal that appears in the grammar file.  But you
can choose a different start symbol using the
<tt>%start_symbol</tt> directive.</p>

<pre>
   %start_symbol  prog
</pre>

<a id='syntax_error'></a>
<h4>The <tt>%syntax_error</tt> directive</h4>

<p>See <a href='#error_processing'>Error Processing</a>.</p>

<a id='token_class'></a>
<h4>The <tt>%token_class</tt> directive</h4>

<p>Undocumented.  Appears to be related to the MULTITERMINAL concept.
<a href='http://sqlite.org/src/fdiff?v1=796930d5fc2036c7&v2=624b24c5dc048e09&sbs=0'>Implementation</a>.</p>

<a id='token_destructor'></a>
<h4>The <tt>%token_destructor</tt> directive</h4>

<p>The <tt>%destructor</tt> directive assigns a destructor to a non-terminal
symbol.  (See the description of the
<tt><a href='%destructor'>%destructor</a></tt> directive above.)
The <tt>%token_destructor</tt> directive does the same thing
for all terminal symbols.</p>

<p>Unlike non-terminal symbols, which may each have a different data type
for their values, terminals all use the same data type (defined by
the <tt><a href='#token_type'>%token_type</a></tt> directive)
and so they use a common destructor.
Other than that, the token destructor works just like the non-terminal
destructors.</p>

<a id='token_prefix'></a>
<h4>The <tt>%token_prefix</tt> directive</h4>

<p>Lemon generates #defines that assign small integer constants
to each terminal symbol in the grammar.  If desired, Lemon will
add a prefix specified by this directive
to each of the #defines it generates.</p>

<p>So if the default output of Lemon looked like this:</p>








|












|











|




|





|















|








1046
1047
1048
1049
1050
1051
1052
1053
1054
1055
1056
1057
1058
1059
1060
1061
1062
1063
1064
1065
1066
1067
1068
1069
1070
1071
1072
1073
1074
1075
1076
1077
1078
1079
1080
1081
1082
1083
1084
1085
1086
1087
1088
1089
1090
1091
1092
1093
1094
1095
1096
1097
1098
1099
1100
1101
1102
1103
1104
1105
1106
1107
1108
1109
1110
1111
1112

<p>Not like this:</p>
<pre>
   list ::= element list.      // right-recursion.  Bad!
   list ::= .
</pre>

<a id='stack_size'></a>
<h4>4.4.17 The <tt>%stack_size</tt> directive</h4>

<p>If stack overflow is a problem and you can't resolve the trouble
by using left-recursion, then you might want to increase the size
of the parser's stack using this directive.  Put an positive integer
after the <tt>%stack_size</tt> directive and Lemon will generate a parse
with a stack of the requested size.  The default value is 100.</p>

<pre>
   %stack_size 2000
</pre>

<a id='start_symbol'></a>
<h4>4.4.18 The <tt>%start_symbol</tt> directive</h4>

<p>By default, the start symbol for the grammar that Lemon generates
is the first non-terminal that appears in the grammar file.  But you
can choose a different start symbol using the
<tt>%start_symbol</tt> directive.</p>

<pre>
   %start_symbol  prog
</pre>

<a id='syntax_error'></a>
<h4>4.4.19 The <tt>%syntax_error</tt> directive</h4>

<p>See <a href='#error_processing'>Error Processing</a>.</p>

<a id='token_class'></a>
<h4>4.4.20 The <tt>%token_class</tt> directive</h4>

<p>Undocumented.  Appears to be related to the MULTITERMINAL concept.
<a href='http://sqlite.org/src/fdiff?v1=796930d5fc2036c7&v2=624b24c5dc048e09&sbs=0'>Implementation</a>.</p>

<a id='token_destructor'></a>
<h4>4.4.21 The <tt>%token_destructor</tt> directive</h4>

<p>The <tt>%destructor</tt> directive assigns a destructor to a non-terminal
symbol.  (See the description of the
<tt><a href='%destructor'>%destructor</a></tt> directive above.)
The <tt>%token_destructor</tt> directive does the same thing
for all terminal symbols.</p>

<p>Unlike non-terminal symbols, which may each have a different data type
for their values, terminals all use the same data type (defined by
the <tt><a href='#token_type'>%token_type</a></tt> directive)
and so they use a common destructor.
Other than that, the token destructor works just like the non-terminal
destructors.</p>

<a id='token_prefix'></a>
<h4>4.4.22 The <tt>%token_prefix</tt> directive</h4>

<p>Lemon generates #defines that assign small integer constants
to each terminal symbol in the grammar.  If desired, Lemon will
add a prefix specified by this directive
to each of the #defines it generates.</p>

<p>So if the default output of Lemon looked like this:</p>
︙︙
1000
1001
1002
1003
1004
1005
1006
1007
1008
1009
1010
1011
1012
1013
1014

    #define TOKEN_AND        1
    #define TOKEN_MINUS      2
    #define TOKEN_OR         3
    #define TOKEN_PLUS       4
</pre>

<a id='token_type'></a><a id='ptype'></a>
<h4>The <tt>%token_type</tt> and <tt>%type</tt> directives</h4>

<p>These directives are used to specify the data types for values
on the parser's stack associated with terminal and non-terminal
symbols.  The values of all terminal symbols must be of the same
type.  This turns out to be the same data type as the 3rd parameter
to the Parse() function generated by Lemon.  Typically, you will
make the value of a terminal symbol be a pointer to some kind of








|








1125
1126
1127
1128
1129
1130
1131
1132
1133
1134
1135
1136
1137
1138
1139

    #define TOKEN_AND        1
    #define TOKEN_MINUS      2
    #define TOKEN_OR         3
    #define TOKEN_PLUS       4
</pre>

<a id='token_type'></a><a id='ptype'></a>
<h4>4.4.23 The <tt>%token_type</tt> and <tt>%type</tt> directives</h4>

<p>These directives are used to specify the data types for values
on the parser's stack associated with terminal and non-terminal
symbols.  The values of all terminal symbols must be of the same
type.  This turns out to be the same data type as the 3rd parameter
to the Parse() function generated by Lemon.  Typically, you will
make the value of a terminal symbol be a pointer to some kind of
︙︙
1037
1038
1039
1040
1041
1042
1043
1044
1045
1046
1047
1048
1049
1050
1051
1052
1053
1054
1055
1056
1057
1058
1059
1060
1061
1062

the grammar designer should keep in mind that the size of the union
will be the size of its largest element.  So if you have a single
non-terminal whose data type requires 1K of storage, then your 100
entry parser stack will require 100K of heap space.  If you are willing
and able to pay that price, fine.  You just need to know.</p>

<a id='pwildcard'></a>
<h4>The <tt>%wildcard</tt> directive</h4>

<p>The <tt>%wildcard</tt> directive is followed by a single token name and a
period.  This directive specifies that the identified token should
match any input token.</p>

<p>When the generated parser has the choice of matching an input against
the wildcard token and some other token, the other token is always used.
The wildcard token is only matched if there are no alternatives.</p>

<a id='error_processing'></a>
<h3>Error Processing</h3>

<p>After extensive experimentation over several years, it has been
discovered that the error recovery strategy used by yacc is about
as good as it gets.  And so that is what Lemon uses.</p>

<p>When a Lemon-generated parser encounters a syntax error, it
first invokes the code specified by the <tt>%syntax_error</tt> directive, if








|










|








1162
1163
1164
1165
1166
1167
1168
1169
1170
1171
1172
1173
1174
1175
1176
1177
1178
1179
1180
1181
1182
1183
1184
1185
1186
1187

the grammar designer should keep in mind that the size of the union
will be the size of its largest element.  So if you have a single
non-terminal whose data type requires 1K of storage, then your 100
entry parser stack will require 100K of heap space.  If you are willing
and able to pay that price, fine.  You just need to know.</p>

<a id='pwildcard'></a>
<h4>4.4.24 The <tt>%wildcard</tt> directive</h4>

<p>The <tt>%wildcard</tt> directive is followed by a single token name and a
period.  This directive specifies that the identified token should
match any input token.</p>

<p>When the generated parser has the choice of matching an input against
the wildcard token and some other token, the other token is always used.
The wildcard token is only matched if there are no alternatives.</p>

<a id='error_processing'></a>
<h2>5.0 Error Processing</h2>

<p>After extensive experimentation over several years, it has been
discovered that the error recovery strategy used by yacc is about
as good as it gets.  And so that is what Lemon uses.</p>

<p>When a Lemon-generated parser encounters a syntax error, it
first invokes the code specified by the <tt>%syntax_error</tt> directive, if
︙︙
1070
1071
1072
1073
1074
1075
1076
1077




































1078
1079

<p>If the parser pops its stack until the stack is empty, and it still
is unable to shift the error symbol, then the
<tt><a href='#parse_failure'>%parse_failure</a></tt> routine
is invoked and the parser resets itself to its start state, ready
to begin parsing a new file.  This is what will happen at the very
first syntax error, of course, if there are no instances of the
"error" non-terminal in your grammar.</p>





































</body>
</html>









>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>



1195
1196
1197
1198
1199
1200
1201
1202
1203
1204
1205
1206
1207
1208
1209
1210
1211
1212
1213
1214
1215
1216
1217
1218
1219
1220
1221
1222
1223
1224
1225
1226
1227
1228
1229
1230
1231
1232
1233
1234
1235
1236
1237
1238
1239
1240

<p>If the parser pops its stack until the stack is empty, and it still
is unable to shift the error symbol, then the
<tt><a href='#parse_failure'>%parse_failure</a></tt> routine
is invoked and the parser resets itself to its start state, ready
to begin parsing a new file.  This is what will happen at the very
first syntax error, of course, if there are no instances of the
"error" non-terminal in your grammar.</p>

<a id='history'></a>
<h2>6.0 History of Lemon</h2>

<p>Lemon was originally written by Richard Hipp sometime in the late
1980s on a Sun4 Workstation using K&amp;R C.  
There was a companion LL(1) parser generator program named "Lime", the
source code to which as been lost.</p>

<p>The lemon.c source file was originally many separate files that were
compiled together to generate the "lemon" executable.  Sometime in the
1990s, the individual source code files were combined together into
the current single large "lemon.c" source file.  You can still see traces
of original filenames in the code.</p>

<p>Since 2001, Lemon has been part of the 
<a href="https://sqlite.org/">SQLite project</a> and the source code
to Lemon has been managed as a part of the 
<a href="https://sqlite.org/src">SQLite source tree</a> in the following
files:</p>

<ul>
<li> <a href="https://sqlite.org/src/file/tool/lemon.c">tool/lemon.c</a>
<li> <a href="https://sqlite.org/src/file/tool/lempar.c">tool/lempar.c</a>
<li> <a href="https://sqlite.org/src/file/doc/lemon.html">doc/lemon.html</a>
</ul>

<a id="copyright"></a>
<h2>7.0 Copyright</h2>

<p>All of the source code to Lemon, including the template parser file
"lempar.c" and this documentation file ("lemon.html") are in the public
domain.  You can use the code for any purpose and without attribution.</p>

<p>The code comes with no warranty.  If it breaks, you get to keep both
pieces.</p>

</body>
</html>
Changes to src/parse.y. 

1
2
3
4
5
6
7
8
9
10
11
12


13
14

15

16

17
18
19
20
21
22
23


/*
** 2001 September 15
**
** The author disclaims copyright to this source code.  In place of
** a legal notice, here is a blessing:
**
**    May you do good and not evil.
**    May you find forgiveness for yourself and forgive others.
**    May you share freely, never taking more than you give.
**
*************************************************************************
** This file contains SQLite's grammar for SQL.  Process this file


** using the lemon parser generator to generate C code that runs
** the parser.  Lemon will also generate a header file containing

** numeric codes for all of the tokens.

*/


// All token codes are small integers with #defines that begin with "TK_"
%token_prefix TK_

// The type of the data attached to each token is Token.  This is also the
// default type for non-terminals.
//

>

|









|
>
>
|
|
>
|
>

>








1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

%include {
/*
** 2001-09-15
**
** The author disclaims copyright to this source code.  In place of
** a legal notice, here is a blessing:
**
**    May you do good and not evil.
**    May you find forgiveness for yourself and forgive others.
**    May you share freely, never taking more than you give.
**
*************************************************************************
** This file contains SQLite's SQL parser.
**
** The canonical source code to this file ("parse.y") is a Lemon grammar 
** file that specifies the input grammar and actions to take while parsing.
** That input file is processed by Lemon to generate a C-language 
** implementation of a parser for the given grammer.  You might be reading
** this comment as part of the translated C-code.  Edits should be made
** to the original parse.y sources.
*/
}

// All token codes are small integers with #defines that begin with "TK_"
%token_prefix TK_

// The type of the data attached to each token is Token.  This is also the
// default type for non-terminals.
//
︙︙
Changes to tool/lemon.c.
︙︙
2634
2635
2636
2637
2638
2639
2640
2641


2642
2643
2644
2645
2646
2647
2648
2649

        if( *psp->declargslot ){
          zOld = *psp->declargslot;
        }else{
          zOld = "";
        }
        nOld = lemonStrlen(zOld);
        n = nOld + nNew + 20;
        addLineMacro = !psp->gp->nolinenosflag && psp->insertLineMacro &&


                        (psp->decllinenoslot==0 || psp->decllinenoslot[0]!=0);
        if( addLineMacro ){
          for(z=psp->filename, nBack=0; *z; z++){
            if( *z=='\\' ) nBack++;
          }
          lemon_sprintf(zLine, "#line %d ", psp->tokenlineno);
          nLine = lemonStrlen(zLine);
          n += nLine + lemonStrlen(psp->filename) + nBack;








|
>
>
|








2634
2635
2636
2637
2638
2639
2640
2641
2642
2643
2644
2645
2646
2647
2648
2649
2650
2651

        if( *psp->declargslot ){
          zOld = *psp->declargslot;
        }else{
          zOld = "";
        }
        nOld = lemonStrlen(zOld);
        n = nOld + nNew + 20;
        addLineMacro = !psp->gp->nolinenosflag
                       && psp->insertLineMacro
                       && psp->tokenlineno>1
                       && (psp->decllinenoslot==0 || psp->decllinenoslot[0]!=0);
        if( addLineMacro ){
          for(z=psp->filename, nBack=0; *z; z++){
            if( *z=='\\' ) nBack++;
          }
          lemon_sprintf(zLine, "#line %d ", psp->tokenlineno);
          nLine = lemonStrlen(zLine);
          n += nLine + lemonStrlen(psp->filename) + nBack;
︙︙
3612
3613
3614
3615
3616
3617
3618










3619
3620
3621
3622
3623
3624
3625

          iStart = i+1;
        }
      }
    }
    fprintf(out,"%s",&line[iStart]);
  }
}











/* The next function finds the template file and opens it, returning
** a pointer to the opened file. */
PRIVATE FILE *tplt_open(struct lemon *lemp)
{
  static char templatename[] = "lempar.c";
  char buf[1000];








>
>
>
>
>
>
>
>
>
>








3614
3615
3616
3617
3618
3619
3620
3621
3622
3623
3624
3625
3626
3627
3628
3629
3630
3631
3632
3633
3634
3635
3636
3637

          iStart = i+1;
        }
      }
    }
    fprintf(out,"%s",&line[iStart]);
  }
}

/* Skip forward past the header of the template file to the first "%%"
*/
PRIVATE void tplt_skip_header(FILE *in, int *lineno)
{
  char line[LINESIZE];
  while( fgets(line,LINESIZE,in) && (line[0]!='%' || line[1]!='%') ){
    (*lineno)++;
  }
}

/* The next function finds the template file and opens it, returning
** a pointer to the opened file. */
PRIVATE FILE *tplt_open(struct lemon *lemp)
{
  static char templatename[] = "lempar.c";
  char buf[1000];
︙︙
4283
4284
4285
4286
4287
4288
4289

4290
4291
4292
4293
4294
4295
4296

  int nLookAhead;
  int szActionType;     /* sizeof(YYACTIONTYPE) */
  int szCodeType;       /* sizeof(YYCODETYPE)   */
  const char *name;
  int mnTknOfst, mxTknOfst;
  int mnNtOfst, mxNtOfst;
  struct axset *ax;


  lemp->minShiftReduce = lemp->nstate;
  lemp->errAction = lemp->minShiftReduce + lemp->nrule;
  lemp->accAction = lemp->errAction + 1;
  lemp->noAction = lemp->accAction + 1;
  lemp->minReduce = lemp->noAction + 1;
  lemp->maxAction = lemp->minReduce + lemp->nrule;








>








4295
4296
4297
4298
4299
4300
4301
4302
4303
4304
4305
4306
4307
4308
4309

  int nLookAhead;
  int szActionType;     /* sizeof(YYACTIONTYPE) */
  int szCodeType;       /* sizeof(YYCODETYPE)   */
  const char *name;
  int mnTknOfst, mxTknOfst;
  int mnNtOfst, mxNtOfst;
  struct axset *ax;
  char *prefix;

  lemp->minShiftReduce = lemp->nstate;
  lemp->errAction = lemp->minShiftReduce + lemp->nrule;
  lemp->accAction = lemp->errAction + 1;
  lemp->noAction = lemp->accAction + 1;
  lemp->minReduce = lemp->noAction + 1;
  lemp->maxAction = lemp->minReduce + lemp->nrule;
︙︙
4371
4372
4373
4374
4375
4376
4377














4378

4379
4380
4381
4382
4383
4384
4385
4386
4387
4388
4389


4390
4391
4392
4393
4394


4395
4396
4397
4398
4399
4400
4401
4402
4403
4404
4405
4406
4407

          }
        }
      }
    }
    fprintf(sql, "COMMIT;\n");
  }
  lineno = 1;














  tplt_xfer(lemp->name,in,out,&lineno);


  /* Generate the include code, if any */
  tplt_print(out,lemp,lemp->include,&lineno);
  if( mhflag ){
    char *incName = file_makename(lemp, ".h");
    fprintf(out,"#include \"%s\"\n", incName); lineno++;
    free(incName);
  }
  tplt_xfer(lemp->name,in,out,&lineno);

  /* Generate #defines for all tokens */


  if( mhflag ){
    const char *prefix;
    fprintf(out,"#if INTERFACE\n"); lineno++;
    if( lemp->tokenprefix ) prefix = lemp->tokenprefix;
    else                    prefix = "";


    for(i=1; i<lemp->nterminal; i++){
      fprintf(out,"#define %s%-30s %2d\n",prefix,lemp->symbols[i]->name,i);
      lineno++;
    }
    fprintf(out,"#endif\n"); lineno++;
  }
  tplt_xfer(lemp->name,in,out,&lineno);

  /* Generate the defines */
  fprintf(out,"#define YYCODETYPE %s\n",
    minimum_size_type(0, lemp->nsymbol, &szCodeType)); lineno++;
  fprintf(out,"#define YYNOCODE %d\n",lemp->nsymbol);  lineno++;
  fprintf(out,"#define YYACTIONTYPE %s\n",








>
>
>
>
>
>
>
>
>
>
>
>
>
>
|
>











>
>



<
|
>
>
|
|
|
|
|
<








4384
4385
4386
4387
4388
4389
4390
4391
4392
4393
4394
4395
4396
4397
4398
4399
4400
4401
4402
4403
4404
4405
4406
4407
4408
4409
4410
4411
4412
4413
4414
4415
4416
4417
4418
4419
4420
4421
4422

4423
4424
4425
4426
4427
4428
4429
4430

4431
4432
4433
4434
4435
4436
4437

          }
        }
      }
    }
    fprintf(sql, "COMMIT;\n");
  }
  lineno = 1;

  /* The first %include directive begins with a C-language comment,
  ** then skip over the header comment of the template file
  */
  if( lemp->include==0 ) lemp->include = "";
  for(i=0; ISSPACE(lemp->include[i]); i++){
    if( lemp->include[i]=='\n' ){
      lemp->include += i+1;
      i = -1;
    }
  }
  if( lemp->include[0]=='/' ){
    tplt_skip_header(in,&lineno);
  }else{
    tplt_xfer(lemp->name,in,out,&lineno);
  }

  /* Generate the include code, if any */
  tplt_print(out,lemp,lemp->include,&lineno);
  if( mhflag ){
    char *incName = file_makename(lemp, ".h");
    fprintf(out,"#include \"%s\"\n", incName); lineno++;
    free(incName);
  }
  tplt_xfer(lemp->name,in,out,&lineno);

  /* Generate #defines for all tokens */
  if( lemp->tokenprefix ) prefix = lemp->tokenprefix;
  else                    prefix = "";
  if( mhflag ){
    const char *prefix;
    fprintf(out,"#if INTERFACE\n"); lineno++;

  }else{
    fprintf(out,"#ifndef %s%s\n", prefix, lemp->symbols[1]->name);
  }
  for(i=1; i<lemp->nterminal; i++){
    fprintf(out,"#define %s%-30s %2d\n",prefix,lemp->symbols[i]->name,i);
    lineno++;
  }
  fprintf(out,"#endif\n"); lineno++;

  tplt_xfer(lemp->name,in,out,&lineno);

  /* Generate the defines */
  fprintf(out,"#define YYCODETYPE %s\n",
    minimum_size_type(0, lemp->nsymbol, &szCodeType)); lineno++;
  fprintf(out,"#define YYNOCODE %d\n",lemp->nsymbol);  lineno++;
  fprintf(out,"#define YYACTIONTYPE %s\n",
︙︙
Changes to tool/lempar.c.
︙︙
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42

** the value of the %name directive from the grammar.  Otherwise, the content
** of this template is copied straight through into the generate parser
** source file.
**
** The following is the concatenation of all %include directives from the
** input grammar file:
*/
#include <stdio.h>
#include <assert.h>
/************ Begin %include sections from the grammar ************************/
%%
/**************** End of %include directives **********************************/
/* These constants specify the various numeric values for terminal symbols
** in a format understandable to "makeheaders".  This section is blank unless
** "lemon" is run with the "-m" command-line option.
***************** Begin makeheaders token definitions *************************/
%%
/**************** End makeheaders token definitions ***************************/

/* The next sections is a series of control #defines.
** various aspects of the generated parser.
**    YYCODETYPE         is the data type used to store the integer codes
**                       that represent terminal and non-terminal symbols.
**                       "unsigned char" is used if there are fewer than
**                       256 symbols.  Larger types otherwise.








<
<



|
<
<
|

|








18
19
20
21
22
23
24


25
26
27
28


29
30
31
32
33
34
35
36
37
38

** the value of the %name directive from the grammar.  Otherwise, the content
** of this template is copied straight through into the generate parser
** source file.
**
** The following is the concatenation of all %include directives from the
** input grammar file:
*/


/************ Begin %include sections from the grammar ************************/
%%
/**************** End of %include directives **********************************/
/* These constants specify the various numeric values for terminal symbols.


***************** Begin token definitions *************************************/
%%
/**************** End token definitions ***************************************/

/* The next sections is a series of control #defines.
** various aspects of the generated parser.
**    YYCODETYPE         is the data type used to store the integer codes
**                       that represent terminal and non-terminal symbols.
**                       "unsigned char" is used if there are fewer than
**                       256 symbols.  Larger types otherwise.
︙︙
225
226
227
228
229
230
231

232
233
234
235
236
237
238

  yyStackEntry *yystackEnd;            /* Last entry in the stack */
#endif
};
typedef struct yyParser yyParser;

#ifndef NDEBUG
#include <stdio.h>

static FILE *yyTraceFILE = 0;
static char *yyTracePrompt = 0;
#endif /* NDEBUG */

#ifndef NDEBUG
/* 
** Turn parser tracing on by giving a stream to which to write the trace








>








221
222
223
224
225
226
227
228
229
230
231
232
233
234
235

  yyStackEntry *yystackEnd;            /* Last entry in the stack */
#endif
};
typedef struct yyParser yyParser;

#ifndef NDEBUG
#include <stdio.h>
#include <assert.h>
static FILE *yyTraceFILE = 0;
static char *yyTracePrompt = 0;
#endif /* NDEBUG */

#ifndef NDEBUG
/* 
** Turn parser tracing on by giving a stream to which to write the trace
︙︙