SQLite Forum

Using parser to do extra work

Using parser to do extra work

(1) By daniel_xu_forever on 2020-03-16 05:15:15 [link] [source]


Recently, I'm reading the source of code of sqlite especially the Parser part.

AFAIK, the parser is a shift-reduce parser generated by lemon.

  sqlite3Parser(pEngine, 0, pParse->sLastToken, pParse);

however, when I check the structure of pParse, there's no statement root that points to the root of AST.

Am I missing it or it's on purpose? or what else could I use?

What I really want to do is be able to analyze the SQL statement and maybe rewrite it if some predefined rules are matched. In the meantime, I wanna keep the original SQL. For example:

we got SQL:

"select name from student where id = 1;"

I want to re-write it as:

"select ? from ? where id = ?"

Look forward to hearing from you guys. Thanks in advance.

(2) By Gunter Hick (gunter_hick) on 2020-03-19 08:42:56 in reply to 1 [link] [source]

Sorry but the select list and the table name do not support parameters. The reason being that these items are checked during the prepare phase, whereas a parameter would imply that you could change the field or even the table during execution of the statement

IIRC that request is outside of SQL Standard.

(3) By ddevienne on 2020-03-19 13:04:29 in reply to 1 [link] [source]

AFAIK, SQLite's parsing does not build an AST per-se, but directly the
execution engine for the statement. Of course VDBE is the real engine
but in any case, what parsing produces is not an AST you can massage
and then pass on to a next-phase to produce a different execution engine.

The grammar actions are not simply assembling AST nodes, they do more complex
stuff I won't try to explain, since I don't really understand it :). --DD

PS: Request for an official AST of SQLite's SQL, provided and supported by the
SQLite authors, was requested in the past, but it's unlikely to happen.

(4) By Richard Hipp (drh) on 2020-03-19 14:53:44 in reply to 3 [link] [source]

The style of implementation for the parse in SQLite is closer to syntax directed translation. A full AST is generated for SELECT statements, but for INSERT, DELETE, UPDATE, CREATE, DROP, and so forth, no AST is ever constructed. Instead, the code generator is passed expression trees and identifiers as individual arguments.

The format of the AST is not an API. We change it constantly, as we add new features and improve performance. If we made the AST an API, that would lock it down forever, and prevent us from improving SQLite moving forward. We are unwilling to do that. The AST (and the bytecode) are implementation details that can, and usually do, change from one release to the next.

(7) By daniel_xu_forever on 2020-03-23 04:14:36 in reply to 4 [link] [source]

Thanks for the replies, I get your point.

I'm still wondering if the follwing X which is type of Select * is the AST you are talking about

cmd ::= select(X).  {
  sqlite3Select(pParse, X, &dest);

(5) By anonymous on 2020-03-19 20:38:30 in reply to 1 [source]

If you want to do advanced pre-processing of SQL statements for sqlite3 you can checkout this project. This project uses lemon as the parser generator and if you browse the source code you will see it is actually bootstrapped with the tool itself. Sorry drh, its hosted on github.


The project 100% functional for what I want it to do, but I have not cleaned up the code to be pretty and tidy up the comments yet. Check out the README for a quick summary of what it is doing and feel free to ask any questions.

(6) By Richard Hipp (drh) on 2020-03-19 21:11:15 in reply to 5 [link] [source]

Sorry drh, its hosted on github.

I don't care if you use Git, as long as you don't make me use it. :-)