Hello,

At LumoSQL we have had interest in using our tools for benchmarking different versions of vanilla SQLite, the reason being our benchmark tool stores its results in a database and is designed for results to be easily aggregated across many unrelated systems and people. This is nothing to do with the well-known SQLite test suites; testing is not benchmarking. LumoSQL knows about VCSs and caches all source downloads, so its fairly efficient.

Doing A Benchmark Run

A benchmarking run looks like this:

     [install LumoSQL from https://lumosql.org/src/lumosql]
$ make benchmark TARGETS="3.14.15 3.33.0"
     [... output... ]
$ tclsh tool/benchmark-filter.tcl                          # list all runs in database 
$ tclsh tool/benchmark-filter.tcl RUN-ID-1 RUN-ID-2        # compare two runs
$ make benchmark TARGETS="3.20.1 3.25.2"                   # runs the benchmark on 2 more versions, adds results to same database
$ tclsh tool/benchmark-filter.tcl -target 3.20.1 -target 3.14.15 # show these 2 side by side
$ tclsh tool/benchmark-filter.tcl MANY MORE OPTIONS        # see tool's documentation

Caveats are mostly covered in the build documentation and tool documentation

Want to help? Good!

You might want to:

• consider "versions of SQLite" to include different build-time flags. We do not currently collect the build-time options in the database.
• help add to the system information collected in tool/benchmark.tcl . We intend the database schema to grow plenty.
• help add to the system identification we collect. There are pros and cons to how much identification and what kind
• try this on non-Unix. We haven't got there yet, although LumoSQL is intended to be as portable as SQLite and TCL. Things like CPU time in the TclX module are system-specific.
• make suggestions for how best to add options for hot/cold runs

We are working on a tool to easily import databases from many people, check them, and aggregate into one single database file. We intend to put this tool online for public submissions as well as making it available for people to run on their own networks.

What's Coming Next

• benchmark database aggregation at lumosql.org
• checksums for each database run
• whatever people have to contribute

Best,

The LumoSQL Authors