SQLite Forum

Native support for pluggable backends in SQLite
Login
Dan Shearer (danshearer) on 2020-11-13 15:47:39:

I received a comment on the section I wrote:

> ... "What's Missing... Nobody Really Knows :-)"

> In terms of performance, reliability and tradeoffs, nobody knows what is missing. That is because there is no consistent way of comparing one version of SQLite with a modified version, and certainly not for repeating these comparisons across many different users, systems and build environments. That's why we built benchmarking, which also works for vanilla SQLite-on-SQLite.

It would have been better to say *"The LumoSQL project hasn't been able to find published benchmarking"*, because there may be unpublished benchmarking, and there are experts on this forum who have an excellent understanding on how to make SQLite run fast.

The [LumoSQL Knowledgebase](https://lumosql.org/src/lumodoc/file?name=doc/lumo-relevant-knowledgebase.md) uses better words on this point under the heading "List of Relevant Benchmarking and Test Knowledge". The paper [Dangers and complexity of sqlite3 benchmarking](https://www.cs.utexas.edu/~vijay/papers/apsys17-sqlite.pdf) listed says:

> ...changing just one parameter in SQLite can change the performance by 11.8X... up to 28X difference in performance

That doesn't mean the benchmarking itself is so hard, but I'm not aware of anyone publishing like-for-like comparisons.

Dan Shearer