SQLite Forum

An sqlite extension for transparent row-level compression using zstd
Login
I'm posting this because I think the idea is great and has lots of potential, but don't expect anything really usable yet.

[sqlite-zstd](https://github.com/phiresky/sqlite-zstd) is a loadable sqlite extension that adds zstd compression functions, as well as dictionary training. 

It also includes functionality that makes a column of a table transparently compressed, by training a dictionary on the existing data, compressing it and replacing the table with an updatable, uncompressed view of the compressed data. This means if you have redundant data (such as JSON) in the database, the data storage will pretty much take only as much as the actual entropy of the data requires, while retaining the nice random access performance etc of sqlite.

The existing extension functions including transparent compression already work fine, it's mostly missing flexibility.

Development is stalled right now since the sqlite-extension API for Rust is not stable yet (depends on an [unmerged PR](https://github.com/rusqlite/rusqlite/pull/631)).