SQLite Forum

VFS extensions for compression & web access
Login

VFS extensions for compression & web access

(1) By mlin (dnamlin) on 2021-02-08 20:56:08 [source]

Dear all, I'd like to show & tell a couple of VFS extensions I've been hacking on:

  • zstd_vfs for read/write storage compression, similar in principle to ZIPVFS (but probably much worse)
  • web_vfs for read-only access to database files by http/https URL

I wrap these both in the Genomics Extension, along with a bunch of domain-specific features, with nicely-packaged language bindings. That's the easiest way to try them out, as the bioinformatics stuff can be ignored without harm.

My aim has been to make SQLite more applicable in a variety of "big data" settings, where now one sticks json.gz files on cloud storage (or other formats typically associated with JVM stacks).

OTOH I wouldn't recommend these tools for critical OLTP applications, as they're purely "use at your own risk" and not generally up to SQLite's quality/testing/portability standards. (I don't have time to press them there myself, but would welcome collaborators!)

-Mike Lin