sqlite wasm custom builds
(1) By Fake User (FakeUser) on 2024-10-17 10:08:53 [link] [source]
I've been using sqlite wasm in prod ~ a year now, and it's super great! Not a huge deal, but getting the 900kb wasm file and 400kb js files smaller would be a big win. So I've been trying to make some custom builds following the docs:
$ ./configure --enable-all
$ make sqlite3.c
$ cd ext/wasm
$ make
This builds sqlite3.wasm to 3.5mb and sqlite3.js to 721kb. They work in the browser, but are unfortunately much larger than theprecompiled bundle :( . I tried disabling some feature flags and rebuilding:
$ ./configure --disable-libtool-lock --disable-largefile --disable-threadsafe --disable-tcl --disable-readline --disable-amalgamation --disable-load-extension --disable-math --disable-json
$ make sqlite3.c -release
$ cd ext/wasm
$ make -release
but that didn't seem to change the size at all.
If anyone has any insight into making custom sqlite wasm builds and turning off unused features, I'd be very stoked to hear whatever you've got.
(2) By Stephan Beal (stephan) on 2024-10-17 10:24:32 in reply to 1 [link] [source]
but getting the 900kb wasm file and 400kb js files smaller would be a big win.
The pending 3.47 release has a new build mode for that which strips a good deal out of it:
$ cd ext/wasm
$ make clean
$ make barebones=1
It leaves out a many features (see ext/wasm/GNUmakefile for the the complete list). However, i've been unable to get it smaller than about 605kb, which is not a truly significant reduction.
We have no intention of publishing releases built with this mode, so it's only available to folks building on their own.
but that didn't seem to change the size at all.
It's not as straightforward as that because the JS side of the code has to be specifically taught to not try to include APIs which are omitted. If it's not, it will fail to load when it tries to bind those functions and they're not there.
The following options you list will have no effect on the wasm build:
--disable-libtool-lock --disable-largefile --disable-threadsafe --disable-tcl --disable-readline --disable-load-extension
Those features are either not relevant to wasm or are very explicitly built out of it (namely: loadable extensions and threading).
--disable-amalgamation
The wasm build requires the amalgamation, so that build mode won't work with this flag (it would pick up any stale amalgamation file you have in your tree, if there is one).
(3) By Fake User (FakeUser) on 2024-10-17 10:48:06 in reply to 2 [link] [source]
Thanks for the super fast reply! I just gave the build another go with a fresh download of sqlite-src-3460100.zip
$ ./configure --enable-all
$ make sqlite3.c -release
$ cd ext/wasm
$ make clean
$ make barebones=1
but its still building sqlite3.wasm to 3.5mb and sqlite3.js to 721kb :/
do you happen to know how the precompiled bundle is configured?
(4) By Stephan Beal (stephan) on 2024-10-17 11:42:21 in reply to 3 [link] [source]
with a fresh download of sqlite-src-3460100.zip
Barebones mode is not in 3.46. It's part of the current trunk, which will soon be 3.47.
but its still building sqlite3.wasm to 3.5mb and sqlite3.js to 721kb :/
By default it builds with -O0 because that compiles several times faster than higher optimization levels on this machine (where the WASM pieces are primarily developed and the build is often run dozens of times per day). The result is faster builds but huger wasm files.
In order to get the smallest size, build with:
$ make clean; make oz # builds with -Oz
That's how the zip files from our download page are built.
However, building with:
$ make clean; make o2 # builds with -O2
will (probably) produce slightly larger binaries but (probably) run approximately 10% faster. We consistently see that -O2 builds run faster than -O3, so don't bother with -O3.
The size of the JS file is not significantly changed by the optimization level but approximately 300kb of the JS is comments. We only strip comments for published builds, but you can do it yourself using any comment-stripping, or non-symbol-renaming minifier, tool you have handy. The one we use is...
$ make ../../tool/stripccomments
$ ../../tool/stripccomments < infile > outfile
Noting that for release builds we retain the license-header comments using the -k flag to that tool, as demonstrated in dist.make
.
(5) By Fake User (FakeUser) on 2024-10-18 00:27:16 in reply to 4 [link] [source]
Oops, my bad. I missed barebones being in the upcoming 3.47 release. Super excited for that! Following your instructions I got a build that's the same size as the release build.
$ ./configure --enable-all
$ make sqlite3.c -release
$ cd ./ext/wasm
$ make oz
I kept nosing around and tried omitting some things in the /ext/wasm/GNUMakefile
SQLITE_OPT = \
-DSQLITE_OMIT_FTS5 \
-DSQLITE_OMIT_RTREE \
-DSQLITE_OMIT_EXPLAIN_COMMENTS \
-DSQLITE_OMIT_UNKNOWN_SQL_FUNCTION \
-DSQLITE_OMIT_STMTVTAB \
-DSQLITE_OMIT_DBPAGE_VTAB \
-DSQLITE_OMIT_DBSTAT_VTAB \
-DSQLITE_OMIT_BYTECODE_VTAB \
-DSQLITE_OMIT_OFFSET_SQL_FUNC \
-DSQLITE_OMIT_LOAD_EXTENSION \
-DSQLITE_OMIT_DEPRECATED \
-DSQLITE_OMIT_UTF16 \
-DSQLITE_OMIT_SHARED_CACHE \
-DSQLITE_THREADSAFE=0 \
-DSQLITE_TEMP_STORE=2 \
-DSQLITE_OS_KV_OPTIONAL=1 \
'-DSQLITE_DEFAULT_UNIX_VFS="unix-none"' \
-DSQLITE_USE_URI=1 \
-DSQLITE_WASM_OMIT_C_TESTS \
-DSQLITE_C=$(sqlite3.c)
This brought sqlite.wasm down to 822kb and sqlite.js down to 701kb (192kb minified), and they still work in the browser for my use case. Not huge, but going in the right direction. Thanks for your help. If you have any tips on how you golfed it down to 605kb I am all ears, but if you have other things to focus on don't sweat it.
(6) By Stephan Beal (stephan) on 2024-10-18 00:42:25 in reply to 5 [link] [source]
If you have any tips on how you golfed it down to 605kb I am all ears
In the trunk ext/wasm:
make barebones=1
should be all you need to do, IIRC (but am away from the computer so cannot readily confirm that). If that doesn't do it, try:
make clean; make oz barebones=1
(7) By Fake User (FakeUser) on 2024-10-18 01:44:19 in reply to 6 [source]
Dude thank you! Building with barebones=1 from the latest trunk totally worked. sqlite3.wasm ends up being 602kb and sqlite3.js 676kb (186kb minified). Heads up, running: "make barebones=1" throws the error:
GNUmakefile:230: *** missing separator. Stop.
Deleting line 230 "undefine barebones" fixed it. It also needed to be run with:
make oz barebones=1
But after that it purrs like a kitten. Thanks again dude, cheers.
(8) By Stephan Beal (stephan) on 2024-10-18 12:25:44 in reply to 7 [link] [source]
Deleting line 230 "undefine barebones" fixed it.
That means your GNU Make predates undefined
. We don't strictly need that line, so i'll remove it, but i cannot promise that i'll avoid newer-than-than GNU Make features in the future (the Make docs don't generally say when a given feature was added, and the benchmark for that particular makefile is "it works for me!").
(9) By Warren Young (wyoung) on 2024-10-18 14:43:57 in reply to 8 [link] [source]
That feature was added to GNU Make in 3.82, way back in July 2010!
I sense GPLv3 avoidance..
(10) By Adrian Ho (lexfiend) on 2024-10-18 15:47:36 in reply to 9 [link] [source]
I suspect FakeUser is running macOS. Apple froze their system GNU make at v3.81, probably for the reason you cited.