SQLite User Forum

wasm: database serialization failed
Login

wasm: database serialization failed

(1) By Randy (randyl) on 2023-10-27 22:30:34 [source]

I'm following the cookbook recipe for downloading a Wasm database, and I get the following error:

Database serialization failed with code SQLITE_NOMEM

According to the navigator.storage API, the database is 514 MB. Am I hitting some size limit imposed by the browser (Chrome)?

Instead of using sqlite3.capi.sqlite3_js_db_export as shown in the recipe, I'm using the Promiser API's 'export' command. I can't find any documentation for it, but it is demonstrated in demo-worker1-promiser.js.

Thanks

(2) By Stephan Beal (stephan) on 2023-10-27 22:52:12 in reply to 1 [link] [source]

According to the navigator.storage API, the database is 514 MB. Am I hitting some size limit imposed by the browser (Chrome)?

Presumably yes but that's unfortunately unknowable. The library has absolutely zero insight into any limits imposed by the JS environment. All it can do is try to allocate until the environment won't let it allocate anymore.

Instead of using sqlite3.capi.sqlite3_js_db_export as shown in the recipe, I'm using the Promiser API's 'export' command. I can't find any documentation for it, but it is demonstrated in demo-worker1-promiser.js.

Indeed - i neglected to add public docs for that. They'll be added to the Worker1 docs in the next hour or two. (Promiser is a thin wrapper around the Worker1 API, so the same docs will apply there.)

The export-db case is specifically optimized to avoid a copy in that case, so it's not creating a second copy, but whether or not exporting 500+ MB, then sending it through postMessage(), will work is entirely up to the browser.

(3) By Randy (randyl) on 2023-10-28 01:46:57 in reply to 2 [link] [source]

Thanks for updating the docs.

I tried switching APIs and calling sqlite3_js_db_export() directly, but got the same SQLITE_NOMEM error. Then I deleted some data from the database and vacuumed, giving a new size of 495 MB. Exporting worked. So my original database was just over the size limit, wherever that limit is coming from.

(4) By Stephan Beal (stephan) on 2023-10-28 03:40:27 in reply to 3 [link] [source]

Then I deleted some data from the database and vacuumed, giving a new size of 495 MB. Exporting worked. So my original database was just over the size limit, wherever that limit is coming from.

It turns out that the default wasm build does indeed impose a max single-alloc limit which might be the culprit here:

src:/file?ci=trunk&name=ext/wasm/api/sqlite3-wasm.c&ln=134-136

=536mb

Why that limit was chosen (most likely by me) is lost to history but it will be increased to 2gb when i'm back on the computer (probably mid-day Saturday).

(5) By Stephan Beal (stephan) on 2023-10-28 04:12:52 in reply to 4 [link] [source]

It turns out that the default wasm build does indeed impose a max single-alloc limit which might be the culprit here:

That limit has been removed (it's now 2gb, imposed in the core lib), the prerelease snapshot has been updated, and the pending 3.44 release, as well as any hypothetical 3.43.X patch releases, will contain this fix.

(6) By Randy (randyl) on 2023-10-29 02:41:35 in reply to 5 [link] [source]

Thanks for the fix. I can confirm it works; I've successfully exported a 600+ MB database using the prerelease snapshot.