Hey everyone,

This is Asen Alexandrov and I am posting on behalf of the WasmLabs team @ VMware OCTO.

As part of our work on patching PHP to build for WebAssembly (in particular, for the wasm32-wasi target) we had to build sqlite3 as a library module for wasm32-wasi [1]. WASI (https://wasi.dev) is a system interface that allows Wasm to run on the server side by safely being able to access the filesystem, etc.

To achieve that some code changes and specific build configure options were necessary (leveraging prior work by Rui Carmo and Nicolas Holzschuch) [2].

In particular, the patch covers the following key issues:

1. Some POSIX calls are not available on WASI, so they had to be disabled in os_unix.c

2. The VFS interface for “unix” was switched to “dotLockIoFinder” as posix file locking is not supported with wasi. And add a stub fcntl.h was added to re-declare some lock type constants from fcntl.h

3. A specific selection of CFLAGS defines and configure options to make sure we build a flavor of libsql that does work on wasi.

We see that there is an extensive effort to provide Wasm support for the browser and some of the things we patched are also addressed in sqlite3-wasi.h. It would also be great if people could build sqlite3 for wasm32-wasi out of the box and provide server-side support as well. To help the community get there we would like to engage with the sqlite developers and (following all best practices, of course) to polish and improve the patch and contribute it upstream to the sqlite codebase.

Can anyone suggest what the best next steps would be? Is there a better forum to ask for guidance?

Thanks,

Asen

[1] https://wasmlabs.dev/articles/running-wordpress-with-mod-wasm/

[2] https://github.com/vmware-labs/webassembly-language-runtimes/tree/main/libs/sqlite

We see that there is an extensive effort to provide Wasm support for the browser and some of the things we patched are also addressed in sqlite3-wasi.h. It would also be great if people could build sqlite3 for wasm32-wasi out of the box and provide server-side support as well.

That is a long-term goal of ours, but we first have to get all the way out of the gate with a single target (browsers). 3.40 will be the first public beta and we anticipate solidifying the current APIs with 3.41. It's unlikely that we'll dig into any other targets until Q1 2023 sometime, maybe even early Q2. That limitation is based solely on team bandwidth: we have only a single developer working on wasm (🙋‍♂️) and a pending cross-country move (to an as-yet-undiscovered new home) is going to interfere with coding through at least March.

i spent several weeks last summer getting the sqlite bits building with wasi and vanilla clang, but both were missing library-level features which we currently require (most notably Emscripten's transparent POSIX I/O proxying). We didn't go so far as to patch sqlite3.c for that - all testing was done with vanilla builds of the amalgamation. We don't currently have a build of sqlite3 which can host a working db with either wasi or vanilla clang :(. The in-memory db support requires one of the other VFSes for certain functionality, so it's not yet possible to build an in-memory-only build of sqlite. Those are all aspects which we aim to improve upon once the bandwidth frees up to work on them. Achieving usable sqlite3.wasm builds across multiple wasm toolchains is a topic near and dear to my heart.

In particular, the patch covers the following key issues:

That is all of definite interest :).

... to polish and improve the patch and contribute it upstream to the sqlite codebase.

One catch there is that we cannot accept patches from non-members due to licensing concerns (see https://sqlite.org/copyright.html). We can, and sometimes do, use such patches as a basis for study in implementing equivalent patches, though.

Can anyone suggest what the best next steps would be? Is there a better forum to ask for guidance?

Currently this forum is the preferred place for public conversations on the topic, and you can email me directly if you prefer off-list contact.

If this subproject generates "too much" noise in this forum we may move it over to the /wasm repository or a new one altogether. That's as yet undecided - we're waiting to see whether wasm-related traffic will be high enough to become a nuisance for the folks following the core library (@such folks, please feel free to provide feedback on that if the wasm traffic crosses your pain thresholds).

Thank you for the swift response!

One catch there is that we cannot accept patches from non-members due to licensing concerns (see https://sqlite.org/copyright.html)

Do you mean sqlite consortium membership or something else? For example is it a matter of me or a colleague signing documents and stuff to avoid any legal implications with suggested code changes or something more complex?

For example is it a matter of me or a colleague signing documents and stuff to avoid any legal implications with suggested code changes or something more complex?

The project requires a document on file but it doesn't accept that document from arbitrary people. It's effectively "by invitation only" and Richard hands out those invitations at his sole discretion.

In any case, i have no doubt that we can put our heads together on this and come up with agreeable solutions. Please check your email - you should see one from me about arranging a meeting.