Building JS/WASM

This page introduces the various builds of the WASM/JS code and how to build customized versions.

sqlite3.js and sqlite3.wasm are the "standard builds." They are built for wide deployment on a number of browsers.

The GNUmakefile in the ext/wasm directory of the source tree creates those files. The rest of this document describes what goes in to building them, so that clients may create custom builds and adapt the builds for toolchains other than Emscripten.

These instructions assume:

WASM Toolchain Compatibility. Though the library's JS API is independent of any specific WASM environment or toolchain, it currently (early 2023) relies on certain services which are, as of this writing, only available from the Emscripten WASM toolchain (most notably, virtualization of the POSIX C I/O APIs). The JS API is structured such that any WASM-related information which is required by the library is provided via a configuration object, to facilitate creation of builds which are customized for other WASM environments.

Foreword: Target Platforms

This project's primary deliverable is "lowest common denominator" JavaScript which can be used in a wide variety of environments, with web browsers taking priority over server-side JS engines. We do not directly support the wide array of sub-platforms, e.g. the diverse node.js-based tools, because:

Where the "vanilla" JS does not work with with a given third-party environment we are happy to help users integrate it, but we require active cooperation from such environments' users in order to do so. Such cooperation invariably requires back-and-forth interaction with the developers as the quirks of any given environment are discovered and accommodated.

By and large, we leave the development of environment-specific builds, e.g. npm, to those who have an active interest and need for them. We do offer one npm build but do not guaranty that it works with every combination of tools from that ecosystem. We also provide separate builds of the core JS files intended to be compatible with certain tools commonly seen in node.js-based build environments, e.g. "bundlers." See the API index for details.

It is not this project's goal to become The One True SQLite WASM Distribution and we welcome both alternative distributions and alternative implementations altogether, in particular ones which cover use cases which we lack the bandwith and topical expertise do justice to.

Quickstart: the Canonical Builds

The canonical build files (i.e. those which live in sqlite3's own source tree) support several different build targets which might be useful to folks other than its own developers. They require a full checkout of the sqlite3 source tree (as opposed to an amalgamation distribution), GNU Make, and the Emscripten SDK. The wabt tools are also strongly recommended.

At its simplest, do the following from the top of a checked-out copy of the sqlite3 source tree:

$ ./configure --enable-all
$ make sqlite3.c
$ cd ext/wasm
$ make

Tip: a parallel build, e.g. with make -j4, should work just fine, but getting it to do so reliably requires running the make sqlite3.c step in advance.

About build reproducibility: any given build may be different, even with the same sqlite source tree, depending on which version of Emscripten is used. The JS code Emscripten generates changes from version to version.

Useful targets include:

Much experimentation has shown that -O2 provides the fastest binaries, but -Oz provides the smallest (noting that wasm-strip is required to make them small). The speed difference between -O2 and -Oz builds is typically only about 10%.

Each build produces a number of deliverables, including:

The remaining deliverables in the top-level directory are various demos and tests, neither necessary nor (in all likelihood) interesting for most folks.

64-bit Integers

JavaScript's core Number type does not support 64-bit integers, which causes some friction between WASM and JS regarding 64-bit integers (whereas both support 64-bit floats).

Many sqlite3 APIs take or return int64 values, but all such APIs are disabled unless sqlite3.js is built with support for JavaScript's BigInt type. This is a build-time option.

The canonical builds enable BigInt support by default. Custom builds may disable it but must be aware that all sqlite3 C APIs which take int64 arguments or return them will not be available in the JS bindings.

WASM Heap Memory

One of WASM's design features is how WASM modules are given RAM to work with. They must have one or the other of the following:

Depending on how the build is configured, it may or may not be possible for the WASM module to grow its amount of memory at runtime. If it cannot, and it runs out of memory, allocation attempts will fail. Whether or not that is outright fatal to the WASM module is determined by a build option. If it is not fatal, application code must check for out-of-memory conditions on its own, as it would in C. If it is configured to be fatal, the module will fail loudly (in the browser's dev console, not necessarily someplace client-visible) and stop working if any allocation fails.

The heap memory size is one of a staggering number of build options which clients may want, or need, to customize for a given deployment. The default heap memory assigned to the module was chosen based on testing and development of the sqlite3 module. As experience is garnered via 3rd-party client applications, future releases of the module may increase the initial amount of memory the WASM module receives.

Preprocessing JS Files

In order to be able to support both conventional JavaScript and ES6 modules (a.k.a. ESM) in the same source files, we must "preprocess" certain source files to filter specific code sections in or out depending on whether they're being built for conventional JS or ESM. Because C preprocessors make a mess of things when used for preprocessing JS code, a custom preprocessor was created specifically for use in this project, colloquially known as the C-Minus Preprocessor, or c-pp. It is maintained as a standalone side project but the sqlite3 source tree contains its own copy.

Specific JS files cannot be used as-is, and instead must be run through c-pp. Though this complicates the creation of custom builds somewhat, it seems to be the "lesser evil" in terms of approaches for maintaining the JS code in such a way as to be usable by both flavors of JS.

Tip: files which require preprocessing have a file extension of .c-pp.X, where X is the typical extension for that file type, e.g. html or js.

The preprocessor's directives do not directly interfere with the source code but do lead to constructs which, if run in JS without preprocessing them, will lead to (at best) syntax errors or (at worst) potentially silent errors. Here's an example snippet of a preprocessed block:

const W =
//#if target=es6-module
  new Worker(new URL(options.proxyUri, import.meta.url));
//#else
  new Worker(options.proxyUri);
//#endif

The JS keywords import and export trigger syntax errors in non-ESM code, so may not be filtered out at runtime via JS's introspection features. The capabilities they provide are essential for client-side use of ESM modules, so cannot simply be hand-waved away as a "nice-to-have." i.e. we have to support ESM, one way or another, and preprocessing gives us a relatively painless way of doing so.

The preprocessor supports a configurable prefix for its keywords and the JS code in this tree uses the prefix //# so that such constructs do not directly interefere with syntax highlighting and code indentation in JS-aware text editors. The preprocessor's keywords may not be indented: they must start on column 0.

After preprocessing, the above block is reduced to one of the following:

const W =
  new Worker(new URL(options.proxyUri, import.meta.url));

or:

const W =
  new Worker(options.proxyUri);

Use of the preprocessor is kept to a minimum. As of this writing, it is used only to block of code which requires the import ESM keyword. All code using the preprocessor can be found by grepping the JS files for '^//#'.

The complete technical details of such preprocessing are maintained in GNUMakefile.

SQLite Encryption Extension (SEE)

The build process supports building with the SQLite Encryption Extension (SEE) variant of the SQLite amalgamation source file. See see.md for details.

The C Files

Building sqlite3.wasm requires two local C files and one local H file, plus whatever system-level headers it requires:

When compiling, only sqlite3-wasm.c gets compiled. It #include's sqlite3.c because it requires access to some of the latter file's internal-use-only state. It also #define's a number of WASM-relevant configuration flags for sqlite3.c. Because it uses state internal to sqlite3.c, sqlite3-wasm.c is only intended to be built with the a sqlite3.c generated from the same version of the project's source tree.

The JS Files (sqlite3-api.js)

The oft-mentioned sqlite3.js is generated out of numerous other files by a build process. This section provides an overview of those files as a guide for those who would like to create custom builds.

Using the ext/wasm directory as the base directory for refering to files, the following files are used to create the JS amalgamation, listed in the order in which they need to be assembled:

Files with the extension .c-pp.js are intended to be processed with c-pp, noting that such preprocessing may be applied after all of the relevant files are concatenated. That extension is used primarily to keep the code maintainers cognisant of the fact that those files contain constructs which will not run as-is in JavaScript.

The result of concatenating those files (except for the ones noted as being external) is typically named sqlite3-api.js. That file contains the entire JS API but it requires sqlite3.wasm to be loaded before it is bootstrapped, a process which exists primarily to configure the JS parts to work with the current WASM environment. The bootstrapping requires pieces specific to each WASM build environment and happens in sqlite3-api-cleanup.js. With a custom build it would be possible to delay the bootstrapping further by replacing parts of sqlite3-api-cleanup.js, perhaps with the goal of enabling the end client to extend the library further before bootstrapping or to provide a more customized configuration to the bootstrap process.

sqlite3-api.js may, depending on the build environment, be compounded further into other JS files.

For example, in Emscripten-driven build, that file gets sandwiched between additional files which wrap it up in such a way that the Emscripten module-initialization process will activate it. That bundle then gets included into the resulting Emscripten-generated sqlite3.js, which includes not only the sqlite3 API but also all of the infrastructure needed for loading sqlite3.wasm and various utility code provided by Emscripten.

The following files are part of the build process but are injected into the build-generated sqlite3.js along with sqlite3-api.js.

Building the JS/WASM Files

Aside from the tools mentioned below, the wabt tools are also strongly recommended, primarily for wasm-strip.

Emscripten

The Emscripten-centric build process is covered on the Emscripten page.

wasi-sdk

sqlite3 can be built with wasi-sdk but cannot currently do much because that SDK has no transparent I/O proxy support like that provided by Emscripten. TODO is creating a build of sqlite3 which contains only those VFSes which do not require POSIX I/O. Currently (2022-11) the memdb VFS relies on the availability of a lower-level VFS so it is not possible to create a build which contains only that VFS. Improving support for wasi-sdk builds is ongoing.

Here is a mini-HOWTO for setting up the wasi-sdk on an Ubuntu-style Linux system:

$ git clone --recursive https://github.com/WebAssembly/wasi-sdk.git
$ cd wasi-sdk
$ sudo apt install ninja-build cmake clang
$ NINJA_FLAGS=-v make package
# ^^^^ go order a pizza, wait for it to arrive, eat it, and
# check back. Maybe it'll be done by then. Maybe. As of this writing,
# it has to compile more than 3000 C++ files.
$ sudo ln -s $PWD/build/wasi-sdk-* /opt/wasi-sdk
# ^^^^^^^ /opt/wasi-sdk is a magic path name for these tools

Or, much faster, grab a prebuilt SDK binary from that git page, unpack it somewhere, then symlink /opt/wasi-sdk to point to it.

However, as mentioned above, a WASM build of sqlite3 created with this toolchain does not yet function.

Plain clang

Like the wasi-sdk build, it is possible to compile sqlite3.wasm using vanilla clang, but the resulting binary is not usable as-is due to missing JS/WASM functionality. Improving that is on the long-term TODO list.

Building for SQLite Encryption Extension (SEE)

Since 2023-03-08, the canonical WASM build supports building against the SQLite Encryption Extension (SEE) by either:

The resulting jswasm/sqlite3.wasm will include the SEE-related APIs and jswasm/sqlite3.[m]js will add those APIs to the sqlite3.capi namespace, including automatic string-type argument conversion.

Caveats:

Exporting New APIs to JS

Exporting new APIs to WASM requires more than simply including them in the sqlite3.c amalgamation and enabling any required feature flags. In short, it requires:

  1. Add them to the list of exported functions for the current build environment. For the Emscripten-based builds, that means adding them to ext/wasm/api/EXPORTED_FUNCTIONS.sqlite3-api, prefixing each function with an underscore character (because Emscripten requires it).
  2. If desirable (it normally is), adding a description of the binding to the JS code, which ensures that it will get installed into the sqlite3.capi namespace.

These steps are described below, with the caveat that ⚠ these are internal details subject to change at any time ⚠! The specifics of how C functions get exported into the WASM and JS APIs are not part of the public interface of the library and are at least partially dependent on the build platform used for creating the WASM file (i.e. part of it is Emscripten-dependent).

That said...

Extending the Exported Functions List

ext/wasm/api/EXPORTED_FUNCTIONS.sqlite3-api is a plain text file listing all C functions which must be exported into the resulting sqlite3.wasm. Every function name is preceeded by an underscore because Emscripten requires it. For maintenance's sake, the list should be kept in alphabetical order. It is acceptable for exports to be listed here but not exposed to JS.

If/when other build platforms are available, they may have their own function exports list.

Adding sqlite3.capi Bindings

Once a function is exported from WASM, we can access it via the sqlite3.wasm.exports namespace, but such bindings are in their "raw" form, meaning that no type conversion is performed on their arguments or result values, e.g. for string arguments. In order to expose the API to the public interface and add any necessary type conversion, we must add an entry to the internal list of bindings.

⚠ ACHTUNG: again, what follows is an internal implementation detail which is subject to change at any time. Do not rely on these instructions from client-level code.

The mapping between WASM-exported functions and their JS-exposed counterparts are stored in ext/wasm/api/sqlite3-api-glue.js in one of three places, depending on the role of the function and whether or not the function uses 64-bit integers in its arguments or result value. The bindings belong in one of the following arrays:

Each of those arrays contains sub-arrays which describe the function arguments and return type, as documented for sqlite3.wasm.xWrap(). During library initialization, a type-converting proxy for each listed function is injected into one of the namespaces described above. If any function listed is missing from the WASM exports, an exception will be triggered and library initialization will fail.

When adding new entries to one of the function-list arrays, it is important that all result and argument types refer to xWrap() mappings, noting that a number of them are extensions defined in that same file and are not documented in the xWrap() docs (because they are extensions, not core-defined conversions). Any typos or unknown type names will lead to an exception during library initialization.