The WASM modules for SQLite aren't the easiest to import and use. Mainly because they don't conform to ESM

If you can tell us what ESM is, we can look into improving that. The builds are created by Emscripten, which is "the" WASM toolkit, and we're kind of at its mercy in terms of how it packages things. For better or worse, we're currently tied to Emscripten's way of doing things, an that is unlikely to change before mid next year at the earliest.

and secondarily because they can't find the wasm and opfs-proxy locations on their own.

See https://sqlite.org/wasm/doc/trunk/gotchas.md

for why that is.

import sqliteWasm from "sqlite-wasm-esm";

We haven't looked into ES6 modules at all because, frankly, they're simply not portable enough yet and are far too limiting in how they permit stuff to be loaded. e.g. Firefox cannot load ES6 modules from a Worker.

What we've done is essentially a collection of ugly find & replaces against the sqlite3.js that you all provide.

See the "gotchas" link above for an easier (IMO) solution. i struggled for several days looking for a 100% transparent way for it to find the wasm file based on the js file, but discrepancies in how relative URIs are resolved in JS APIs, depending on how the JS file is loaded, make that literally impossible to do 100% transparently. We cannot tell clients "you may only load this from the main thread" or "you may only load this from a worker constructor" or "you may only load this via importScripts()." Each approach has its pros and cons, and we have features which work in Workers but no the UI thread, and vice versa. All three approaches have to be able to work.

Any chance you all would either: (a) update the wasm artifacts to be esm compliant

If you mean ES6 modules the answer is, for the time being, unequivocally no. ES6 modules are way down the list of things to investigate and explore, and a pending move back to Berlin is forcing me to be exceptionally choosy about what gets explored and improved for the time being. My schedule won't free up for full-time hacking again until, best case, March 2023. Until then, i'm going to have to say "no!" more often than i'd care to ;).

That said: i've tried to structure the main JS files (sqlite3-api-*.js) in such a way that we can easily repackage them for use with arbitrary WASM environments and near-arbitrary packaging schemes (like ES6 modules). Only practice will reveal whether that goal has truly been reached or not.

and make use of import.meta.url

No idea what that is :/ but i will look it up. sqlite has always been a C-centric project, and this foray into JS and WASM is new for us. There will be a learning curve for us and we currently only have one developer dedicated to these components (/me raises hand), so we will move slowly at times. You have my apologies for that, but there's no current workaround for it.

or (b) allow us to submit patches to the relevant source files (extern-post, extern-pre, post-js-footer, post-js-header)?

This project does not accept patches from non-members for reasons described in detail at:

https://www.sqlite.org/copyright.html

Note that all of the files named pre/post-anything are Emscripten-specific. They're there in order to account for Emscripten-isms and, in some cases, replace Emscripten functionality when we can find a better way of doing the same thing and Emscripten provides a hook for us to do so (like the relative URI "gotcha" at the link above - we have to override certain Emscripten functionality to get that working at all).

What we can do is use posted patches as a basis for implementing similar functionality, so please continue to post your improvements and we will certainly get around to investigating them. What i cannot currently promise, however, is a quick turnaround. Real Life has unpredictably collided at an unfortunate angle with the public beta phase for the JS/WASM pieces and that will slow me down considerably.

Yeah ESM == ES6 Modules.

In terms of portability, I wonder how many people still do JS development without a build tool (babeljs/vite/parcel/etc.) to iron out all the incompatible features.

But yes, if you want to ensure nobody ever has to use a build tool to use your module then you'll need to preclude the use of ES6.

We'll keep working on our wrapper as I think it creates a better developer experience for those either using modern JS build tools (and thus getting ES6 module support everywhere -- even workers) or environments (e.g., Deno) that support ES6 modules.

i've tried to structure the main JS files (sqlite3-api-*.js) in such a way that we can easily repackage them for use with arbitrary WASM environments and near-arbitrary packaging schemes (like ES6 modules)

I'll look over these again to see if we can streamline how we are re-packaging things into ES6 modules.

No idea what that [import.meta.url] is

It's metadata about where the currently executing ES6 module lives. JS build tools are also aware of this property and re-write it when targeting environments that don't support ES6 modules.

i'm going to have to say "no!" more often than i'd care to ;).

np. I've always been bad at saying "no!" and applaud you in holding the line.

I wonder how many people still do JS development without a build tool (babeljs/vite/parcel/etc.) to iron out all the incompatible features.

We don't use any JS build tools directly. We use the emsdk (Emscripten) to compile its module, but all of our own JS code is lovingly hand-written in Emacs and structured in such a way that we can eventually plug it in to arbitrary wasm environments. (We have built and loaded the library with other wasm toolchains, but they're missing certain C/WASM library-level support which we currently rely on.) It will be a cold day in That Really Hot Place before we start using any node.js-based tools for development.

Folks are more than welcomed - even encouraged - to make any changes they need to fit it into their non-GNU-make-based build systems, but the odds that my cold, cold heart will ever warm up to any of the node.js-based tools are slim to none. Been there, done that, and don't care to return to it.

But yes, if you want to ensure nobody ever has to use a build tool to use your module then you'll need to preclude the use of ES6.

i knew that the request for es6 modules would come eventually, and i am most definitely not averse to exploring that option, but other priorities will, in all likelihood, push it back until early next year.

That said: if you want to provide a how-to to re-packaging it to create an es6 module out of it, or (even better), show us how to add that support in such a way that em6 modules would automatically make use of it but conventional loading would still behave properly, we would be all for that. If we can pack it into a single deliverable, that would be fantastic, but a second option which produces an es6-only deliverable would be a welcomed second choice.

BTW, you're welcome to contact me off-list about this if you prefer. If there's enough interest/participants specific to the JS/WASM bits, we may consider opening up the forum on the /wasm repo for those topics. We've discussed that internally but have left it as an open point pending how much traffic those topics end up consuming on the forum.

My current ability to commit time to es6 module support is directly tied to how much support i can get from someone like you who knows their way around them (my practical experience with them is literally zero, so without such support it's going to be a slow, uphill trip). With your help, perhaps we can burn through it quickly.

Forgot to mention:

https://github.com/overtone-app/sqlite-wasm-esm/blob/main/build/build.sh

That is extremely fragile. The symbol names you're referring to can change in literally any given commit. Nothing the sqlite3.js bundle which is not attached to the sqlite3 namespace object is part of any stable interface and subject to any number of changes or removal at any time. Even the ostensibly public APIs are currently in "public beta" and subject to change, based (we hope) on feedback from folks who've tried them. It's long been my experience that 3rd-party users often have the best feature/improvement suggestions.

Currently the only 100% set-in-stone APIs are the direct exports of the C APIs. As we leave the public beta phase (perhaps 1-2 sqlite releases? We don't yet know) we will apply strong stability constraints on future API changes.

If we can pack it into a single deliverable, that would be fantastic, but a second option which produces an es6-only deliverable would be a welcomed second choice.

An example of what i mean by that first option:

[stephan@nuc:~/f/s/lite/ext/wasm]$ f diff
Index: ext/wasm/api/sqlite3-api-opfs.js
==================================================================
--- ext/wasm/api/sqlite3-api-opfs.js
+++ ext/wasm/api/sqlite3-api-opfs.js
@@ -164,11 +164,18 @@
     }/*metrics*/;      
     const promiseReject = function(err){
       opfsVfs.dispose();
       return promiseReject_(err);
     };
-    const W = new Worker(options.proxyUri);
+    const W = (function(){
+      const args = [options.proxyUri];
+      if('undefined'!==typeof imports && imports.meta && imports.meta.url){
+        // es6 module
+        args.push(imports.meta.url);
+      }
+      return new Worker(...args);
+    })();
     W._originalOnError = W.onerror /* will be restored later */;
     W.onerror = function(err){
       // The error object doesn't contain any useful info when the
       // failure is, e.g., that the remote script is 404.
       error("Error initializing OPFS asyncer:",err);

i'll do what i can to encapsulate your build.sh changes into constructs like that. If you have any concrete suggestions for such changes, please let me know.

BTW, this part:

cat <(echo "export default function install(wrapper) {") lib/sqlite3.js <(echo "wrapper.self = self; }") > dist/sqlite3.js

can be more clearly written (IMO) as:

{
  echo "export default function install(wrapper) {"
  cat lib/sqlite3.js
  echo "wrapper.self = self; }"
} > dist/sqlite3.js

An example of what i mean by that first option:

My bad: we cannot use the "import" keyword in non-module code. Trying the above patch (with the corrected s/imports/import/g) leads to the module failing to start with this error from Chrome:

Uncaught SyntaxError: Cannot use import statement outside a module

We'll need to find a different approach to building for es6 modules which avoids that symbol in non-esm builds. That might mean breaking down the scripts into smaller parts and using esm-specific pieces for some parts. i'll mull that over. Maybe simply running it through a C pre-processor would suffice, with different defines for ems and conventional builds.

Your current approach to sed'ing the script is not maintainable long-term and i can't commit to trying to remember to not change any given line just to keep 3rd-party sed scripts working.

Maybe simply running it through a C pre-processor would suffice, with different defines for esm and conventional builds.

Follow-up: it turns out that running JS code through a C preprocessor, though ostensibly unproblematic, has quirks which make it untenable. That annoyance prompted me to dive down a rabbit hole to develop a suitable preprocessor which we can potentially use in the sqlite3 JS build process to create separate post-processed source files with ESM-compatible constructs, e.g. using the import keyword where appropriate. It's not yet been added to the JS build process, but we now have a suitable tool for the job should that route prove interesting or necessary.

The documentation is confusing. This sentence seems to imply that there's always a "." at the start of the include path list:

Regardless of the include path, a filename which matches a file without any path expansion is considered a better match.

But that clashes with the previous sentence:

The include search path is defined by passing one or more -Idirname flags to the app and if no such flags are provided, -I. is assumed.

To solve the error, set the type attribute to module when loading the script in your HTML code. When working with ECMAScript modules and JavaScript module import statements in the browser, you'll need to explicitly tell the browser that a script is module. To do this, you have to add type="module" onto any ‹script› tags that point to a JavaScript module. Once you do this you can import that module without issues.

<script type="module" src="./index.js"></script>

If you are working on Node.js or react applications and using import statements instead of require to load the modules, then ensure your package.json has a property "type": "module" as shown below.

{ // ... "type": "module", // ... }

Moreover, In some cases, you may have to use both import and require statements to load the module properly.

// import { parse } from 'node-html-parser'; parse = require('node-html-parser');

This error "Cannot use import statement outside a module " can happen in different cases depending on whether you're working with JavaScript on the server-side with Node.js , or client-side in the browser. There are several reasons behind this error, and the solution depends on how you call the module or script tag.

Any chance you all would either: (a) update the wasm artifacts to be esm compliant and make use of import.meta.url

If you're able to do a full build of the wasm parts, please try this from a checkout of the current sqlite3 trunk:

$ ./configure --enable-all
$ make sqlite3.c shell.c
$ cd ext/wasm
$ make esm

the difference from the current normal build is that it produces ext/wasm/jswasm/sqlite3.mjs instead of sqlite3.js, and Emscripten then makes use of import.meta for figuring out where to load the wasm file from. This does not change any of our own code, so it might not result in a working esm, but (A) that might not matter and (B) it might at least get us a step closer.

The makefile does not currently build both sqlite3.js and sqlite3.mjs because they're separate builds from emscripten's point of view but would output sqlite3.wasm for both, so the builds would hammer each other when compiling in parallel. How we might best work around that is not yet clear, but we want to avoid shipping two separate wasm files if we can at all avoid it.

See also:

• http://philiplassen.com/2021/08/11/node-es6-emscripten.html
• https://github.com/emscripten-core/emscripten/issues/14383

Awesome! Thanks so much. I'll pull these down now and try them out.

Prior to seeing your latest message(s) I had gotten the required changes down to a minimal set which you can see as a diff against the sqlite master branch here: https://github.com/sqlite/sqlite/commit/3c5b9f28a666a31170d01c4d61e6decc25f2a67d

if that's useful to you.

Obviously in your case you'd want some checks (e.g., typeof import !== 'undefined') in that code to make it work in non ES6 environments.

Your current approach to sed'ing the script is not maintainable

Yep, definitely not.

Obviously in your case you'd want some checks (e.g., typeof import !== 'undefined') in that code to make it work in non ES6 environments.

It turns out that that approach is outright illegal. We're not allowed to use that symbol name at all in non-modules, even to check for its existence :(. That seems somewhat tyrannical but that's what Chrome is telling me.

The same is true for export. I'm also interested in ES6 support and have added a few comments to the GitHub diff.

For what it's worth, I've managed to get this working with deno (and still be browser compatible) with the following one-liner to extern-post-js.js:

-  const originalInit = self.sqlite3InitModule;
+  const originalInit = sqlite3InitModule;

This removes the explicit reference to the global self object and:

• In the case of an ES6 module, refers to the sqlite3InitModule declared in module scope.
• In the case of a browser script, refers to the sqlite3InitModule on the global object.

This removes the explicit reference to the global self object and:

Which, IMO, is poor style :/, but...

In the case of an ES6 module, refers to the sqlite3InitModule declared in module scope. In the case of a browser script, refers to the sqlite3InitModule on the global object.

but i'm slowly coming to grasp why that change is necessary for ESM support. i'll patch that in the trunk copy. i'm not happy about it, because i hate referencing global-scope symbols implicitly, but... so be it.

The WASM modules for SQLite aren't the easiest to import and use. Mainly because they don't conform to ESM and secondarily because they can't find the wasm and opfs-proxy locations on their own.

That is very possibly remedied now. The trunk version now builds sqlite3.mjs along with sqlite3.js, where ".mjs" is apparently the preferred extension for ES6 modules:

https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Modules#aside_%E2%80%94_.mjs_versus_.js

(Edit: funnily enough, that link works in Firefox but not Chrome. Search that page for ".mjs" to find the relevant citation.)

That file can be imported as a module with something like:

    <script type="module">
      import {default as sqlite3InitModule} from "./jswasm/sqlite3.mjs";
      sqlite3InitModule().then((sqlite3)=>{
        console.log("Loaded sqlite3",sqlite3.version);
      });
    </script>

That build is now plugged in to our core testing process so will be tested along with the main build. What we don't yet test is running this build from a Worker because at least one major browser (Firefox) cannot load ES6 modules from workers:

https://bugzilla.mozilla.org/show_bug.cgi?id=1247687

Edit: we now have test apps which load sqlite3 as an ES6 module both in the main thread and worker module (for browsers which support that (i.e. not Firefox)).

A snapshot of this build is at:

https://wasm-testing.sqlite.org/tmp/sqlite-wasm-3410000-snapshot.zip (Updated since the initial post to add ES6 worker module test app)

but we make no guarantees whatsoever about how long it will remain there before being deleted.

As this is our very first concentrated attempt at creating an ES6 module, any feedback on whether this works in your real-world applications would be greatly appreciated.

Thanks very much! I can confirm that this works perfectly for Deno.

Given that it uses the location API (self.location.href) it may not work for NodeJS. However, that's only my speculation until a NodeJS user can test it.

Given that it uses the location API (self.location.href) it may not work for NodeJS. However, that's only my speculation until a NodeJS user can test it.

The JS code is shamelessly browser-centric in multiple places. We're not currently focused on any non-browser environments so compatibility with those is entirely dependent on feedback from folks who use those.

Over the next six months, we can expect to see the WASM file (as distinct from the JS) become more portable to server-side platforms (it currently requires Emscripten for a working library), from which people can bind whatever WASM-capable environment they have to it (python, node, whatever). That's a distinctly different goal than all-around-portable JS, however. The latter is currently a non-goal.