The next attempt to use Sqlite WASM with Next.js was with sqlite3.js file.

const dbWorker = new Worker( new URL("../lib/dbWorker.ts?sqlite3.dir=path/to", import.meta.url) );

The problem is the URL query is translated to http://localhost:3000/_next/static/chunks/lib_dbWorker_ts_sqlite3_dir_path_to.js

, so obviously, sqlite3.js can't parse it.

I tried to use sqlite3.mjs with Next.js, and there was an error related to duplicate export default.

The duplicate export is an Emscripten artifact/bug which we work around in the build process. Are you using a custom build process?

and there was another exports error, so I deleted this:

That part is a necessary workaround for Emscripten issues #18071 and #18237, which combine to force us to manually re-export the module symbols. So far nobody has reported any unspecified exports errors related to that part, nor have we encountered them in testing.

For some reason, there was a URL with file:// even when I ran the app with a server.

That's not coming from us. Nowhere do we define any URL schemes in JS code - we always inherit them from the environment.

If next.js, which i had never heard of until your post, is run server-side then it is not suitable for use with our JS code. We currently target only browser-side use and have no immediate plans for supporting server-side JS. Until we do, the WASM file itself provides all one needs to create their own server-side SQLite wrapper. We do not aim to provide The One True JS API and do hope to see others create alternatives, in particular for environments we do not use ourselves.

On a side note, I understand you don't want to depend on TypeScript (I'm not here to debate whether it's a good decision or not), but TypeScript can be used also with plain JS files (to validate and even type them). It could prevent invalid source code, at least.

i'm not clear what that has to do with us? None of the code we've published is, to the very best of my knowledge, "invalid." It all loads and runs in our test and development environments, with the caveat that our environments encompass only browsers running through conventional web services.

And no, TypeScript is not an option for us. We aim to keep the JS code hand-written (not transpiled), insofar as technically possible¹ and completely free of voodoo such as...

On another side note, Sqlite WASM could be ideally placed in NPM, and if so, I suppose ESM only is the best approach. Modern bundlers workaround Firefox and Firefox ESM in WebWorker was recently fixed anyway.

To repeat what i've said on that topic several times in HackerNews threads:

Trying to support a tool one neither uses nor understands is like trying to wave away darkness with one's hand². Nobody in the sqlite project uses node.js or npm in any capacity except for "under the hood" in the Emscripten build process, nor do we have any ambitions to do so. Though NPM support has not been outright ruled out for the future, it is currently nowhere on our radar.

Folks who actively use NPM are far better suited to provide NPM-compatible components and support. We're happy to work with them to bend the build process to accommodate that, but we're not currently interested in taking on the maintenance burden of supporting a platform and tools we don't otherwise use.

const dbWorker = new Worker( new URL("../lib/dbWorker.ts?sqlite3.dir=path/to", import.meta.url) ); The problem is the URL query is translated to ...lib_dbWorker_ts_sqlite3_dir_path_to.js ... so obviously, sqlite3.js can't parse it.

That's clearly a case of a 3rd-party tool trying to be far too clever and outright mangling the URL. There's nothing we can do to protect ourselves from such mistranslation but if you can suggest a workaround for it, i'm willing to implement it so long as i doesn't unduly affect other environments.

1. ^{^} Noting that we currently rely on certain Emscripten-driven transformations, but also actively work to minimize those.
2. ^{^} With my apologies to Alan W. Watts for bending his analogy.

This is what I am using. I copy pasted it from docs:

git clone https://github.com/sqlite/sqlite.git git clone https://github.com/emscripten-core/emsdk.git

cd sqlite ./configure --enable-all make sqlite3.c

cd ../emsdk ./emsdk install latest ./emsdk activate latest source ./emsdk_env.sh

cd ../sqlite/ext/wasm make release

Next.js is how most developers will (try to) use the official build, so it makes sense to know it. No, I don't use it on the server side.

As for TS, you did not read what I wrote. I did not write that you should transpile code from TypeScript to JavaScript. I wrote TypeScript can help you even with plain, manually written JS. Prettier is the same, but if you insist on developer asceticism, that's fine. I only wrote that because you are new in the JS ecosystem; hence, you are unaware of existing tools. That's all. Do whatever you want.

Next.js is how most developers will (try to) use the official build, so it makes sense to know it.

Within the sqlite project we use neither node.js, next.js, npm, react, nor git¹, just to name a few, despite whatever the trends outside the project are. In terms of our volunteer development time, it makes sense to focus on what we're skilled at and leave the use of other tools to those who are skilled with them.

As for TS, you did not read what I wrote. I did not write that you should transpile code from TypeScript to JavaScript. I wrote TypeScript can help you even with plain, manually written JS.

TypeScript is yet another high-level tool we would like to avoid. As fitting for the traditions and conventions of all of Richard's projects, we prefer to write everything from the bottom up, without relying on tools we neither require nor have any control over.

I only wrote that because you are new in the JS ecosystem;

On the contrary, i've been writing JS since the 20th century, and actively developed C++ bindings for the SpiderApe and V8 JS engine for 7 or 8 years, but purposefully (perhaps stubbornly) choose to eschew most of the JS ecosystem trends, in particular those involving node and npm. "Asceticism" is indeed an appropriate description.

1. ^{^} The sqlite git repo is a read-only mirror of the canonical source tree.

... my build wasn't complete for some reason. .mjs I downloaded looks fine.

Great :-D. Thank you for reporting that.

Anyway, it still doesn't work in Chrome for some reason.

sqlite3.mjs?3c54:34 Uncaught ReferenceError: exports is not defined at eval (sqlite3.mjs?3c54:34:1)

When I commend this code, Chrome is OK.

// if (typeof exports === "object" && typeof module === "object") { // module.exports = sqlite3InitModule; // } else if (typeof exports === "object") { // exports["sqlite3InitModule"] = sqlite3InitModule; // }

But .wasm isn't still loaded.

sqlite3.mjs?3c54:70 Not allowed to load local resource: file:///Users/danielsteigerwald/dev/test/lib/jswasm/sqlite3.wasm

Investigating what Webpack does with that.

else if (typeof exports === "object")

Can try replacing that part with:

else if (typeof exports === "object" && !!exports)

:-?

One highly unfortunate quirk of JS is that (typeof null === "object"). That rarely comes into play because null isn't often used outside of JSON, but it's possible that exports is indeed null in that environment.

Edit: nevermind: the second copy of that is from us, so i can fix that one, but the first one gets injected by Emscripten. i'll patch the 2nd one, in any case, but it's largely moot until/unless Emscripten extends that check.

Edit: the error message indicates that exports has the undefined value. Both FF and Chrome include "null" in the error when trying to perform such an op on a null value.

i overlooked this bit earlier:

sqlite3.mjs?3c54:70 Not allowed to load local resource: file:///Users/danielsteigerwald/dev/test/lib/jswasm/sqlite3.wasm

No current browsers will load WASM files from file:// URLs (and i should have picked up on that earlier but that detail escaped me). They have to be served from an HTTP service. That's a security limitation/feature/quirk we have no influence on.

It's a little bit hard to report bugs here because you are constantly misreading what I wrote. You are answering things I didn't write and telling me something I know and have no problem with. The problem is simple: The JS code you carefully manually write does not work with tools > 90 % of developers are using - bundlers. Not .js nor .mjs is working with Next.js - probably the most versatile JavaScript framework. Take a look at wa-sqlite, for example. It just works.

Update: I will share a workaround for other developers to help them avoid this waterboarding-like DX.

Update: I think I know where the problem is. Bundlers need statically analyzable code. Look:

// if (Module["locateFile"]) {
//   wasmBinaryFile = "sqlite3.wasm";
//   if (!isDataURI(wasmBinaryFile)) {
//     wasmBinaryFile = locateFile(wasmBinaryFile);
//   }
// } else {
wasmBinaryFile = new URL("sqlite3.wasm", import.meta.url).href;
//   // console.log(wasmBinaryFile);
// }

Module["locateFile"] = function (path, prefix) {
  return wasmBinaryFile;
}.bind(sqlite3InitModuleState);

const W = new Worker(new URL("sqlite3-opfs-async-proxy.js", import.meta.url));

Take a look at: new URL(STRING_LITERAL_MUST_BE_HERE)

Then it works.

Update: This is a quick fix for .mjs. I don't know how to fix .js and whether it's even fixable, but I don't care because CommonJS is dying, and ESM is the future.

Btw, I rigorously followed "Building JS/WASM Bundles" docs, and I got a slightly different file than in https://sqlite.org/wasm/uv/snapshot.html which made additional confusion. Docs are probably not up to date.

Update: I think I know where the problem is. Bundlers need statically analyzable code. Look:

// if (Module["locateFile"]) {
//   wasmBinaryFile = "sqlite3.wasm";
//   if (!isDataURI(wasmBinaryFile)) {
//     wasmBinaryFile = locateFile(wasmBinaryFile);
//   }
// } else {
wasmBinaryFile = new URL("sqlite3.wasm", import.meta.url).href;
//   // console.log(wasmBinaryFile);
// }
Module["locateFile"] = function (path, prefix) {
  return wasmBinaryFile;
}.bind(sqlite3InitModuleState);

It's not clear to me what you're reporting as the problem there. The commented-out block is Emscripten-generated/injected code which we have no influence over. The second one closely resembles our override of that function:

Module['locateFile'] = function(path, prefix) {
  return new URL(path, import.meta.url).href;
}.bind(sqlite3InitModuleState);

We cannot sensibly reference wasmBinaryFile from that override because that name is an undocumented Emscripten-internal detail which we cannot rely upon and can change from any given version of Emscripten to the next. That override is in the same resulting amalgamated sqlite3.(m)js, compiled by Emscripten, but is not part of the Emscripten-created code.

const W = new Worker(new URL("sqlite3-opfs-async-proxy.js", import.meta.url));

Take a look at: new URL(STRING_LITERAL_MUST_BE_HERE)

If you're implying that the URL constructor requires a string literal then that's an unfortunate case of a flawed downstream tool. That particular path, in the canonical copy of that code, looks like:

const W = new Worker(new URL(options.proxyUri, import.meta.url));

It's necessarily a symbolic reference because that value can change depending on how the that file (or a functional equivalent) is loaded.

Btw, I rigorously followed "Building JS/WASM Bundles" docs, and I got a slightly different file than in https://sqlite.org/wasm/uv/snapshot.html which made additional confusion. Docs are probably not up to date.

They can differ based on the EMSDK version used for the build, as emcc generates a large chunk of the resulting sqlite3.mjs file and emcc's output often differs from one version to the next. The EMSDK used for the snapshot builds tends to be "the latest," or within a few weeks of it. A mention of that discrepancy will be added to our doc and the word "bundle" will be removed to eliminate any unfortunate association with "bundlers" in the JS ecosystem sense of the term (none of which are covered by those docs).

It's not clear to me what you're reporting as the problem there.

Because the code you write does not work while every other SQLite WASM distribution does. I even explained why your code is not working, and the result was yet another FUD.

It's a little bit hard to report bugs here because you are constantly misreading what I wrote. You are answering things I didn't write and telling me something I know and have no problem with.

After having looked back over my responses: it's apparent that i misinterpreted your intent regarding TypeScript, but i've otherwise responded only to what you've posted as problems. If a pasted-in error message were not intended to be understood as problem reports then i'm definitely guilty of having misinterpreted them and am admittedly confused about their purpose, if not as problem reports.

The JS code you carefully manually write does not work with tools > 90 % of developers are using - bundlers

Given that you're the first to file any complaints about bundlers, and also the first to bring up next.js, in the 8 months since development started, the implication is that 90% of people having problems with it is an over-estimation. (That's not to say that bundler-related issues are unexpected - it was only a matter of time before it came up.)

The diverse and ever-changing tools JS developers use for their projects are out of scope for us, plain and simple. We are a tiny team without the developer bandwidth to proactively support arbitrary tools which we do not otherwise use. We provide a lowest common denominator rather than try to shoe-horn downstream users into a subset of tools or methodologies which we have pre-selected for them. There is no lower common denominator than hand-written, "vanilla" JS.

That's not going to change, but...

That there are issues integrating it into specific frameworks is a given. Every toolchain works differently and has its array of quirks. Discovering and resolving incompatibilities takes time and effort which we do not/cannot specifically devote proactively. We can resolve such issues only in cooperation with people who use the conflicting tools, such as yourself. We're more than happy to help folks get this integrated into their toolchains/environments, to the extent of providing alternative builds of sqlite.(m)js for specific environments, as we recently did for ES6 modules, but we require those folks' assistance to do so.

The diverse and ever-changing tools

Every toolchain works differently and has its array of quirks.

Sorry, but what you wrote is just malicious FUD. String literals are required in ESM as well. It's not "a flaw". It's a ubiquitous design decision. It's almost impossible to parse dynamic code statically; hence string literal is required. Other Sqlite WASM distributions don't have this problem. I don't think constant blaming of the ecosystem is productive.

String literals are required in ESM as well.

They demonstrably are not. We run ESM module tests for every build and every one of our URL constructor calls uses a symbolic reference as its first argument.

It's almost impossible to parse dynamic code statically;

Static analysis is not a requirement of ours or the platforms we run have run it on to date. It's a requirement of a piece of your toolchain.

We are technically capable of producing alternate, non-configurable builds which hard-code strings (with the exception of Emscripten-generated parts), but we require guidance and cooperation from people who actively use the tools which impose those requirements to do so. We cannot magically divine what those requirements are, nor are we willing to invest the energy to become experts in arbitrary JS tools to figure them out ourselves.

Other Sqlite WASM distributions don't have this problem.

We can offer two options:

1. Cooperate (politely) with us to create a build which works in your environment, insofar as we can (with the very real caveat that we have no influence whatsoever over the Emscripten-generated code).

2. Select a different SQLite WASM distribution, as folks have been doing since more than 10 years now. It is not our aim to be The One True Solution to the problem of hosting sqlite browser-side and a mono-culture is certainly not something we want to create.

"The static import declaration..."

"It's called static import, because it's static. What do we mean by "static"? When you use an import declaration, it must be at the top level of the module, outside of any control-flow statements, and it must use a string literal to say what module to import from, not just a string. That's so the relationship between modules can be determined by static analysis (by just parsing, not running, the code)."

https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/import

Bundlers need to analyze the code statically for the same reason.

It's a requirement of a piece of your toolchain.

Not mine. Bundling is a requirement for any serious work with JavaScript in the 21st century. No sane developer releases JavaScript source code as is.

We can offer two options:

I am cooperating as much as I can :-) I figured out where is the problem. I recommended limiting your focus to ESM build only because I don't think CJS makes sense anymore (and the workaround does not work with bundlers anyway).

Everyone is using bundlers nowadays, which is, BTW, yet another reason why you don't have to care about CJS.

Let's make ESM build compatible with bundlers (string literal is the only requirement I am aware of), and many developers (me included) will be happy.

I am cooperating as much as I can :-)

Your tone has been decidedly combative so far (e.g. see the comment about waterboarding). i'm happy to help you get this running in your environment so long as the tone remains civil/non-inflammatory. (And in case it's not clear: i'm currently the only "JS Guy" on the team, so it's me or nobody! ;)

With that behind us, let's get to solving this...

I recommended limiting your focus to ESM build only because I don't think CJS makes sense anymore (and the workaround does not work with bundlers anyway).

The CJS/AMD bits (none of which we make any use of) are injected by Emscripten. However, looking closely at the Emscripten-generated code now, they elide those parts for the mjs build which means we can elide our counterpart/workaround in that build.

"The static import declaration..."

For the record: we don't use static imports anywhere except demo apps, nor does the Emscripten-generated code use them. We load workers with URL objects with configurable/dynamic strings, which is a distinctly different use case.

We can create a build which uses static strings for at least some of those cases, but such builds will be less portable/configurable (which "probably" isn't a big deal in practice).

i will prototype a bundler-friendlier build for that on Friday, but will require some back and forth with you to find out whether it works as desired. We can do that either off-list (stephan at this domain) or here, as you prefer (fellow forum-goers presumably prefer that we do it off-list).

For my own clarity: bundlers are only concerned with ESM builds, correct? Performing this surgery for a non-ESM build will require yet another separate build.

Not mine. Bundling is a requirement for any serious work with JavaScript in the 21st century. No sane developer releases JavaScript source code as is.

It's your problem in that none of our use cases to date require a bundler, so it's definitely not our problem ;). It's somebody's problem, in any case ;).

Everyone is using bundlers nowadays

Not everyone. i used a bundler on a single project in 2016 and haven't needed a bundler or npm since.

... which is, BTW, yet another reason why you don't have to care about CJS.

We're actually not concerned with AMD/CJS - those bits get added by Emscripten because it's trying to be portable, and we have to add one workaround in our code in response to that. That workaround can, however, be removed from MJS builds, eliminating one of the failures you reported.

The current plan is:

• Update the main MJS build to remove the AMD/CJS bits entirely.
• Create a bundler-friendly ESM build which replaces, where possible, dynamic strings in URL constructors with static ones. This build will tentatively be called sqlite3-bundler-friendly.mjs, but suggestions for names are welcomed. The API itself does not change, with the tiny exception that certain configurable parts, like the WASM file load path, will become non-configurable.

What's missing from that list?

Update the main MJS build to remove the AMD/CJS bits entirely.

Done in the js-bundler-friendly branch.

Create a bundler-friendly ESM build which replaces, where possible, dynamic strings in URL constructors with static ones.

There are several i'll need your feedback on...

The OPFS filename parser passes client-provided strings to the URL constructor. That same function is duplicated in the "async half" of that VFS (a dedicated worker).

In a few places i've removed usage of URL() altogether from this build where it's just used to get at URL parameters, the implication being that various config options which can be passed on via URL flags simply won't work for that build (but also "shouldn't" be necessary in this build).

Two of those required making a wild guess on how a bundler wants to see them, specifically the bottom two diff chunks in src:/info/b7b896fb448a7f46.

The current prerelease snapshot contains a jswasm/sqlite3-bundler-friendly.mjs with these changes. Alternately, if you have a checkout of the sqlite tree, these changes are currently in the js-bundler-friendly branch. It is completely untested with a JS bundler because i've no clue how to set one up and it's 4:30 in the morning CET so am not going down that rabbit hole :).

If you can look those over and point out any potential bundling problem spots, we'll be one step closer to having a bundler-friendly build.

I just downloaded the prerelease snapshot, but I don't see that file.

I can make my build as I did previously, but some step was missing because the generated code was different. Is this complete?

git clone https://github.com/sqlite/sqlite.git
git clone https://github.com/emscripten-core/emsdk.git

cd sqlite
./configure --enable-all
make sqlite3.c

cd ../emsdk
./emsdk install latest
./emsdk activate latest
source ./emsdk_env.sh

cd ../sqlite/ext/wasm
make release

I just downloaded the prerelease snapshot, but I don't see that file.

My fault - i was working in multiple branches here and a snapshot was built later on from a branch which doesn't have that file. A new one, from this branch, was just pushed and contains the missing file:

[stephan@nuc:~/f/s/wasm]$ unzip -l sqlite-wasm-snapshot-20230127-3410000.zip
Archive:  sqlite-wasm-snapshot-20230127-3410000.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
...
   366173  2023-01-27 16:24   sqlite-wasm-snapshot-20230127-3410000/jswasm/sqlite3-bundler-friendly.mjs

Achtung: it was discovered today that Emscripten 3.1.31 creates a broken build. 3.1.30 works fine. (This actually exemplifies why we prefer to write all the code ourselves instead of relying on voodoo-level JS tools we can't fix. ;))

To ensure that you have 3.1.30, replace:

./emsdk install latest

with:

./emsdk install 3.1.30

That can be done in an existing checkout - no need to re-clone. It will take a while to download and unpack the other version, though. If you're using the snapshot build, it's irrelevant, in any case.

git clone https://github.com/sqlite/sqlite.git

We don't actually use that repository: it's a read-only mirror. However, it "should" be recent enough. It's updated every hour, IIRC.

make release

Try, for speed's sake, just "make". The result will be a much larger WASM file but it will build in half the time.

The instructions otherwise look okay.

i need to leave for a while, then sleep, then i'll see about getting this beast set up locally.

This actually exemplifies why we prefer to write all the code ourselves instead of relying on voodoo-level JS tools we can't fix.

Everybody makes mistakes. To Err is Human. The remedy is versioning, not NIH syndrome ;-)

Try, for speed's sake, just "make". The result will be a much larger WASM file but it will build in half the time.

Thank you. Fortunately, I have an M1 chip so it's faaast.

then i'll see about getting this beast set up locally.

Feel free to contact me anytime at https://twitter.com/steida or 420775326783 Signal.

I just replaced my patched file with sqlite3-bundler-friendly.mjs and it looks like it works. Thank you for your help and cooperation.

I suppose you can calmly delete the non-ESM version with that URL query workaround because it will not work with bundlers anyway. As far as I know, they will always rewrite the web worker URL. The fewer hacks and confusion, the better. ESM is OK. Now, I can finally test concurrency and API 😂

If I may, I would recommend using JSDOC.

https://www.typescriptlang.org/docs/handbook/jsdoc-supported-types.html

People using VSCode etc. will see the documentation in IDE. Even auto-complete is possible. And web pages can be generated from source code. It's useful, and there is no commitment from your side.

As for API. May I ask why the exec method uses a callback? It's sync so it can return rows instead.

As for API. May I ask why the exec method uses a callback? It's sync so it can return rows instead.

That subset of the SQLite library API has existed for a long time in the C versions. It would be strange and generally unexpected for it to be chopped out of the JS version(s). It would be similarly odd to change its behavior.

The API is what is known colloquially as a "convenience method". The prepare() and step() API effectively unwraps exec(), making rows available per step().

I just replaced my patched file with sqlite3-bundler-friendly.mjs and it looks like it works.

Holy cow, seriously? i was expecting at least another 2 or 3 iterations. Even so, i'll take the win!

i'll get that merged into trunk and added to the docs after some sleep.

I suppose you can calmly delete the non-ESM version with that URL query workaround

Not a chance. All of my apps, and no small number of test and demo apps, use that version. ES6 and bundler support will remain second-tier targets, in terms of development focus, for the foreseeable future. The former for portability reasons (Firefox only recently got the ability to load modules from Workers) and the latter because they require otherwise unnecessary tooling.

The fewer hacks and confusion, the better.

On the contrary, bundlers are the hack, requiring the addition of tooling which is otherwise entirely unnecessary. But we'll just have to agree to continue to disagree on that point, and that's okay. As someone wiser than myself once said, "the world would be a mighty strange place if we were all the same."

If I may, I would recommend using JSDOC.

That's been a topic of internal discussion, up to and including (surprise!) extending the project's own doc tools to be able to extract docs from JS. For the time being, though, the primary docs are currently the markdown-format ones. Depending on the JS build, the docs might or might not be stripped from those because they account for more half of the JS:

[stephan@nuc:~/f/s/lite/ext/wasm/jswasm]$ wc -c sqlite3-api.js
437662 sqlite3-api.js
[stephan@nuc:~/f/s/lite/ext/wasm/jswasm]$ ../../../tool/stripccomments < sqlite3-api.js | wc -c
196808

Without JS-level build structure, i.e. node.js, we can't usefully extract jsdocs, anyway, and it will be a very cold day in that otherwise very hot place before we add such infrastructure. It's just not our thing.

It's useful, and there is no commitment from your side.

i don't discount that it's useful to some (likely even many) folks, but it requires yet more node.js tooling on our side, and i don't mind admitting that my personal allergy to All Things Node.js just happens to align strongly with this project's tendency to avoid external tooling except when functionally necessary (as opposed to "nice to have," for which cases homegrown solutions are always preferred).

That said: i'm not poo-poo'ing the value of jsdocs as a convention, and all of us in this project have a deep love of software docs, but jsdocs is not a great fit for this particular project because of the tooling needed to support it. However, our docs will certainly evolve over time, and heading in the direction of jsdocs has not been entirely ruled out.

Related trivia: if we count at 60 lines per printed page, the JS/WASM docs now amount to...

[stephan@nuc:~/f/s/wasm]$ echo $(($(cat *.md | wc -l) / 60)) pages
130 pages

As for API. May I ask why the exec method uses a callback? It's sync so it can return rows instead.

Keep in mind that a result set can be arbitrarily large, in particular with the rise of 256MB dbs via OPFS. As this project is populated entirely by C programmers, we focus on efficiency more than is conventional in the JS community. (i'm not claiming that we always should, but it's an age-old habit which is difficult to break and, in fact, accounts for much of the fun of writing C.)

Call chaining was also an important aspect for me:

db.exec(...).exec(...).exec(...).close();

In any case, DB.exec() can actually be told to return various things, one of which is the complete set of result rows as either an array of arrays, an array of objects, or an array of single values, to avoid having to use a callback for small cases. Something like:

const rows = db.exec({
  sql: ...,
  rowMode: 'object',
  returnValue: 'resultRows'
});

Now that you mention it, that might be a sensible default if it's passed no callback and the SQL contains a statement which might produce result rows. i'll look into that.

Holy cow, seriously?

I didn't check yet what you changed, but my patch was only related to string literal and removing that CJS/AMD stuff.

All of my apps, and no small number of test and demo apps, use that version.

You can use ESM in all browsers. You don't need a bundler. It's finally working everywhere (except for some old bundlers, but who cares).

Firefox only recently got the ability to load modules from Workers.

Irrelevant because Firefox doesn't support OPFS anyway. When it will, Web Worker ESM will be there as well :-)

On the contrary, bundlers are the hack

Let me cite someone too: “There are no solutions, there are only trade-offs; and you try to get the best trade-off you can get, that's all you can hope for.”

...because they account for more half of the JS:

A good practice is releasing source code without minification etc. because it's handy in VSCode. You can click, and voila, you see the source code. Minification etc. belongs to the app itself (bundlers).

but it requires yet more node.js tooling on our side,

Why do you think you need Node.js? Just write JS comments in JSDOC format, that's all. Even without extraction, it's useful for people because VSCode and other IDEs and editors use it. It's just a format of comments, nothing more. Of course, it could be helpful also for you because of TS type checking, etc., but you don't have to use it. Just write code comments in JSDOC format.

we focus on efficiency more than is conventional

So callback is a chunk API, thank you for the explanation. Btw, that callback could be better named onRow, so people will not have to think about what it is for and why it isn't called for no rows.

Call chaining was

Isn't call chaining just syntax sugar? Does it maintain a state between calls? Anyway, it's a very minor issue. I was just curious why such API.

I didn't check yet what you changed, but my patch was only related to string literal and removing that CJS/AMD stuff.

For future maintenance reference: what is the rule of thumb for avoiding dynamic strings in the URL() constructor? Why it is not a problem to pass a dynamic string in some contexts, e.g. the OPFS internals?

You can use ESM in all browsers.

When additional portability has no onerous cost to it, i will invariably choose wider portability over The Latest Thing. Chalk it up to a cultural difference between "the C ethos" and "the JS ethos". Those two cultures have a number of inherent, insurmountable conflicts (the former's tendency to prefer manual dependency management over DaaS (Dependencies as a Service) being another one of them).

Irrelevant because Firefox doesn't support OPFS anyway.

OPFS is a single feature of the larger API. It was the catalyst for this whole effort, but is not "the" feature. With the addition of support for localStorage and sessionStorage, OPFS actually has competition (however silly those backends might sound, they work and are in every JS-capable browser).

FF is this API's main dev platform, in any case. Chrome/ium is used primarily for testing OPFS support.

A good practice is releasing source code without minification ...

That's actually my preference, but we have self-imposed size limits which, ideally, should be kept.

Why do you think you need Node.js? Just write JS comments in JSDOC format, that's all. Even without extraction, it's useful for people because VSCode and other IDEs and editors use it.

You still need the tooling to verify that the doc syntax is valid, in particular for someone who's not used to jsdoc. Even seasoned doxygen users will, on occasion, mess up the syntax without realizing it until the docs are processed.

If someone familiar with jsdoc were to come in, refactor all the docs for it, and walk out, i'd be happy to pick up where they left off, but that initial overhaul is a larger undertaking than i can currently commit to. Long-term, it's not ruled out, though. Writing software docs is actually a hobby of mine.

Btw, that callback could be better named onRow, so people will not have to think about what it is for and why it isn't called for no rows.

Fair point but that ship has sailed. The exec() API is probably the single most heavily-used piece in sqlite3.js, so changing that now would be unduly painful. (But see below!)

Isn't call chaining just syntax sugar?

In short, it's based on prior art. This particular DB class is something like the 8th or 9th i've written for binding sqlite to a scripting language¹ and having exec() return "this" by default has, by and large, long since proven to be useful in prior bindings, so that's what we have now.

That said: the sqlite3.oo1 API is intended to be one solution to help users get started, but is not intended to be The Solution. It is my hope that people will create alternative APIs which suit their cases better and that we can improve our API through osmosis of features introduced in other ones. The name "oo1" was chosen specifically to convey the notion that "there might eventually be an oo2 some day with a different API, and perhaps even an oo3..."

1. ^{^} To the very best of my fallible knowledge, my 2006 binding of sqlite to the SpiderMonkey JS engine, now long-since defunct, was first-ever published JS binding of sqlite. At the time someone else claimed to be working on one but never released any code. (That 8-space code indentation is painful to look at now.)

For future maintenance reference:

Only paths to existing files must be string literals. A bundler does it only because it merges files possibly placed in different directories. OPFS internals is not files processed by a bundler.

Thank you for your changes!

Thank you for your changes!

You are very welcomed and thank you for the feedback which led to them. Before merging i rolled back the changes to URL constructors which do not refer to in-tree files and will be mindful of using static strings for any future in-tree URLs we may need in the future.

Edit: i just now realized that these changes broke the worker1/promiser variants. i will need to roll back the changes to those particular parts and provide separate bundler-friendly variants of those.

Btw, setting Next.js is a piece of cake.

Install Node.js

npx create-next-app@latest

Replace pages/index with

import { useEffect } from "react";

export default function Home() {
  // This runs code only in the browser.
  useEffect(() => {
    const dbWorker = new Worker(new URL("../lib/dbWorker", import.meta.url));
  }, []);
  return null;
}

// lib/dbWorker.js, 
import sqlite3 from "./sqlite3.mjs";

sqlite3().then((sqlite3: any) => {
  const capi = sqlite3.capi /*C-style API*/,
    oo = sqlite3.oo1; /*high-level OO API*/
  // console.log(
  //   "sqlite3 version",
  //   capi.sqlite3_libversion(),
  //   capi.sqlite3_sourceid()
  // );
});

Then npm run dev, and that's all.

Do you have a link to a fuller example that shows some loading / querying of data from the instance? If not that's OK... just thought it might save some time. Thanks

This is where and how I use SQLite WASM: evoluhq/evolu

When I follow the directions above (see here), I see this in the console and I see a blank screen.

Any ideas?

Uncaught Error: Expecting self.sqlite3InitModule to be defined by the Emscripten build.
    at eval (sqlite3.js?f311:10052:11)
    at eval (sqlite3.js?f311:10119:1)
    at ./lib/sqlite3.js (lib_dbWorker_js.js:41:1)
    at options.factory (lib_dbWorker_js.js:608:31)
    at __webpack_require__ (lib_dbWorker_js.js:81:33)
    at fn (lib_dbWorker_js.js:263:21)
    at eval (dbWorker.js:2:69)
    at ./lib/dbWorker.js (lib_dbWorker_js.js:30:1)
    at options.factory (lib_dbWorker_js.js:608:31)
    at __webpack_require__ (lib_dbWorker_js.js:81:33)

I think it would be of great value to extend these instructions to include actual use of the database by the app, say, insert some rows and query them.

Thanks for any insights you can give me!

Any ideas? ... Uncaught Error: Expecting self.sqlite3InitModule to be defined by the Emscripten build.

That looks very much like you're loading a custom build which was either not assembled by Emscripten or was built with a custom module name.

That error is "impossible" to trigger from the canonical builds and is there for the sake of folks who create custom builds. The sources themselves are set up to be able to be built in other contexts, but doing so leaves the user "on their own." i'm happy to help resolve it, but need more context than just an error message. Ideally a link to a source tree where this is reproducible.

I think it would be of great value to extend these instructions to include actual use of the database by the app, say, insert some rows and query them.

We have numerous demonstration apps for that, all available in the wasm distribution's zip file. Prerelease snapshots are available here and the 3.41 (likely within the next week) will include the latest wasm/js code as well. "The problem" with including example snippets in the docs is that developers tend to copy/paste them without understanding the larger context around them and then come asking why they don't work. The demo apps provide that missing context needed for understanding the whole constellation.

Sorry, I didn't intend for you to respond to this request. The OP had indicated that it was "simple" to get things to work using the Next.js framework and I know that you cannot help there.

So far I have not found any examples of this working in the "open" world using any modern frameworks. I have hopes for the work being done here: https://github.com/overtone-app/sqlite-wasm-esm.

My own attempt to replicate the OP's example is here: https://github.com/mandolyte/sqlitenext

I'm sure such examples, tutorials, whatever will come in time. I'll keep trying to figure it out on my own and asking questions of those who have made more progress than I have.

Thanks for your efforts to make this work with WASM. I would love to show our team how our apps could make use of it.