SQLite Forum

SQlite-WASM bundler friendly worker
Login

SQlite-WASM bundler friendly worker

(1) By mmomtchev on 2023-03-03 15:45:22 [source]

Hello, first time poster here

I have been fiddling with the recent SQLite WASM version and I had trouble making the bundler friendly worker to actually bundle:

(()=>{
  importScripts('sqlite3.js');
  sqlite3InitModule().then((sqlite3)=>{
    sqlite3.initWorker1API();
  });
})();

May I suggest that you do not use importScripts, as most tools do not support it. Even webpack - which is the most complete - does not support it. A much more compatible approach would be:

import sqlite3 from './sqlite3-bundler-friendly.mjs';

sqlite3().then((sqlite3) => {
  sqlite3.initWorker1API();
});

Also, I see that sqlite3-worker1-promiser-bundler-friendly.js also installs itself in window/self - just like the CDN version.

Normally a module that expects to be bunlded should export its main object - but this is more polishing than a real problem.

(2) By Stephan Beal (stephan) on 2023-03-04 02:11:23 in reply to 1 [link] [source]

I have been fiddling with the recent SQLite WASM version and I had trouble making the bundler friendly worker to actually bundle:

Forewarning/caveat: all support related to any and all node.js-based tooling is based on feedback from people who use those tools (such as yourself). Within this project none of us use, nor have any need whatsoever for, any such tools so we cannot realistically claim to "support" them out of the box. We are, however, interested in improving the integration process for such targets. With that in mind...

May I suggest that you do not use importScripts, as most tools do not support it.

We cannot remove importScripts() support but we can offer alternative versions of specific code to make it work better in ESM environments. The core parts are ignorant of such details, just the integration-level bits require some reshaping for that.

Background (feel free to skip this part): our first and foremost target platform is plain vanilla JS in browsers, and every modern browser supports importScripts(). We could not fully target ESM/import until just recently. Up until the past few months, Firefox (an important target platform for us) did not support loading Workers from ESM modules, and it will likely be a couple of years before the majority of its user base has such support because lots of people don't like to upgrade things which work for them. Thus we have no long-term plans to ever go ESM-only.

Also, I see that sqlite3-worker1-promiser-bundler-friendly.js also installs itself in window/self - just like the CDN version.

Presumably my understanding of CDN differs from yours. We don't distribute anything via any CDN.

We can/will rework sqlite3-worker1-promiser-bundler-friendly.js to use the ESM approach. The *-bundler-friendly* files exist solely for use by the node.js-using folks, not us, so it would of course make sense to get them to actually work as-is for the node folks :).

In the coming days (hopefully Sunday) i will get back to you on that and provide a prerelease snapshot for you to try out.

Thank you for the feedback and please don't hesitate to keep it coming! It's clear that integration with node-centric environments needs improvement, but we require continued assistance and feedback from its users to do so.

(3) By Cecil (mandolyte) on 2023-03-04 16:40:14 in reply to 1 [link] [source]

I have had limited success with https://github.com/overtone-app/sqlite-wasm-esm/

In particular:

  • in-memory working in the Qwik framework, but not OPFS
  • unable so far to get anything working in Next.js

I filed an issue about OPFS but so far no response... The project may be abandoned.

If you get something working, please announce it here! Much of the intricacies here are beyond me. But I'm willing to try things out.

(4) By Stephan Beal (stephan) on 2023-03-05 07:59:18 in reply to 2 [link] [source]

In the coming days (hopefully Sunday) i will get back to you on that and provide a prerelease snapshot for you to try out.

It is my hope that this checkin resolves the Worker1-loading problems for bundlers (but see the notes at the bottom of this post). Summary:

  • sqlite3-worker1-bunder-friendly.js has been renamed to sqlite3-worker1-bunder-friendly.mjs.
  • It works like an ES6 module rather than vanilla JS.
  • It loads sqlite3-bundler-friendly.mjs instead of sqlite3.js.

A prerelease snapshot with this change was just uploaded.

Any feedback on whether that works as-is, or requires further hammering, will be eagerly received.

Alas, it's not entirely clear to me how to adapt the Promiser1 API for similar usage (see related commentary in the checkin). Since neither FF nor Safari currently support loading Workers as modules (as distinct from loading Workers from within modules), and that API requires that capability for use with ESM, solving that particular problem is way down the priority list. For that matter, because neither of those browsers currently support that, it's not clear to me whether you'll be able to usably import the Worker1 API into an ESM at all.

(5) By Daniel Steigerwald (steida) on 2023-03-05 14:44:00 in reply to 3 [link] [source]

I am using the Sqlite bunder build in https://github.com/evoluhq/evolu without problems.

(6) By Daniel Steigerwald (steida) on 2023-03-05 14:46:13 in reply to 2 [link] [source]

it will likely be a couple of years before the majority of its user base has such support because lots of people don't like to upgrade things which work for them.

Firefox doesn't ask for an update. It does automatically.

(7) By Stephan Beal (stephan) on 2023-03-05 14:46:29 in reply to 5 [link] [source]

I am using the Sqlite bunder build in https://github.com/evoluhq/evolu without problems.

i hope today's changes didn't break it for you! Only the standalone "worker1 loader piece" was changed, not the core library. IIRC you're embedding the core lib, so won't (shouldn't) be affected.

(8.1) By Daniel Steigerwald (steida) on 2023-03-06 13:07:51 edited from 8.0 in reply to 7 [link] [source]

I'm still using 3.41; waiting for the official release. Btw, do you plan to use NPM? It's not hard and would make life easier for many developers. Thank you.

(9) By Stephan Beal (stephan) on 2023-03-07 06:18:48 in reply to 8.1 [link] [source]

Btw, do you plan to use NPM? It's not hard and would make life easier for many developers.

That's not a maintenance burden our single-member volunteer JS team has the bandwidth to take on.

It's not simply a matter of building and pushing bundles to the npm network. We'd also have to adopt completely new (to us) tooling, which we otherwise have no use for, devise and maintain tests for those bundles, and then keep up to date with the related tooling.

We welcome all third-party efforts to distribute any portion of sqlite through the countless ecosystem-specific package systems, but we do not welcome the maintenance burden of doing so ourselves.

(10) By mmomtchev on 2023-03-07 11:51:59 in reply to 9 [link] [source]

I am working on an HTTP VFS layer for SQLite WASM inspired from the already existing one for sql.js. It will contain SQLite-WASM, including TypeScript definitions, and I plan to distribute it via NPM as a bundler-friendly ES6 module.

(11) By mmomtchev on 2023-03-07 11:58:12 in reply to 2 [link] [source]

Yes, indeed, I meant the CDN-compatible version - the one that is to be included in a <script> tag. importScripts is probably the best choice for that one - however the bundler-friendly one should probably use either a static

import xxx from ...

or the asynchronous function-like import() which returns a Promise.

AFAIK, no bundler supports importScripts(). Thus said, the provided worker code is optional and can be easily replaced by user code, it is a simple two-liner. As this part will always be very dependent on the user-environment, maybe it is best if it is simply left to the final user.

(12) By Stephan Beal (stephan) on 2023-03-07 12:06:07 in reply to 11 [link] [source]

Thus said, the provided worker code is optional and can be easily replaced by user code, it is a simple two-liner. As this part will always be very dependent on the user-environment, maybe it is best if it is simply left to the final user.

We now (in the current trunk, not in 3.41) provide both options, in any case, because it's easy to do so. Whether or not the Worker1 API will actually work when started from a Module-type Worker is still an open question, though. Apparently (based solely on MDN compatibility charts) neither Safari nor Firefox currently support that.