I'm curious about blockers and estimates. Firefox should support OPFS already, and if I remember correctly, there was some issue with Safari. May I ask for details?

I'm curious about blockers and estimates.

We don't specifically track the browsers for that stuff. The OPFS bits are written to the documentation, so should "just work" if and when the other browsers support it. If we happen to become aware that some other non-dev-channel release of a given browser now supports it, we will of course make sure that it works on that platform, insofar as possible, but we don't make any effort to follow the status of dev-channel/canary builds.

Firefox should support OPFS already,

FF v110 (this browser) apparently does not, though i seem to recall recently reading that it's included in their nightly builds.

if I remember correctly, there was some issue with Safari. May I ask for details?

As to Safari's status, i follow that even less closely than FF's. My very vague, and perhaps incorrect, recollection from a meeting last August or September with some Safari folks is that they didn't yet have the capability to update blobs in-place. That was ages ago, though.

OK, thank you for the answer.

Do you plan any further development, or it's SQLite Wasm done?

Do you plan any further development, or it's SQLite Wasm done?

No software is ever really "done" so long as it's maintained, and we have no plans on abandoning it so long as JS and WASM remain relevant.

So I tested Sqlite Wasm in Safari Technology Preview, which failed on FS.genericErrors[44]. Safari OPFS is probably not ready yet.

if (attr.timestamp !== undefined) {
  node.timestamp = attr.timestamp;
}
if (attr.size !== undefined) {
  MEMFS.resizeFileStorage(node, attr.size);
}
}, lookup: function (parent, name) {
throw FS.genericErrors[44];

Firefox 111 beta works out-of-the-box 🥳

So I tested Sqlite Wasm in Safari Technology Preview, which failed on FS.genericErrors[44]. Safari OPFS is probably not ready yet.

The "FS" object is Emscripten's virtual filesystem API, which we don't use for OPFS, so that error is indicative of a bug in either Emscripten or Safari - i can't say which with any certainty but suspect Emscripten because it's a relatively volatile platform compared to Safari.

Emscripten has a successor filesystem API called WASMFS which has transparent OPFS support, but we don't build against it for several reasons:

1. It's less portable - it doesn't build or work on some ARM platforms (mobile devices or Raspberry Pi).

2. Experience has shown it to still be a "moving target." What support we used to have for it was broken by incompatible changes from one release to the next.

3. Its file-locking support is not granular enough to support multi-tab concurrency of OPFS-hosted DBs. Because their API is high-level and application-agnostic, they have no real choice but to hold a lock on the file from the time of the first write until the file is closed. OPFS-custom VFSes, like ours or the one from wa-sqlite, have finer-grained support for concurrency (but are of course specific to sqlite dbs).

However, their WASMFS will enable arbitrary applications to host files on OPFS using the exact same API as their other (virtual) filesystem backends, so it will be an important feature once it stabilizes.

Firefox 111 beta works out-of-the-box 🥳

That is fantastic news :). My system is still at v110, despite having run a system update yesterday :(.

Can you please explain the problem with multi-tab concurrency? I serialize (run one by one) all reads/writes between tabs; where is the problem? Thank you

Can you please explain the problem with multi-tab concurrency? I serialize (run one by one) all reads/writes between tabs; where is the problem?

WASMFS is a general-purpose filesystem interface, completely ignorant of the application. It supports several different virtual filesystems, one of which is OPFS, behind a generic filesystem interface.

When you open a file in OPFS, you get a FileSystemFileHandle to it, which only allows async operations. In order to perform synchronous operations, the client (e.g. the WASMFS OPFS driver) has to use FileSystemFileHandle.createSyncAccessHandle() to acquire a so-called "sync access handle." Per the whatwg documentation:

Creating a FileSystemSyncAccessHandle takes an exclusive lock on the entry associated with fileHandle. This prevents the creation of further FileSystemSyncAccessHandles or FileSystemWritableFileStreams for the entry, until the access handle is closed.

My understanding, based on a discussion with one of the Emscripten devs about half a year ago, is that WASMFS's OPFS driver delays creating the sync access handle until a synchronous operation is performed, at which point it acquires a sync access handle, which locks the file in OPFS. (Note that the high-level WASMFS API knows nothing about this - its public API does not include locking-related operations.)

At that point the WASMFS OPFS driver keeps that sync access handle open because it has no way of knowing whether it will be needed again, and opening/closing the sync handle for each synchronous operation has a tremendous performance penalty (up to 400% in our basic benchmarks) and may (depending on how the handle is used) create opportunities for file-modification race conditions.

If you open a db hosted in OPFS-via-WASMFS from 2+ tabs, the first one which requires a synchronous operation will lock it and the other tab won't be able to perform any synchronous operations, nor any write ops (synchronous or not), with it until the first db is closed. Until that time, the 2nd (3rd, 4th, etc) tab(s) will report locking-related errors.

That said: that was based on an old discussion, and WASMFS is under active development, so it's possible that my understanding of OPFS-via-WASMFS is no longer current.

Note that the above does not directly apply to either the wa-sqlite OPFS VFS or our OPFS VFS. Neither uses WASMFS and both manage the sync access handles at a lower level than is possible via the high-level WASMFS API.

It's out ✨

So the only browser where we can't use OPFS is Safari. Can you please ask your friend from the Apple Safari team where the problem is? Thank you.

It's out ✨

Woot!

Can you please ask your friend from the Apple Safari team where the problem is?

My only contact with them was a one-time meeting 6-ish months ago - i don't have any contacts in the Safari team. It would not surprise me a bit if someone in this forum either belongs to that team or knows someone on that team, though. If they're listening in, perhaps they'll pass on the request.

According to MDN, FileSystemSyncAccessHandle on Safari supports the required operations since 15.2 (Dec. 2021). If they currently do support it, and our JS is not seeing it, then it's presumably an incompatibility with our browser-side feature detection, and any assistance from Safari users in correcting that detection (or related incompatibilities) would be appreciated. When it comes to Safari support, i'm 100% dependent on MDN, caniuse.com, and feedback from its users.

My OS's package repositories don't yet have FF v111 but it should be available to me "real soon now."

I got it. You don't have an Apple device to run Safari. So I should try to fix it on my machine and copy-paste the fix here. Will do.

Did you get some new insights? I tried running https://github.com/dellisd/sqldelight-sqlite-wasm on latest Safari and Safari tech preview.

It both fails in sqlite3InitModule() from sqlite3.js. Safari throws SQLite3Error: sqlite3 result code 1: no such vfs: opfs. In Safari tech preview, the promise returned by sqlite3InitModule() is never resolved, nor does it throw an error.

One problem is that safari does not support spawning web workers from inside web workers (https://bugs.webkit.org/show_bug.cgi?id=25212). And this happens in sqlite3.js (search for new Worker). Safari tech preview supports it though.

The relevant line for sqlite not working on safari is 10842: "new Worker(new URL("sqlite3-opfs-async-proxy.js", import.meta.url));" On current safari it throws an error as workers cannot be created, on safari tech preview the worker is created, but never starts and therefore the promise, where the worker is created is never resolved in its onmessage function.

I think this is where I am stuck and can't give more insight.

The relevant line for sqlite not working on safari is 10842: "new Worker(new URL("sqlite3-opfs-async-proxy.js", import.meta.url));" On current safari it throws an error as workers cannot be created, on safari tech preview the worker is created, but never start

That particular line is in the ESM build. Firefox also still cannot load workers from ESM modules but can from vanilla JS (even from another worker, as of v111). It would be interesting to know whether Safari can use the vanilla JS build for that. That can be tested by checking whether the OPFS test group on this page works:

https://wasm-testing.sqlite.org/tester1-worker.html

The OPFS group is labeled group #6 as of this writing. The output should look something like:

Group #6: OPFS: Origin-Private File System
     6.1: OPFS db sanity checks
          5 assertion(s) in 237.39 ms
     6.2: OPFS export/import
          1 assertion(s) in 82.69 ms
     6.3: OPFS utility APIs and sqlite3_js_vfs_create_file()
          15 assertion(s) in 164.42 ms
Group #6: 21 assertion(s) in 484.51 ms

(edit: the above timings are from a mid-range Android tablet running Chrome. It will likely run faster on most devices.)

Here are the results on safari:

"Group #6: OPFS: Origin-Private File System SKIPPING group: requires Worker thread in a compatible browser"

Probably, because safari cant start workers from workers. and on safari tech preview:

"Loading and initializing sqlite3 WASM module..."

It stays on this, I guess because the worker is created but never started and therefore the code hangs, because the promise is neither rejected nor resolved.

I got the non module version working using this polyfill: https://github.com/dmihal/Subworkers However it throws another error: Unhandled Promise Rejection: DataCloneError: The object can not be cloned.

The flow of events is the following for opening the database and trying to make a read query:

xopen xread xlock xaccess xfilesize xunlock xlock xaccess xread xfilesize xunlock -> crash

I'll try to use subworkers on your tests.

https://sqlitewasm.oscar-app.de/tester1-worker.html

This is the adapted test. It now gives the following results on safari:

[Debug] Cannot test db_config(SQLITE_DBCONFIG_LOOKASIDE) – "while lookaside memory is in use." (tester1.js, line 1196)

[Debug] db.onclose.before cleaning up: – (db)=>{ (tester1.js, line 1123) //console.debug("db.onclose.before dropping modules"); //sqlite3.capi.sqlite3_drop_modules(db.pointer, 0); }

[Debug] db.onclose.after cleaning up: – StructCtor {ondispose: Array, setupModule: function, addOnDispose: function, …} (tester1.js, line 1137) StructCtor {ondispose: Array, setupModule: function, addOnDispose: function, installMethod: function, installMethods: function}StructCtor

[Debug] db.onclose.after cleaning up: – StructCtor {ondispose: Array, setupModule: function, addOnDispose: function, …} (tester1.js, line 1137) StructCtor {ondispose: Array, setupModule: function, addOnDispose: function, installMethod: function, installMethods: function}StructCtor

[Error] SQLite3Error: sqlite3 result code 1: no such table: p error (tester1.js:136) (anonyme Funktion) (tester1.js:326) asyncFunctionResume (anonyme Funktion) promiseReactionJobWithoutPromise promiseReactionJob

[Error] Unhandled Promise Rejection: SQLite3Error: sqlite3 result code 1: no such table: p (anonyme Funktion) rejectPromise promiseReactionJob

"Loading and initializing sqlite3 WASM module..."

It stays on this, I guess because the worker is created but never started and therefore the code hangs, because the promise is neither rejected nor resolved.

We can reasonably consider that to be a bug, but it's unclear whether we can resolve it by rejecting the promise or whether that case is undetectable for us. Perhaps we can set a timer on the OPFS worker-load init and reject the promise if the sub-Worker constructor silently fails (that is, it neither throws nor starts up the sub-worker code). i will experiment with that.

(Edit: that's done an in trunk.)

Thank you for the feedback.

In case you don't know this, the tests on wasm-testing.sqlite.org can be found in the source tree under ext/wasm. It simply needs a compatible web server running with that directory as the root. The makefile has an "httpd" target which starts up althttpd for the simple reason that it's the http server which members of this project tend to use, but the tests "should" work with any web server which can emit the COOP/COEP headers. Without those headers, the tests as a whole will work but OPFS will not.

Cannot test db_config(SQLITE_DBCONFIG_LOOKASIDE) – "while lookaside memory is in use." (tester1.js, line 1196)

That's part is just noise - that particular test will be removed. It was added before i understood that it cannot work.

SQLite3Error: sqlite3 result code 1: no such table: p error (tester1.js:136) (anonyme Funktion) (tester1.js:326) asyncFunctionResume (anonyme Funktion) promiseReactionJobWithoutPromise promiseReactionJob

Presumably promiseReactionJob is part of the polyfill you're using? We can maybe resolve this lack of proper rejection with a timer, as mentioned in my last comment. If that works, the above failure would instead be reported as an incompatible browser. i will work on that this evening and post here when the test site is updated.

Safari with nested promises (16.4) is out. Do you plan to test it?

Safari with nested promises (16.4) is out. Do you plan to test it?

Nope. My testing is limited to platforms i use, which rules out the commercial OSes.

Presumably promiseReactionJob is part of the polyfill you're using? We can maybe resolve this lack of proper rejection with a timer, as mentioned in my last comment. If that works, the above failure would instead be reported as an incompatible browser. i will work on that this evening and post here when the test site is updated.

That change is now experimentally in trunk and on the test server:

https://wasm-testing.sqlite.org/tester1-worker.html

We could try Epiphany (Gnome Web) as one more Open sour e Browser that uses the Same Webkit as Safari

We could try Epiphany (Gnome Web) as one more Open sour e Browser that uses the Same Webkit as Safari

Epiphany is new to me. It does run our overall test suite but it's is missing the SharedArrayBuffer class, which our OPFS driver requires in order to synchronize communication between two threads.

It relies on WebKitGTK which has its own version tagging in the WebKit GitHub project, so its hard to compare it to the version used by safari, which itself does not provide an easy way to get its webkit version. https://github.com/WebKit/WebKit/tags

Are you going to create an issue on bugzilla webkit? https://bugs.webkit.org/buglist.cgi?bug_status=open&content=sharedarraybuffer&no_redirect=1&order=Importance&product=WebKit&query_format=specific

Are you going to create an issue on bugzilla webkit?

No. Keeping track of the specific bugs and shortcomings of any given browser is way out of scope for us. Our JS code is written based on publicly-available documentation (primarily MDN and caniuse.com) and user feedback, and we do not otherwise delve into or track the specifics of individual browsers. It's interesting to hear about them, but actively following them is not something we have the bandwidth for¹.

1. ^{^} Noting that we have only a single JS-related developer on the team 🙋‍♂️.

Safari tech preview now gives correctly: Ignoring inability to install OPFS sqlite3_vfs:"Timeout while waiting for OPFS async proxy worker."

Safari tech preview now gives correctly: Ignoring inability to install OPFS sqlite3_vfs:"Timeout while waiting for OPFS async proxy worker."

Is that correct, though? My (mis?)understanding is that the 16.x tech preview supports nested workers? If they do, the OPFS sqlite_vfs "should" work.

Or was this using the polyfill (which i would not expect to work)? If so, then that is indeed the expected/hoped-for result.

BTW: if you prefer, feel free to contact me off-list about this (stephan at this domain).

After some debugging of the SQLite code, I managed to boil down the issue to one core issue.

It's simply that navigator.storage.getDirectory() fails in subworkers with an InvalidStateError: https://bugs.webkit.org/show_bug.cgi?id=255458.

Why this was never detected is because Web Inspector simply is completely blind to any and all subworkers: https://bugs.webkit.org/show_bug.cgi?id=255402.

Key to figuring out what was going on was to patch the SQLite logger in the subworker sqlite3-opfs-async-proxy.js in such a way that it uses postMessage() to send its (error) logs over to the worker sqlite3-bundler-friendly.mjs where I could console.log() the messages.

This means that this call never succeeds:

• https://github.com/sqlite/sqlite/blob/41334cf960a472a31729e2914b7c3c7f9014a555/ext/wasm/api/sqlite3-opfs-async-proxy.js#L844

This in turn means that the initial promise sqlite3InitModule() never resolves.

I have pinged Apple folks.

Cheers, Tom

Yeah, I tried with both Safari Tech Preview and Safari and the sqlite3InitModule promise never resolves. Any work-arounds to get it to work?

Any work-arounds to get it to work?

Nope. That limitation makes the browser fundamentally incompatible with our OPFS driver. According to the ticket history it's since been resolved, but we can only speculate about when it will reach users.

There is a third-party solution which might work for you. That project makes use of "Asyncify" to automate bridging the gap between the synchronous and asynchronous APIs and it reportedly works with Safari. In contrast, this project uses hand-written code which coincidentally uncovers the above-linked Safari limitation.

Safari 17 is out and OPFS is working.