SQLite User Forum

SAHPool VFS: How to close the VFS
Login

SAHPool VFS: How to close the VFS

(1) By Matt (Matt_TOTW) on 2025-01-28 17:31:09 [source]

Hello, I am wondering if it is possible to close the sahpool once it has been installed (with installOpfsSAHPoolVfs). For context, what I want to do is to first create and use an in-memory db (with oo1.DB) and then, with user permission, import that into OPFS. Subsequent sessions will use the db in the opfs with the sahpool vfs so my understanding is that I need to use that vfs for the import because of the opaque file names - let me know if I am mistaken on that! The reason I want to close the importing vfs is a) I don't need it as I will continue to use the in-memory db for the duration of the session, and more importantly b) If a session is opened in another tab, I want to be able to use the db that was imported into the opfs. Additional tabs beyond that will use a shared service (as pointed to in the docs, thanks rhashimoto!). Many thanks!

(2) By Stephan Beal (stephan) on 2025-01-28 17:43:36 in reply to 1 [link] [source]

Hello, I am wondering if it is possible to close the sahpool once it has been installed

Yes, but... removing the VFS will also remove all data associated with the VFS. Once activated, there is currently no API for removing/disabling the VFS without also removing the associated data. That seems like a reasonable thing to be able to do, though, so i've noted it here on my local TODO list and will report back in this thread either when it's working or if there's some as-yet-unrecognized hurdle which makes it unfeasible (there is currently no reason to believe that's the case, though).

For completeness's sake, the current ("nuclear option") API can be found by searching this page for "removeVfs".

(3) By Matt (Matt_TOTW) on 2025-01-28 17:57:44 in reply to 2 [link] [source]

Thanks Stephan, it’s as I suspected but it’s great to know you’ll be giving it some attention, thanks! I did find removeVfs and yep, found it to be very nuclear indeed!

(4) By Stephan Beal (stephan) on 2025-01-29 11:38:26 in reply to 2 [link] [source]

... will report back in this thread either when it's working or if there's some as-yet-unrecognized hurdle which makes it unfeasible (there is currently no reason to believe that's the case, though).

There is an experimental version of this in the opfs-sahpool-pause branch and there's a snapshot build on the wasm site which contains this feature. See the diff in that first link for the docs and a test which demonstrates it, but here's a summary:

  • Assuming that ospu is a OpfsSAHPoolUtil instance...
  • ospu.pauseVfs() will (A) close all opened SAH handles and (B) unregister the VFS from SQLite. It throws if any db instance is open with that VFS because the only other alternative would be to close the file handles out from under the db, invoking Undefined Behavior. In other words: the VFS can only be paused if it's not in active use.
  • ospu.isPaused() returns true if the VFS is paused, else false.
  • async ospu.unpauseVfs() will (A) reacquire the SAH handles, (B) re-register the VFS with SQLite, and (C) return a Promise which resolves to ospu.

Would that API help you solve the problem you are trying to solve? If not, suggestions are welcomed. (Similarly, if you have a better name for this feature than "pause," please suggest it.)

Known caveats:

  • This feature requires far more testing before i'm comfortable merging it to trunk for the 3.49 release, but "so far so good."

  • The VFS setup is computationally slow, with its initialization time being directly related to its capacity (O(N)) because it has to asynchronously open each OPFS handle during initialization/unpausing. The higher the VFS's capacity, the longer pausing and unpausing will take.

(5) By Matt (Matt_TOTW) on 2025-01-29 18:01:43 in reply to 4 [link] [source]

This sounds really great, and definitely accomplishes what I was looking for. ‘isPaused’ and ‘unpauseVfs’ are interesting additions. I am not surprised (and fwiw agree) that it should error if a db is open, very sensible and manageable if you ask me. I was expecting that the solution would be to simply tear it down. What I mean by that is that to use it again would require another call to ‘installOpfsSAHPoolVfs’. Consequently, I was expecting the name to be “uninstall” or “close” or similar but the api as you have specified it seems to fit well with “pause”. “stop” could also work and is perhaps a bit more indicative of it releasing its handles. Big thanks and thumbs up from me!!

(6.1) By Stephan Beal (stephan) on 2025-01-31 16:37:56 edited from 6.0 in reply to 5 [link] [source]

This sounds really great, and definitely accomplishes what I was looking for. ‘isPaused’ and ‘unpauseVfs’ are interesting additions.

Matt, i'm wrapping up testing of this feature for merging into trunk. If, in the mean time, you have a better name for this feature than "pause" then please let me know. Once it's released with that name (in 3.49) then we'll be stuck with it forever.

Edit:

There is a very basic demo app which uses this new capability to coordinate access to an SAHPool-hosted db between two workers on the wasm testing site. Search that page for "semi-concurrency" (near the bottom) for the link. Now that the proof-of-concept in place my plan is to merge this to trunk within the next few days (but i'd like to consider the "pause" name a bit more first).

During development of that test app it quickly became clear that it would be helpful to be able to start the VFS pre-paused by passing a startPaused:true config flag, but the internals do not currently lend themselves well to that. Hopefully i can find a way to work that in. "The problem" is that pausing assumes that the storage infrastructure is already in place, which is not the case during VFS initialization (complicated by the fact that initialization may require modifying the capacity of the VFS or even wiping it clean).

(7) By Matt (Matt_TOTW) on 2025-01-31 20:59:03 in reply to 6.1 [link] [source]

Impressive, the demo looks good to me. As an alternative to ‘pause’, my only other suggestion would be attachVfs, isVfsAttached, unattachVfs, but I don’t have a strong feeling about it. I’m curious about the start paused flag. This is cool and I can see why you called your demo ‘semi-concurrency’. In fact for my use case it may be good enough to replace my current ‘shared service’ thing (with a provider and clients and broadcast channels etc) as I don’t really need ‘true concurrency’ ie simultaneous connections. I just need the ability for separate windows to access the same db, and this will almost always be one at a time. Very cool, and actually not what I originally had in mind. But yep, nice!

(8) By Stephan Beal (stephan) on 2025-02-02 11:22:11 in reply to 6.1 [link] [source]

Once it's released with that name (in 3.49) then we'll be stuck with it forever.

Update: the accelerated 3.49 release timeline doesn't allow for squeezing this in to that release, and it's a new feature, not a bug fix, so won't be in any 3.49.x patch releases. After 3.49 is released, the pause/unpause capability will be merged for the pending 3.50 release.

(9) By Matt (Matt_TOTW) on 2025-02-02 18:04:55 in reply to 8 [link] [source]

Great, there’s no need to rush it. What naming did you end up going with? Did you stick with ‘pause’?

(10.1) By Stephan Beal (stephan) on 2025-02-20 04:47:07 edited from 10.0 in reply to 1 [link] [source]

Hello, I am wondering if it is possible to close the sahpool once it has been installed (with installOpfsSAHPoolVfs).

Matt, the new (un)pause capability is in the trunk so will (unless something goes horribly wrong) be part of 3.50 and works as described elsewhere in this thread.

A snapshot build containing this feature is on the wasm site and a very primitive demo of two workers coordinating this way can be seen at https://wasm-testing.sqlite.org/ (search that page for "cooperative").

One feature i would like to add but have not yet figured out how to do is initialize/register the VFS in a pre-paused state. Because the initialization may have side effects beyond simply opening the SyncAccessHandles (e.g. resizing the pool or even emptying all of its contents), it's not currently possible to initialize the VFS in a pre-paused state. It must acquire the SAHs in order to initialize. This means that clients coordinating access to the VFS will have to coordinate their calls to installOpfsSAHPoolVfs() as well as their calls to un/pauseVfs(). Whether or not that limitation can be sensibly solved is not yet clear. Ideally, it would be nice to be able to pass paused:true as an initialization option so that multiple clients could all do so without stepping on each other's toes.

Edit: if you end up using this for cross-tab/page/worker concurrency, i'd be really interested in seeing your solution and linking to it from the docs.