Client-side VFS implementation - StructBinder

(1) By mmomtchev on 2023-03-03 17:28:55 [source]

I am trying to add an HTTP VFS implementation similar to https://github.com/phiresky/sql.js-httpvfs for sql.js

It is going to be very similar to the OPFS implementation following the same sync<->async proxy model.

In order to install my handlers, I am calling sqlite3.vfs.installVfs with the same methods as in the OPFS implementation.

However I am getting an exception in installMethod() - when trying to access sqlite3.StructBinder.StructType.

This structure is available during bootstrap for the OPFS implementation to register, but it is removed once the bootstrap is complete.

I did investigate it and I found this comment:

/* It's conceivable that we might want to expose
   StructBinder to client-side code, but it's only useful if
   clients build their own sqlite3.wasm which contains their
   one C struct types. */
delete sqlite3.StructBinder;

Are client-side VFS implementations something you do not intend to support?

Is there any simple work-around for this issue?

(2) By mmomtchev on 2023-03-03 21:25:28 in reply to 1 [link] [source]

Ok, I did expose sqlite3.StructBinder myself and the result is that it seems to work. May I suggest that you simply remove the delete statement?

(3) By Stephan Beal (stephan) on 2023-03-04 02:23:08 in reply to 2 [link] [source]

May I suggest that you simply remove the delete statement?

i'm torn on that. The StructBinder is an internal implementation detail, not part of the public API. If we do not delete then someone will start using it and will complain loudly when that API changes or is replaced. It is to our long-term advantage to not expose that to clients.

/* It's conceivable that we might want to expose
   StructBinder to client-side code, but it's only useful if
   clients build their own sqlite3.wasm which contains their
   one C struct types. */

i wrote that (including the typo!) but am still ot currently keen on exposing that API to clients. It's not 100% ruled out, but it's not something to do willy-nilly because once we do then we're stuck with it for better or worse.

Are client-side VFS implementations something you do not intend to support?

Sure, but they don't have to use StructBinder to do that. That's just our home-grown way of internally gluing the JS and C views of the structs together.

Is there any simple work-around for this issue?

Not an immediate one, no. We know that other projects can and do offer VFSes without StructBinder, so we know it's possible. How they do it, i've honestly got no clue.

Let me (A) mull this over more and (B) discuss this with our project lead (Richard a.k.a. drh) and see how he feels about the ongoing maintenance and support burden of publicizing StructBinder (noting that i'd be the one taking on that burden ;).

(4) By mmomtchev on 2023-03-07 12:16:54 in reply to 3 [link] [source]

I just checked, and there is no way around StructBinder in sqlite3.vfs.installVfs - because the I/O methods structure is in fact a C structure.

There are obviously two levels to this: simply using existing C-structs and defining new ones. As far as I see, if you do not allow the creation of new structures, the interface won't be actually exposed - only installVfs will use it.

This is a problem that is quite common in the Node.js internal libraries and there is a trick that their team will always use - make the property name a Symbol. This way it will be accessible only to the internal methods:

const secretName = Symbol('StructBinder');

sqlite3.vfs[secretName] = StructBinder;

and then to access it:

sqlite3.vfs[secretName]();

(5) By Stephan Beal (stephan) on 2023-03-07 12:52:12 in reply to 4 [link] [source]

sqlite3.vfssecretName;

That Symbol would need to be bound somewhere public because StructBinder is used across multiple disparate scopes, the only common points of reference for them being the sqlite3 object and the global scope. Accessing it via a Symbol would, unless i'm misunderstanding (which is certainly possible), still leave StructBinder exposed to client-level code (indirectly, via access to the Symbol object).

In any case, i now have a better understanding of the issue and will work on a reformulation which should resolve this issue for you. That should be available in the next few hours.

(6) By Stephan Beal (stephan) on 2023-03-07 12:59:36 in reply to 5 [link] [source]

In any case, i now have a better understanding of the issue and will work on a reformulation which should resolve this issue for you. That should be available in the next few hours.

Well, that was easy. The current trunk fixes this and a new prerelease snapshot has been uploaded. Thank you for the feedback!