SQLite Forum

Any examples of virtual tables that use idxInfo->idxStr?
Login

Any examples of virtual tables that use idxInfo->idxStr?

(1) By David Jones (vman59) on 2021-05-24 20:48:01 [link] [source]

For it's described usage (an opaque pointer for conveying information between xBestIndex() and xFilter()), having idxStr be a char * rather than void * would seem to make things awkward in the general case. What did the designer of the interface have in mind?

(2.1) By Richard Hipp (drh) on 2021-05-24 22:41:43 edited from 2.0 in reply to 1 [link] [source]

idxStr does need to be a string. SQLite might make a copy of idxStr (to become the 4th parameter to the OP_VFilter opcode in the byte-code engine) and when doing so, it copies the string through the first 0x00 character.

Where in the documentation does it say that idxStr can be a pointer to arbitrary data? That sounds like something that needs to be fixed.

To answer the question posed in the title - the built-in R-Tree virtual table uses idxStr.

(3) By Larry Brasfield (larrybr) on 2021-05-24 23:46:37 in reply to 2.1 [source]

Section "2.3.3 Outputs" here says, "The information in idxNum and idxStr is arbitrary as far as the SQLite core is concerned. The SQLite core just copies the information through to the xFilter method. Any desired meaning can be assigned to idxNum and idxStr as long as xBestIndex and xFilter agree on what that meaning is."

I suspect Mr. Jones has interpreted that language to be a meaningless variation on the truly opaque, void-pointer-referent theme seen in other SQLite interfaces.

Perhaps the fragment "core just copies the information" needs to explain that the NUL-termination convention must be respected.

(4.1) By David Jones (vman59) on 2021-05-25 01:18:41 edited from 4.0 in reply to 3 [link] [source]

Yes, that was my mis-reading. "Any desired meaning" doesn't automatically imply 'text' to some people.

(5) By anonymous on 2021-05-25 01:29:18 in reply to 1 [link] [source]

A convention that I have used in some programs is for idxStr to contain one character per constraint value passed to xFilter (in the argv array), folowed by the null terminator, so that the filter knows what each argument means.

I looked now and the R-tree extension does something similar, actually.

(6) By Larry Brasfield (larrybr) on 2021-05-25 16:53:30 in reply to 4.1 [link] [source]

FYI: This is now clarified in the doc sources and soon will be in the website docs.

Thanks for the pickup.

(7) By Gunter Hick (gunter_hick) on 2021-05-26 09:17:48 in reply to 2.1 [link] [source]

Is there a specific set of circumstances where SQLite chooses to pass a "text copy" of idxStr to vFilter instead of the verbatim pointer? Seems we have been lucky so far...

(8) By Max (Maxulite) on 2023-01-24 14:33:17 in reply to 1 [link] [source]

This is probably the only place in the whole universe where this topic is seriously discussed so I decided not to open a new one, but to continue here

It appears I implemented a universal approach many years ago using idxStr arbitrarily not restricting the usage to string-only data. Everything always worked as expected (or at least as I expected it from arbitrary pointer). But recently rereading the docs (for unrelated reasons) I noticed the new sentence "The SQLite core just copies the information from xBestIndex through to the xFilter method, assuming only that the char sequence referenced via idxStr is NUL terminated". This discussion explained that as Mr. Jones, I assumed this opaque pointer feature, but long before him.

But ffter examining the sources I didn't find the corresponding part in the code.

So we have pointer assignment in the xBestIndex->xFilter logic

static int whereLoopAddVirtualOne(

.......

pNew->u.vtab.idxStr = pIdxInfo->idxStr;

... no copying, and for the OP_VFilter mentioned by drh the following...

sqlite3VdbeAddOp4(v, OP_VFilter, iCur, addrNotFound, iReg, pLoop->u.vtab.idxStr, pLoop->u.vtab.needFree ? P4_DYNAMIC : P4_STATIC);

also no string copying.

There are several other places where symbol idxStr mentioned, but they all look unrelated.

By Gunter Hick

> Seems we have been lucky so far...

It looks we have. But If there are plans to use idxStr strictly as a string, please let us know.

Thanks

(9) By Gunter Hick (gunter_hick) on 2023-01-24 15:08:54 in reply to 8 [link] [source]

Same hope here. The structures I need to pass here are quite complex and I cannot guarantee that they are free of embedded NUL bytes. Apart from being referenced in the index info parameter of xBesIndex and being passed into xFilter, the actual pointer is also required for calling the destructor if "needFree" is set.

(10) By David Jones (vman59) on 2023-01-24 15:49:10 in reply to 8 [link] [source]

Where I can, I use the single byte per argument approach (reserving NUL to be end of list marker). In one other case, I encoded the information I needed to pass as text-encoded tuples, which the filter routine had to parse. If it was a really complex structure, I'd probably resort to JSON.

(11.1) By Bjoern Hoehrmann (bjoern) on 2023-01-24 21:24:11 edited from 11.0 in reply to 8 [link] [source]

It seems to me from a cursory glance that sqlite3VdbeAddOp4 -> sqlite3VdbeChangeP4 -> vdbeChangeP4Full -> sqlite3Strlen30/sqlite3DbStrNDup is a path that might lead to string copying the idxStr.

Also, idxStr seems to be used in a number of string formatting parts (like debug prints, possibly EXPLAIN) which might result in out-of-bounds reads if the data is not in fact a null-terminated C string and possible memory disclosure problems.

(12) By Max (Maxulite) on 2023-01-25 06:57:04 in reply to 9 [link] [source]

Assuming a bad outcome with arbitrary structures is still possible, probably the quickest path to safety is to use something like base64 encoding-decoding. For small structures the overhead will be negligible I think.

(13.1) By Max (Maxulite) on 2023-01-25 12:48:20 edited from 13.0 in reply to 11.1 [link] [source]

While currently considering changing the contents to the string (see my base64 remark), I still would like to know why we're still lucky despite your probably right path to how sqlite should definitely make the copy of idxStr

What I found (version 3.40.1)

  • sqlite3VdbeAddOp4(... OP_VFilter, passes either P4_DYNAMIC or P4_STATIC as the last (p4type parameter) depending on the needFree, both constants are negative
  • sqlite3VdbeAddOp4 calls sqlite3VdbeAddOp3 that initializes p4type member of the operation record (not to be confused with the parameter p4type) to P4_NOTUSED (zero)
  • p4type parameter goes inside sqlite3VdbeChangeP4 as n
  • So in order for vdbeChangeP4Full to be called the following should be true

if( n>=0 || pOp->p4type ){ vdbeChangeP4Full(p, pOp, zP4, n);....

the expression is false, cause n is negative and p4type member is 0. So the string is never copied.