A couple of typos in https://sqlite.org/vec1/doc/trunk/doc/vec1intro.md

2.2.2. Training and Index Building

nbucket should be set to roughly the square of the expected size of the db.

s/square/square root/ (I hope!)

2.2.4. Reranking

these vectors are heavily compressed (often 32 time or more)

s/time/times/

Thanks! Should now be fixed.

Dan.

I may be missing something, but I wasn't able to compile it without threads (for Wasm)?

Seems like the single threaded version is missing some "stubs" like vec1CondWait (if they can be implemented as stubs)?

Hi. Thanks for this. Hopefully now fixed here:

https://sqlite.org/vec1/info/ee8a8a8e238072ed

Dan.

Thank you!

I was able to compile it for Wasm as a dylink.0 shared module. :)

Not many tests though, beyond I can "link" separately compiled bits at runtime and they load/run (this was the first one I tested with).

There's also a printf wrapped in a #if 1 which I assume is a leftover debug? And a few other such #ifs too.

Oh, and clang -Wall uncharacteristically prints a bunch of warnings (unused variables, maybe from commented out debug code, but plenty of signed to unsigned compares). I can paste the log, but I imagine GCC would report the same. I've quieted those, for now.

The Microsoft compiler also complains a lot: 36 warnings on current code compared to 1 from clang v21.

Most of them seem silly, but a few seem like it zigs when all the other compilers zag, and could produce unintended results.

I assume you saw this Dan, thanks for the fixes. :)

The only warning left is an unused kk variable, that's used in the VEC1_HAVE_AVX2 configuration, which is perfectly fine by me.

Thanks very much for testing things!

Any chance of getting a JSON object saying how a vec1 table is currently configured? Some information that would be useful:

• what is the index mode
• what is the distance (l2/cos)
• is there a model
• how big is the model
• what were its training params
• how many vectors was the model trained on
• how many vectors are in the table (if quicker than SELECT COUNT(*) FROM table)

It allows checking the table is setup as intended, print the config for benchmarks etc, and decide if retraining would be useful.

Sorry for the delay - was afk for an extended period.

I think you should be able to query for a description of the mode/model at any rate. And the number of training vectors seems like it might be quite useful as well. Will try to work that out.

Deciding when retraining might be useful is difficult. At the moment for non-NN queries the "distance" column contains some stats on the vector. So:

sqlite> select rowid, distance FROM v1 LIMIT 10;
╭───────┬────────────────────────────────────────────────────────────────────╮                           
│ rowid │                              distance                              │                           
╞═══════╪════════════════════════════════════════════════════════════════════╡                           
│     0 │ {bucket:853, coarse_error:0.438291, reconstruction_error:0.188406} │                           
│     1 │ {bucket:445, coarse_error:0.419078, reconstruction_error:0.193036} │                           
│     2 │ {bucket:853, coarse_error:0.379638, reconstruction_error:0.166144} │                           
│     3 │ {bucket:216, coarse_error:0.412718, reconstruction_error:0.185848} │                           
│     4 │ {bucket:800, coarse_error:0.426382, reconstruction_error:0.178255} │                           
╰───────┴────────────────────────────────────────────────────────────────────╯ 

The idea being that if either the coarse or reconstruction error for recently inserted vectors appears to be higher than for those the model was trained on, or if new vectors do not seem to be evenly distributed between buckets, retraining may help. Not sure how easy that is going to be to use though. Any ideas anyone has are hereby solicited!

Dan.

The scenario I was thinking of is cruder and long term. eg a table has been use for over a year, so at startup the app initialization code has to make a quick pragmatic decision about whether to retrain. Comparing the proportion of rows used for training versus those present in the table is a simple heuristic.

The query above could be done by selecting a random proportion of entries and checking bucket distribution.