While running the AIxCC competition, we believe a bug was found in the fts5 module. This bug is present in the latest check-in of SQLite as of July 31, 2024.

Here are two potential ways to trigger this bug, which will crash on a null pointer dereference:

CREATE VIRTUAL TABLE t USING fts5(s, tokenize='trigram case_sensitive ');
CREATE VIRTUAL TABLE t USING fts5(s, tokenize='trigram remove_diacritics ');

We believe the problem is that the trigram tokenizer in fts5 expects that the case_sensitive and remove_diacritics options will have a "1" or "0" argument after them to denote their value, but does not verify that that value actually exists before attempting to access it. A potential inspired by the unicode61 and ascii tokenizers, is to check that a multiple of two arguments exist at all times:

if ( nArg%2 ){
	rc = SQLITE_ERROR;
}else{
// ... All the logic
}

These two commands also crashed the Windows sqlite3.exe tool. Just FYI.

The fix is out in SQLite 3.46.1: https://sqlite.org/releaselog/3_46_1.html

Improved robustness while parsing the tokenize= arguments in FTS5. Forum post 171bcc2bcd.