SQLite Forum

Crash in FTS5 Module
Login

Crash in FTS5 Module

(1) By AIxCC Reporting (aixcc_reporting) on 2024-08-06 21:33:29 [link] [source]

While running the AIxCC competition, we believe a bug was found in the fts5 module. This bug is present in the latest check-in of SQLite as of July 31, 2024.

Here are two potential ways to trigger this bug, which will crash on a null pointer dereference:

CREATE VIRTUAL TABLE t USING fts5(s, tokenize='trigram case_sensitive ');
CREATE VIRTUAL TABLE t USING fts5(s, tokenize='trigram remove_diacritics ');

We believe the problem is that the trigram tokenizer in fts5 expects that the case_sensitive and remove_diacritics options will have a "1" or "0" argument after them to denote their value, but does not verify that that value actually exists before attempting to access it. A potential inspired by the unicode61 and ascii tokenizers, is to check that a multiple of two arguments exist at all times:

if ( nArg%2 ){
	rc = SQLITE_ERROR;
}else{
// ... All the logic
}

(2) By jose isaias cabrera (jicman) on 2024-08-07 13:25:28 in reply to 1 [link] [source]

These two commands also crashed the Windows sqlite3.exe tool. Just FYI.

(3) By SeverKetor on 2024-08-08 02:05:21 in reply to 1 [source]

It appears to have been fixed on trunk by DRH now. Running either example statement now results in "Runtime error: error in tokenizer constructor"

(4) By Simon Willison (simonw) on 2024-08-14 16:34:17 in reply to 3 [link] [source]

The fix is out in SQLite 3.46.1: https://sqlite.org/releaselog/3_46_1.html

Improved robustness while parsing the tokenize= arguments in FTS5. Forum post 171bcc2bcd.