SQLite

Consider the following test case:

PRAGMA encoding = 'UTF-16';
CREATE VIRTUAL TABLE vt0 USING fts4(c0);
INSERT INTO vt0 VALUES ('');
INSERT INTO vt0(vt0) VALUES('integrity-check'); -- database disk image is malformed

Unexpectedly, the integrity check indicates an error. It seems that FTS5 is not affected.

The INSERT in the script above is equivalent to this:

INSERT INTO vt0 VALUES (CAST(X'FEFF' AS TEXT));

In utf-16, the sequence 0xFE 0xFF is a BOM - a byte-order-mark. SQLite leaves this byte sequence unchanged when it converts the text value to utf-16 to pass to FTS4. Then FTS4 tokenizes it. But, when FTS4 binds the value to a statement to insert into the %_content table, SQLite strips the BOM away before writing the value to disk. Then, when the integrity check is run, the contents of the FTS index are slightly different to the index that would be produced by rebuilding based on the contents of the %_content table. Hence the error.

The unicode61 tokenizer handles this, as it ignores the BOM. But the default tokenizer ("simple") fails as above.

I noticed that this bug report is still open. Will this issue still be addressed?

This is an obscure corner case in an obsolete extension that is unlikely to ever be encountered in practice, and a fix is very difficult. So rather than fix it, we merely document the issue and ask users to avoid the unusual set of circumstances that can bring it to pass. And the circumstances that lead to this problem are quite ununsual and so we do not believe anybody has ever hit them before or is likely to hit them in the furture. (Unless they are running a fuzzer. ;-))