SQLite Forum

ext/fts5/fts5_tokenize: not handle tokens that contain embedded nul characters
Login
Thanks for reporting this. A unicode61 tokenizer configured to treat unicode "control-characters" (class Cc), was treating embedded nul characters as tokens. Which causes all manner of problems. Now fixed here:

[](https://sqlite.org/src/info/b7b7bde9b7a03665)