SQLite Forum

sqlite3_bind_text16 & weird AI
Login
Every SQLite database has a single encoding (UTF8, UTF16LE, or UTF16BE)
in which it stores *all* text.  If your database is a UTF8 database, then
regardless of what the encoding is when you pass it in, that encoding
will be converted into UTF8.  If you want to store UTF16LE in a UTF8
database, you'll need to store your UTF16LE text as a BLOB, not as TEXT.

If you create your database as a UTF16LE database, then it will store your
UTF16LE text as UTF16LE.  It will still strip off the Byte Order Mark (BOM).
But it won't do any other conversion.  If you have a UTF16LE database, then
all text will be stored as UTF16LE in that database, including any text that
you try to insert as UTF8 and UTF16BE.  All text is converted into
UTF16LE for storage.

## Why is the BOM removed?

Your original post asked why the BOM was being removed.  The logic to do
that BOM removal was added back in 2004 and hasn't been touched much since
then, as it has just worked.  So we've been scratching our heads trying to
remember out *why* we put it there.
Your most recent post might have hit upon the answer: because if we
have to convert the text to UTF8, we don't want the BOM to persist.