SQLite Forum

bug report: table data display error when importing text file encoding utf-16le, utf-16be using .import command
Login

bug report: table data display error when importing text file encoding utf-16le, utf-16be using .import command

(1) By anonymous on 2020-09-12 11:19:46

Hi, I downloaded sqlite 3.33.0 recently.

my operating system is Simplified Chinese WinXP + 203 hotfixes.<br>
<br>
in sqlite 3.33.0, I inserted some Simplified Chinese Characters into table,<br>
but Simplified Chinese Character displayed error.<br>

do like below

C:\>sqlite3.exe<br>
sqlite> pragma encoding='utf16le';<br>
sqlite> create table test (name, age integer);<br>
sqlite> insert into test values ('tom', 10);<br>
sqlite> insert into test values ('jerry', 20);<br>
sqlite> insert into test values ('张三', 30);<br>
sqlite> insert into test values ('李四', 40);<br>
sqlite> select * from test;<br>
tom|10<br>
jerry|20<br>
张三|30&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;; Chinese displayed correctly.<br>
李四|40&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;; Chinese displayed correctly.<br>

now, suppose text file utf16le.txt, encoding is utf-16le without/with BOM, its contents like:

tom&#124;10<br>
jerry&#124;20<br>
张三&#124;30<br>
李四&#124;40<br>

line delimiter is \\n.

<br>
do like below:

C:\>sqlite3.exe<br>
sqlite> pragma encoding='utf16le';<br>
sqlite> create table test2 (name, age integer);<br>
sqlite> .import utf16le.txt test2&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;;import utf16le.txt to test2.<br>
sqlite> select * from test2;<br>
t|&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;;english displayed error, bug.<br>
|&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;;english displayed error, bug.<br>
|&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;;Chinese displayed error, bug.<br>
|&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;;Chinese displayed error, bug.<br>
<br>
it has same bug that importing UTF-16be text file to UTF-16be table.<br>
<br>
this bug found in sqlite 3.33.0 and earlier version.<br>

(2) By Stephan Beal (stephan) on 2020-09-12 12:26:31 in reply to 1 [link]

**@an admin user: this post can be deleted.** The OP tried to edit it while it was pending moderation, resulting in [a separate post being created](/forumpost/6ae4947d26) and i didn't notice it until after approving this one.