SQLite Forum

Random crash doing SELECT

Random crash doing SELECT

(1.2) By zero (zeroxia) on 2021-07-22 05:58:29 edited from 1.1 [source]


My program uses 1 write connection, 3 read connections. SQLite is open as "SQLITE_OPEN_NOMUTEX".

When a thread needs to access the SQLite, it needs to get a connection by waiting on a condition variable. So effectively once a thread waited successfully, it exclusively owns that connection object.

If a thread wants the write connection, it needs to wait until no read connection is busy, and once write connection is busy, no read connection can be retrieved.

It's been running a while on MacOS, Windows and QNX without problem.

Yesterday I run the program in linux docker containers, some mysterious crash happened.

One is centos 7 container:

statement is:

"SELECT \"tile_blob\" FROM \"tiles_table\"\n WHERE \"tile_meta\" = @tile_meta AND \"tile_id\" = @tile_id;"

Call stack:

Program terminated with signal 11, Segmentation fault.
#0  0x0000000000806ad2 in sqlite3VdbeAddOp3 (p=0x7f84d00279e8, op=96, p1=0, p2=4, p3=0) at /home/jenkins/agent/workspace/nav/thirdparty/develop/linux-centos7x64/thirdparty/sqlite/sqlite-amalgamation/sqlite3.c:77795
77795   /home/jenkins/agent/workspace/nav/thirdparty/develop/linux-centos7x64/thirdparty/sqlite/sqlite-amalgamation/sqlite3.c: No such file or directory.
warning: File "/usr/local/lib64/libstdc++.so.6.0.22-gdb.py" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load:/usr/bin/mono-gdb.py".
To enable execution of this file add
        add-auto-load-safe-path /usr/local/lib64/libstdc++.so.6.0.22-gdb.py
line to your configuration file "/root/.gdbinit".
To completely disable this security protection add
        set auto-load safe-path /
line to your configuration file "/root/.gdbinit".
For more information about this security protection see the
"Auto-loading safe path" section in the GDB manual.  E.g., run from the shell:
        info "(gdb)Auto-loading safe path"
Missing separate debuginfos, use: debuginfo-install glibc-2.17-292.el7.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.15.1-37.el7_7.2.x86_64 libcom_err-1.42.9-16.el7.x86_64 libselinux-2.5-14.1.el7.x86_64 openssl-libs-1.0.2k-19.el7.x86_64 pcre-8.32-17.el7.x86_64 zlib-1.2.7-18.el7.x86_64
(gdb) bt
#0  0x0000000000806ad2 in sqlite3VdbeAddOp3 (p=0x7f84d00279e8, op=96, p1=0, p2=4, p3=0) at /home/jenkins/agent/workspace/nav/thirdparty/develop/linux-centos7x64/thirdparty/sqlite/sqlite-amalgamation/sqlite3.c:77795
#1  0x000000000080728f in sqlite3VdbeAddOp4Int (p=0x7f84d00279e8, op=96, p1=0, p2=4, p3=0, p4=5) at /home/jenkins/agent/workspace/nav/thirdparty/develop/linux-centos7x64/thirdparty/sqlite/sqlite-amalgamation/sqlite3.c:78040
#2  0x0000000000846b94 in sqlite3OpenTable (pParse=0x7ffd1e0d6ab0, iCur=0, iDb=0, pTab=0x7f84dc016778, opcode=96) at /home/jenkins/agent/workspace/nav/thirdparty/develop/linux-centos7x64/thirdparty/sqlite/sqlite-amalgamation/sqlite3.c:120293
#3  0x000000000087877c in sqlite3WhereBegin (pParse=0x7ffd1e0d6ab0, pTabList=0x7f84d002afc8, pWhere=0x7f84d002ad48, pOrderBy=0x0, pResultSet=0x7f84d002aac8, wctrlFlags=0, iAuxArg=320)
    at /home/jenkins/agent/workspace/nav/thirdparty/develop/linux-centos7x64/thirdparty/sqlite/sqlite-amalgamation/sqlite3.c:150096
#4  0x000000000085e8f3 in sqlite3Select (pParse=0x7ffd1e0d6ab0, p=0x7f84d002af48, pDest=0x7ffd1e0d5e70) at /home/jenkins/agent/workspace/nav/thirdparty/develop/linux-centos7x64/thirdparty/sqlite/sqlite-amalgamation/sqlite3.c:135208
#5  0x00000000008803fd in yy_reduce (yypParser=0x7ffd1e0d60c0, yyruleno=82, yyLookahead=1, yyLookaheadToken=..., pParse=0x7ffd1e0d6ab0) at /home/jenkins/agent/workspace/nav/thirdparty/develop/linux-centos7x64/thirdparty/sqlite/sqlite-amalgamation/sqlite3.c:157152
#6  0x0000000000884586 in sqlite3Parser (yyp=0x7ffd1e0d60c0, yymajor=1, yyminor=...) at /home/jenkins/agent/workspace/nav/thirdparty/develop/linux-centos7x64/thirdparty/sqlite/sqlite-amalgamation/sqlite3.c:158453
#7  0x000000000088576b in sqlite3RunParser (pParse=0x7ffd1e0d6ab0, zSql=0x149f253 ";", pzErrMsg=0x7ffd1e0d6c58) at /home/jenkins/agent/workspace/nav/thirdparty/develop/linux-centos7x64/thirdparty/sqlite/sqlite-amalgamation/sqlite3.c:159732
#8  0x00000000008531e8 in sqlite3Prepare (db=0x7f84d001d0c8, zSql=0x149f1f0 "SELECT \"tile_blob\" FROM \"tiles_hd_8590\"\n    WHERE \"tile_meta\" = @tile_meta AND \"tile_id\" = @tile_id;", nBytes=101, prepFlags=128, pReprepare=0x0, ppStmt=0x7ffd1e0d6ea8, pzTail=0x0)
    at /home/jenkins/agent/workspace/nav/thirdparty/develop/linux-centos7x64/thirdparty/sqlite/sqlite-amalgamation/sqlite3.c:128665
#9  0x0000000000853410 in sqlite3LockAndPrepare (db=0x7f84d001d0c8, zSql=0x149f1f0 "SELECT \"tile_blob\" FROM \"tiles_hd_8590\"\n    WHERE \"tile_meta\" = @tile_meta AND \"tile_id\" = @tile_id;", nBytes=101, prepFlags=128, pOld=0x0, ppStmt=0x7ffd1e0d6ea8, pzTail=0x0)
    at /home/jenkins/agent/workspace/nav/thirdparty/develop/linux-centos7x64/thirdparty/sqlite/sqlite-amalgamation/sqlite3.c:128737
#10 0x00000000008535e6 in sqlite3_prepare_v2 (db=0x7f84d001d0c8, zSql=0x149f1f0 "SELECT \"tile_blob\" FROM \"tiles_hd_8590\"\n    WHERE \"tile_meta\" = @tile_meta AND \"tile_id\" = @tile_id;", nBytes=101, ppStmt=0x7ffd1e0d6ea8, pzTail=0x0)
    at /home/jenkins/agent/workspace/nav/thirdparty/develop/linux-centos7x64/thirdparty/sqlite/sqlite-amalgamation/sqlite3.c:128821

Another is Ubuntu 1804 container, statement is:

statement is

"SELECT \"tile_blob\" FROM \"tiles_table\"\n WHERE \"tile_meta\" = @tile_meta AND \"tile_id\" = @tile_id;"


Call stack:

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `./ft -ns -tc=... ...'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007eff04f84b03 in sqlite3ApiExit (db=0x40000000000, rc=0) at /home/jenkins/agent/workspace/nav/thirdparty/develop/linux-ubuntu-1804/thirdparty/sqlite/sqlite-amalgamation/sqlite3.c:28090
28090   /home/jenkins/agent/workspace/nav/thirdparty/develop/linux-ubuntu-1804/thirdparty/sqlite/sqlite-amalgamation/sqlite3.c: No such file or directory.
[Current thread is 1 (Thread 0x7eff04b35900 (LWP 47426))]
(gdb) bt
#0  0x00007eff04f84b03 in sqlite3ApiExit (db=0x40000000000, rc=0) at /home/jenkins/agent/workspace/nav/thirdparty/develop/linux-ubuntu-1804/thirdparty/sqlite/sqlite-amalgamation/sqlite3.c:28090
#1  0x00007eff04fbc772 in sqlite3Step (p=0x7efee4026d78) at /home/jenkins/agent/workspace/nav/thirdparty/develop/linux-ubuntu-1804/thirdparty/sqlite/sqlite-amalgamation/sqlite3.c:83508
#2  0x00007eff04fbc913 in sqlite3_step (pStmt=0x7efee4026d78) at /home/jenkins/agent/workspace/nav/thirdparty/develop/linux-ubuntu-1804/thirdparty/sqlite/sqlite-amalgamation/sqlite3.c:83553

The "sqlite-amalgamation" used to build SQLite is:

#define SQLITE_VERSION        "3.32.3"
#define SQLITE_SOURCE_ID      "2020-06-18 14:00:33 7ebdfa80be8e8e73324b8d66b3460222eb74c7e9dfd655b48d6ca7e1933cc8fd"

Is there any suggestion to help me to pinpoint the problem? Thanks.

(2) By jose isaias cabrera (jicman) on 2021-07-23 15:08:55 in reply to 1.2 [link] [source]

Hi. You haven't provided much information, so, from what I can quickly come up to debug your problem would be to try to print to a text file or the screen output what is the content of:

"SELECT "tile_blob" FROM "tiles_table"n WHERE "tile_meta" = @tile_meta AND "tile_id" = @tile_id;"

Once you get that, use the sqlite3 tool and connect to the DB and try pasting that result on the tool and see if you get a result, or if it also crashes. Other than this, I don't know how else to help you.


(3) By Rico Mariani (rmariani) on 2021-07-23 17:41:25 in reply to 1.2 [link] [source]

It's very unlikely that sqlite3VdbeAddOp3 has a bug like this. I can suggest a few things that might help:

  • first try a variety of experiments with your load on this system but without any of the concurrency going on

  • simplify your concurrency for test purposes so that it's ultra dumb, just one person totally exclusive

  • if that works, expand the concurrency options slowly and look to see where things start to go wrong

The most likely situation is that on this system you aren't getting the concurrency guarantees that you think you're getting or that you want. That will be hard to diagnose and it could cause bizarre failures that are nearly impossible to figure out. Sanity check the stuff and then add slowly for clues as to where things start going wrong.

The likelihood is that whatever is going wrong has very little to do with SQLite per se.