version 3.37.0 has 2 OOM bugs in sqlite3StrAccumEnlarge and sqlite3BtreeIndexMoveto, could cause sqlite3 DOS.
(1) By spwpun (leeping) on 2021-11-12 15:48:18 [link] [source]
sqlite3 clone from git latest version, and build with -fsanitize=address -fno-omit-frame-pointer
, then run sqlite3 without args, just type .read 0.sql/1.sql
0.sql:
CREATE TABLE v0 ( v1 INTEGER PRIMARY KEY DESC ON CONFLICT REPLACE CHECK( max ( 'Al' , v1 ) + julianday ( ( v1 IN ( v1 , v1 , v1 ) ) ) <= 'x' ) ) ; INSERT INTO v0 ( v1 ) VALUES ( 0 ) ,( 42 ) ,( 10 ) ; INSERT INTO v0 SELECT randomblob ( 1 ) FROM v0 , v0 AS t JOIN v0 ; SELECT DISTINCT v1 AS REVENUE FROM v0 AS r WHERE v1 = v1 AND v1 = 10 AND v1 = 0 ORDER BY v1 ; UPDATE v0 SET v1 = ( SELECT group_concat ( substr ( v0 . v1 , 'GERMANY' ) , v1 ) FROM v0 AS p ) ;
stack info:
==5469==AddressSanitizer CHECK failed: ../../../../src/libsanitizer/sanitizer_common/sanitizer_common.cc:118 "((0 && "unable to mmap")) != (0)" (0x0, 0x0)
#0 0x7ffff6f01bf2 (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xe9bf2)
#1 0x7ffff6f20575 in __sanitizer::CheckFailed(char const*, int, char const*, unsigned long long, unsigned long long) (/usr/lib/x86_64-linux-gnu/libasan.so.4+0x108575)
#2 0x7ffff6f0b482 (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xf3482)
#3 0x7ffff6f17895 (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xff895)
#4 0x7ffff6e407fd (/usr/lib/x86_64-linux-gnu/libasan.so.4+0x287fd)
#5 0x7ffff6ef6b0a in __interceptor_malloc (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xdeb0a)
#6 0x555555882029 in sqlite3MemMalloc /home/spwpun/sqlite/build/sqlite3.c:24644
#7 0x55555571de3b in mallocWithAlarm /home/spwpun/sqlite/build/sqlite3.c:28337
#8 0x55555571de3b in sqlite3Malloc /home/spwpun/sqlite/build/sqlite3.c:28367
#9 0x555555893dbd in sqlite3BtreeIndexMoveto /home/spwpun/sqlite/build/sqlite3.c:71306
#10 0x5555559e8beb in sqlite3BtreeInsert /home/spwpun/sqlite/build/sqlite3.c:74483
#11 0x555555b99547 in sqlite3VdbeExec /home/spwpun/sqlite/build/sqlite3.c:92683
#12 0x555555bfc31e in sqlite3Step /home/spwpun/sqlite/build/sqlite3.c:85121
#13 0x555555bfc31e in sqlite3_step /home/spwpun/sqlite/build/sqlite3.c:85178
#14 0x5555556959d6 in exec_prepared_stmt /home/spwpun/sqlite/build/shell.c:14380
#15 0x5555556959d6 in shell_exec /home/spwpun/sqlite/build/shell.c:14689
#16 0x55555569f284 in runOneSqlLine /home/spwpun/sqlite/build/shell.c:21884
#17 0x5555556ddf69 in process_input /home/spwpun/sqlite/build/shell.c:21994
#18 0x5555556cee3b in do_meta_command /home/spwpun/sqlite/build/shell.c:20385
#19 0x5555556dd7bb in process_input /home/spwpun/sqlite/build/shell.c:21963
#20 0x5555555d7546 in main /home/spwpun/sqlite/build/shell.c:22822
#21 0x7ffff5e21bf6 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21bf6)
#22 0x5555555dcf59 in _start (/home/spwpun/sqlite/build/sqlite3+0x88f59)
[Inferior 1 (process 5469) exited with code 01]
1.sql:
CREATE TABLE v0 ( v1 INTEGER PRIMARY KEY DESC ON CONFLICT REPLACE CHECK( max ( 'y' , v1 ) + julianday ( ( v1 IN ( v1 , v1 , v1 ) ) ) <= '13' ) ) ; INSERT INTO v0 ( v1 ) VALUES ( 99 ) ,( 3 ) ,( 10 ) ; INSERT INTO v0 SELECT randomblob ( 10 ) FROM v0 , v0 AS t JOIN v0 ; SELECT DISTINCT v1 AS REVENUE FROM v0 AS r ORDER BY v1 NULLS LAST ; UPDATE v0 SET v1 = ( SELECT group_concat ( substr ( v0 . v1 , 'MED PACK' ) , v1 ) FROM v0 AS p ) ;
stack info:
==5476==AddressSanitizer CHECK failed: ../../../../src/libsanitizer/sanitizer_common/sanitizer_common.cc:118 "((0 && "unable to mmap")) != (0)" (0x0, 0x0)
#0 0x7ffff6f01bf2 (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xe9bf2)
#1 0x7ffff6f20575 in __sanitizer::CheckFailed(char const*, int, char const*, unsigned long long, unsigned long long) (/usr/lib/x86_64-linux-gnu/libasan.so.4+0x108575)
#2 0x7ffff6f0b482 (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xf3482)
#3 0x7ffff6f17895 (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xff895)
#4 0x7ffff6e448f1 (/usr/lib/x86_64-linux-gnu/libasan.so.4+0x2c8f1)
#5 0x7ffff6e3f23d (/usr/lib/x86_64-linux-gnu/libasan.so.4+0x2723d)
#6 0x7ffff6ef6ef7 in realloc (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xdeef7)
#7 0x555555881f92 in sqlite3MemRealloc /home/spwpun/sqlite/build/sqlite3.c:24714
#8 0x555555721e82 in sqlite3Realloc /home/spwpun/sqlite/build/sqlite3.c:28567
#9 0x555555762e2a in sqlite3StrAccumEnlarge /home/spwpun/sqlite/build/sqlite3.c:29858
#10 0x555555763929 in sqlite3StrAccumEnlarge /home/spwpun/sqlite/build/sqlite3.c:29900
#11 0x555555763929 in enlargeAndAppend /home/spwpun/sqlite/build/sqlite3.c:29895
#12 0x55555595dcae in groupConcatStep /home/spwpun/sqlite/build/sqlite3.c:121883
#13 0x555555bc0988 in sqlite3VdbeExec /home/spwpun/sqlite/build/sqlite3.c:93906
#14 0x555555bfc31e in sqlite3Step /home/spwpun/sqlite/build/sqlite3.c:85121
#15 0x555555bfc31e in sqlite3_step /home/spwpun/sqlite/build/sqlite3.c:85178
#16 0x5555556959d6 in exec_prepared_stmt /home/spwpun/sqlite/build/shell.c:14380
#17 0x5555556959d6 in shell_exec /home/spwpun/sqlite/build/shell.c:14689
#18 0x55555569f284 in runOneSqlLine /home/spwpun/sqlite/build/shell.c:21884
#19 0x5555556ddf69 in process_input /home/spwpun/sqlite/build/shell.c:21994
#20 0x5555556cee3b in do_meta_command /home/spwpun/sqlite/build/shell.c:20385
#21 0x5555556dd7bb in process_input /home/spwpun/sqlite/build/shell.c:21963
#22 0x5555555d7546 in main /home/spwpun/sqlite/build/shell.c:22822
#23 0x7ffff5e21bf6 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21bf6)
#24 0x5555555dcf59 in _start (/home/spwpun/sqlite/build/sqlite3+0x88f59)
[Inferior 1 (process 5476) exited with code 01]
(2) By Richard Hipp (drh) on 2021-11-12 17:33:22 in reply to 1 [link] [source]
The error appears to be inside of ASAN, probably at https://github.com/gcc-mirror/gcc/blob/master/libsanitizer/sanitizer_common/sanitizer_common.cpp#L53, not something in SQLite. It appears that you are running out of memory on your system, and ASAN is choking because of that.
What does the command "prlimit" show on your system? In particular, what is your "address space limit"? Mine shows "unlimited", and when I run your scripts I get:
Error: near line 5: stepping, string or blob too big (18)
I modified your first scripts by adding ".stats on" and ".echo" giving the following:
.echo on CREATE TABLE v0 ( v1 INTEGER PRIMARY KEY DESC ON CONFLICT REPLACE CHECK( max ( 'Al' , v1 ) + julianday ( ( v1 IN ( v1 , v1 , v1 ) ) ) <= 'x' ) ) ; INSERT INTO v0 ( v1 ) VALUES ( 0 ) ,( 42 ) ,( 10 ) ; INSERT INTO v0 SELECT randomblob ( 1 ) FROM v0 , v0 AS t JOIN v0 ; SELECT DISTINCT v1 AS REVENUE FROM v0 AS r WHERE v1 = v1 AND v1 = 10 AND v1 = 0 ORDER BY v1 ; .stats on UPDATE v0 SET v1 = ( SELECT group_concat ( substr ( v0 . v1 , 'GERMANY' ) , v1 ) FROM v0 AS p ) ;
I compile thusly:
CC=clang CFLAGS='-fsanitize=address -fno-omit-frame-pointer -Os -g' ./configure --enable-all --enable-debug make sqlite3
Then running your script, I get the error above, and the ".stat" output shows that the process used a bit of 9GB of RAM at its peak, before giving up and reporting the error. My desktop has 32GB of RAM. How much RAM is on your test system?
It seems to me that SQLite is doing exactly the right thing here, and that the problem is in your test procedure or in your copy of libasan.so. Why do you think this is a problem with SQLite? What do you think it should be doing differently?
(3) By spwpun (leeping) on 2021-11-13 03:12:11 in reply to 2 [link] [source]
Thanks for quick reply!
Firstly, I checked my test system prlimit
, the AS is unlimited:
spwpun@ubuntu:~/sqlite/build$ prlimit RESOURCE DESCRIPTION SOFT HARD UNITS AS address space limit unlimited unlimited bytes CORE max core file size 0 unlimited bytes CPU CPU time unlimited unlimited seconds DATA max data size unlimited unlimited bytes FSIZE max file size unlimited unlimited bytes LOCKS max number of file locks held unlimited unlimited locks MEMLOCK max locked-in-memory address space 16777216 16777216 bytes MSGQUEUE max bytes in POSIX mqueues 819200 819200 bytes NICE max nice prio allowed to raise 0 0 NOFILE max number of open files 1024 4096 files NPROC max number of processes 31544 31544 processes RSS max resident set size unlimited unlimited bytes RTPRIO max real-time priority 0 0 RTTIME timeout for real-time tasks unlimited unlimited microsecs SIGPENDING max number of pending signals 31544 31544 signals STACK max stack size 8388608 unlimited bytes
my server's RAM is 2GB, that's unchangeable. Then, I test it on Ubuntu 18.04 in VMware with RAM 10GB, then it reported the error like yours, so frustrated for my server!
(4) By Richard Hipp (drh) on 2021-11-15 16:39:28 in reply to 1 [source]
Here is a link to the OP's tweet about this topic: