SQLite Forum

version 3.37.0 has 2 OOM bugs in sqlite3StrAccumEnlarge and sqlite3BtreeIndexMoveto, could cause sqlite3 DOS.
Login

version 3.37.0 has 2 OOM bugs in sqlite3StrAccumEnlarge and sqlite3BtreeIndexMoveto, could cause sqlite3 DOS.

(1) By spwpun (leeping) on 2021-11-12 15:48:18 [link] [source]

sqlite3 clone from git latest version, and build with -fsanitize=address -fno-omit-frame-pointer, then run sqlite3 without args, just type .read 0.sql/1.sql

0.sql:

CREATE TABLE v0 ( v1 INTEGER PRIMARY KEY DESC ON CONFLICT REPLACE CHECK( max ( 'Al' , v1 ) + julianday ( ( v1 IN ( v1 , v1 , v1 ) ) ) <= 'x' ) ) ; INSERT INTO v0 ( v1 ) VALUES ( 0 ) ,( 42 ) ,( 10 ) ; INSERT INTO v0 SELECT randomblob ( 1 ) FROM v0 , v0 AS t JOIN v0 ; SELECT DISTINCT v1 AS REVENUE FROM v0 AS r WHERE v1 = v1 AND v1 = 10 AND v1 = 0 ORDER BY v1 ; UPDATE v0 SET v1 = ( SELECT group_concat ( substr ( v0 . v1 , 'GERMANY' ) , v1 ) FROM v0 AS p ) ;

stack info:

==5469==AddressSanitizer CHECK failed: ../../../../src/libsanitizer/sanitizer_common/sanitizer_common.cc:118 "((0 && "unable to mmap")) != (0)" (0x0, 0x0)
    #0 0x7ffff6f01bf2  (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xe9bf2)
    #1 0x7ffff6f20575 in __sanitizer::CheckFailed(char const*, int, char const*, unsigned long long, unsigned long long) (/usr/lib/x86_64-linux-gnu/libasan.so.4+0x108575)
    #2 0x7ffff6f0b482  (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xf3482)
    #3 0x7ffff6f17895  (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xff895)
    #4 0x7ffff6e407fd  (/usr/lib/x86_64-linux-gnu/libasan.so.4+0x287fd)
    #5 0x7ffff6ef6b0a in __interceptor_malloc (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xdeb0a)
    #6 0x555555882029 in sqlite3MemMalloc /home/spwpun/sqlite/build/sqlite3.c:24644
    #7 0x55555571de3b in mallocWithAlarm /home/spwpun/sqlite/build/sqlite3.c:28337
    #8 0x55555571de3b in sqlite3Malloc /home/spwpun/sqlite/build/sqlite3.c:28367
    #9 0x555555893dbd in sqlite3BtreeIndexMoveto /home/spwpun/sqlite/build/sqlite3.c:71306
    #10 0x5555559e8beb in sqlite3BtreeInsert /home/spwpun/sqlite/build/sqlite3.c:74483
    #11 0x555555b99547 in sqlite3VdbeExec /home/spwpun/sqlite/build/sqlite3.c:92683
    #12 0x555555bfc31e in sqlite3Step /home/spwpun/sqlite/build/sqlite3.c:85121
    #13 0x555555bfc31e in sqlite3_step /home/spwpun/sqlite/build/sqlite3.c:85178
    #14 0x5555556959d6 in exec_prepared_stmt /home/spwpun/sqlite/build/shell.c:14380
    #15 0x5555556959d6 in shell_exec /home/spwpun/sqlite/build/shell.c:14689
    #16 0x55555569f284 in runOneSqlLine /home/spwpun/sqlite/build/shell.c:21884
    #17 0x5555556ddf69 in process_input /home/spwpun/sqlite/build/shell.c:21994
    #18 0x5555556cee3b in do_meta_command /home/spwpun/sqlite/build/shell.c:20385
    #19 0x5555556dd7bb in process_input /home/spwpun/sqlite/build/shell.c:21963
    #20 0x5555555d7546 in main /home/spwpun/sqlite/build/shell.c:22822
    #21 0x7ffff5e21bf6 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21bf6)
    #22 0x5555555dcf59 in _start (/home/spwpun/sqlite/build/sqlite3+0x88f59)

[Inferior 1 (process 5469) exited with code 01]

1.sql:

CREATE TABLE v0 ( v1 INTEGER PRIMARY KEY DESC ON CONFLICT REPLACE CHECK( max ( 'y' , v1 ) + julianday ( ( v1 IN ( v1 , v1 , v1 ) ) ) <= '13' ) ) ; INSERT INTO v0 ( v1 ) VALUES ( 99 ) ,( 3 ) ,( 10 ) ; INSERT INTO v0 SELECT randomblob ( 10 ) FROM v0 , v0 AS t JOIN v0 ; SELECT DISTINCT v1 AS REVENUE FROM v0 AS r ORDER BY v1 NULLS LAST ; UPDATE v0 SET v1 = ( SELECT group_concat ( substr ( v0 . v1 , 'MED PACK' ) , v1 ) FROM v0 AS p ) ;

stack info:

==5476==AddressSanitizer CHECK failed: ../../../../src/libsanitizer/sanitizer_common/sanitizer_common.cc:118 "((0 && "unable to mmap")) != (0)" (0x0, 0x0)
    #0 0x7ffff6f01bf2  (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xe9bf2)
    #1 0x7ffff6f20575 in __sanitizer::CheckFailed(char const*, int, char const*, unsigned long long, unsigned long long) (/usr/lib/x86_64-linux-gnu/libasan.so.4+0x108575)
    #2 0x7ffff6f0b482  (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xf3482)
    #3 0x7ffff6f17895  (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xff895)
    #4 0x7ffff6e448f1  (/usr/lib/x86_64-linux-gnu/libasan.so.4+0x2c8f1)
    #5 0x7ffff6e3f23d  (/usr/lib/x86_64-linux-gnu/libasan.so.4+0x2723d)
    #6 0x7ffff6ef6ef7 in realloc (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xdeef7)
    #7 0x555555881f92 in sqlite3MemRealloc /home/spwpun/sqlite/build/sqlite3.c:24714
    #8 0x555555721e82 in sqlite3Realloc /home/spwpun/sqlite/build/sqlite3.c:28567
    #9 0x555555762e2a in sqlite3StrAccumEnlarge /home/spwpun/sqlite/build/sqlite3.c:29858
    #10 0x555555763929 in sqlite3StrAccumEnlarge /home/spwpun/sqlite/build/sqlite3.c:29900
    #11 0x555555763929 in enlargeAndAppend /home/spwpun/sqlite/build/sqlite3.c:29895
    #12 0x55555595dcae in groupConcatStep /home/spwpun/sqlite/build/sqlite3.c:121883
    #13 0x555555bc0988 in sqlite3VdbeExec /home/spwpun/sqlite/build/sqlite3.c:93906
    #14 0x555555bfc31e in sqlite3Step /home/spwpun/sqlite/build/sqlite3.c:85121
    #15 0x555555bfc31e in sqlite3_step /home/spwpun/sqlite/build/sqlite3.c:85178
    #16 0x5555556959d6 in exec_prepared_stmt /home/spwpun/sqlite/build/shell.c:14380
    #17 0x5555556959d6 in shell_exec /home/spwpun/sqlite/build/shell.c:14689
    #18 0x55555569f284 in runOneSqlLine /home/spwpun/sqlite/build/shell.c:21884
    #19 0x5555556ddf69 in process_input /home/spwpun/sqlite/build/shell.c:21994
    #20 0x5555556cee3b in do_meta_command /home/spwpun/sqlite/build/shell.c:20385
    #21 0x5555556dd7bb in process_input /home/spwpun/sqlite/build/shell.c:21963
    #22 0x5555555d7546 in main /home/spwpun/sqlite/build/shell.c:22822
    #23 0x7ffff5e21bf6 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21bf6)
    #24 0x5555555dcf59 in _start (/home/spwpun/sqlite/build/sqlite3+0x88f59)

[Inferior 1 (process 5476) exited with code 01]

(2) By Richard Hipp (drh) on 2021-11-12 17:33:22 in reply to 1 [link] [source]

The error appears to be inside of ASAN, probably at https://github.com/gcc-mirror/gcc/blob/master/libsanitizer/sanitizer_common/sanitizer_common.cpp#L53, not something in SQLite. It appears that you are running out of memory on your system, and ASAN is choking because of that.

What does the command "prlimit" show on your system? In particular, what is your "address space limit"? Mine shows "unlimited", and when I run your scripts I get:

Error: near line 5: stepping, string or blob too big (18)

I modified your first scripts by adding ".stats on" and ".echo" giving the following:

.echo on
CREATE TABLE v0 ( v1 INTEGER PRIMARY KEY DESC ON CONFLICT REPLACE CHECK( max ( 'Al' , v1 ) + julianday ( ( v1 IN ( v1 , v1 , v1 ) ) ) <= 'x' ) ) ; INSERT INTO v0 ( v1 ) VALUES ( 0 ) ,( 42 ) ,( 10 ) ;
INSERT INTO v0 SELECT randomblob ( 1 ) FROM v0 , v0 AS t JOIN v0 ; SELECT DISTINCT v1 AS REVENUE FROM v0 AS r WHERE v1 = v1 AND v1 = 10 AND v1 = 0 ORDER BY v1 ;
.stats on
UPDATE v0 SET v1 = ( SELECT group_concat ( substr ( v0 . v1 , 'GERMANY' ) , v1 ) FROM v0 AS p ) ;

I compile thusly:

CC=clang CFLAGS='-fsanitize=address -fno-omit-frame-pointer -Os -g' ./configure --enable-all --enable-debug
make sqlite3

Then running your script, I get the error above, and the ".stat" output shows that the process used a bit of 9GB of RAM at its peak, before giving up and reporting the error. My desktop has 32GB of RAM. How much RAM is on your test system?

It seems to me that SQLite is doing exactly the right thing here, and that the problem is in your test procedure or in your copy of libasan.so. Why do you think this is a problem with SQLite? What do you think it should be doing differently?

(3) By spwpun (leeping) on 2021-11-13 03:12:11 in reply to 2 [source]

Thanks for quick reply!

Firstly, I checked my test system prlimit, the AS is unlimited:

spwpun@ubuntu:~/sqlite/build$ prlimit
RESOURCE   DESCRIPTION                             SOFT      HARD UNITS
AS         address space limit                unlimited unlimited bytes
CORE       max core file size                         0 unlimited bytes
CPU        CPU time                           unlimited unlimited seconds
DATA       max data size                      unlimited unlimited bytes
FSIZE      max file size                      unlimited unlimited bytes
LOCKS      max number of file locks held      unlimited unlimited locks
MEMLOCK    max locked-in-memory address space  16777216  16777216 bytes
MSGQUEUE   max bytes in POSIX mqueues            819200    819200 bytes
NICE       max nice prio allowed to raise             0         0 
NOFILE     max number of open files                1024      4096 files
NPROC      max number of processes                31544     31544 processes
RSS        max resident set size              unlimited unlimited bytes
RTPRIO     max real-time priority                     0         0 
RTTIME     timeout for real-time tasks        unlimited unlimited microsecs
SIGPENDING max number of pending signals          31544     31544 signals
STACK      max stack size                       8388608 unlimited bytes

my server's RAM is 2GB, that's unchangeable. Then, I test it on Ubuntu 18.04 in VMware with RAM 10GB, then it reported the error like yours, so frustrated for my server!

(4) By Richard Hipp (drh) on 2021-11-15 16:39:28 in reply to 1 [link] [source]

Here is a link to the OP's tweet about this topic:

https://twitter.com/den80369019/status/1460277629668323334