SQLite User Forum

Assertion violation and Heap Buffer Overflow - Grammar-based Fuzzers
Login
Chronological

Assertion violation, Segmentation Fault, and Heap Buffer Overflow - Grammar-based Fuzzers

(1.5) By Maik (maikbe) on 2023-03-16 17:01:49 edited from 1.4 [source]

Hello,

I am a researcher in software testing from the University of Stuttgart, Germany. We are testing grammar-based fuzzers and have chosen SQLite3 as one of our fuzz targets for our experiments. I think we have found four unique issues: two heap buffer overflows, one segmentation fault, and one assertion violation. However, I don't know whether there are more, so I provide for each issue sometimes multiple examples. Please find the test cases below.

Version

$ fossil info
project-name: SQLite
repository:   /home/rocky/sqlite3/sqlite.fossil
local-root:   /home/rocky/sqlite3/
config-db:    /home/rocky/.fossil
project-code: 2ab58778c2967968b94284e989e43dc11791f548
checkout:     8f45ad27403e971d88ec62e674c03f82eb19df0b 2023-03-16 11:50:44 UTC
parent:       2c56b984a0bd3be5ec326a2109ea7b8f1d4ef63c 2023-03-16 10:17:30 UTC
tags:         trunk
comment:      Update the tracing output for the query-invariant checker such that it shows the SQL that is run to
              verify that a found query-invariant discrepency is valid. Changes to testing logic only. (user: drh)
check-ins:    28516

Build

We compiled SQLite3 with AFL++ and ASAN enabled. However, I guess compiling with ASAN and assertions enabled should be sufficient.

# AFL++

cd ~/
git clone git@github.tik.uni-stuttgart.de:ac136848/AFLplusplus.git
cd AFLplusplus/
make all


# SQLite3

CC=~/AFLplusplus/afl-cc \
CXX=~/AFLplusplus/afl-c++ \
AFL_CC_COMPILER=LLVM \
AFL_LLVM_INSTRUMENT=PCGUARD \
./configure

CC=~/AFLplusplus/afl-cc \
CXX=~/AFLplusplus/afl-c++ \
AFL_CC_COMPILER=LLVM \
AFL_LLVM_INSTRUMENT=PCGUARD \
make sqlite3.c

CC=~/AFLplusplus/afl-cc \
CXX=~/AFLplusplus/afl-c++ \
AFL_CC_COMPILER=LLVM \
AFL_LLVM_INSTRUMENT=PCGUARD \
AFL_USE_ASAN=1 \
~/AFLplusplus/afl-cc -O3 \
  -DSQLITE_DEBUG \
  -DSQLITE_MAX_EXPR_DEPTH=0 \
  -DSQLITE_THREADSAFE=0 \
  -DSQLITE_ENABLE_UPDATE_DELETE_LIMIT \
  -DSQLITE_OMIT_LOAD_EXTENSION \
  -DSQLITE_OMIT_JSON \
  -DSQLITE_OMIT_DEPRECATED \
  -I. shell.c sqlite3.c -o sqlite3

Execution

cat test*.sql | ./sqlite3

Issue 1 - Assertion Violation

Input: test1.sql

CREATE  TEMPORARY  TABLE t2(mx_payload CONSTRAINT constr0 COLLATE NOCASE);
CREATE  TRIGGER  IF  NOT  EXISTS trig0 UPDATE  OF mx_payload, mx_payload, mx_payload, mx_payload, mx_payload ON t2 FOR  EACH  ROW  WHEN 0xffff BEGIN  VALUES (''); END ;
CREATE  TEMPORARY  TRIGGER  IF  NOT  EXISTS trig0 AFTER  DELETE  ON t2 FOR  EACH  ROW  WHEN  NULL  BEGIN  INSERT  INTO t2(mx_payload) VALUES (0xffff), ('') ON  CONFLICT  DO  NOTHING  RETURNING  FALSE  AS mx_payload; INSERT  INTO t2(mx_payload) VALUES (''), (42e-300) ON  CONFLICT  DO  NOTHING  RETURNING  TRUE  AS mx_payload; END ;

Output:

sqlite3: sqlite3.c:119119: void sqlite3AddReturning(Parse *, ExprList *): Assertion `pParse->bReturning==0' failed.
Aborted (core dumped)

Input: test2.sql

CREATE  TEMP  TABLE t1(rootpage PRIMARY  KEY  ON  CONFLICT  IGNORE ) WITHOUT  ROWID ;
CREATE  TEMP  TRIGGER trig0 DELETE  ON t1 FOR  EACH  ROW  BEGIN  SELECT t2.* FROM (temp.t1 NOT  INDEXED ), (t1 AS t2 NOT  INDEXED , t1 AS t1 NOT  INDEXED ) GROUP  BY  NULL ,  FALSE ; END ;
CREATE  TRIGGER  IF  NOT  EXISTS temp.trig0 INSTEAD  OF  DELETE  ON t1 FOR  EACH  ROW  WHEN 0.0 BEGIN  INSERT  OR  FAIL  INTO t1 VALUES (0.0) ON  CONFLICT  DO  NOTHING  RETURNING *; INSERT  OR  FAIL  INTO t1 VALUES (0.0) ON  CONFLICT  DO  NOTHING  RETURNING *; SELECT  DISTINCT t1.*; END ;

Output:

sqlite3: sqlite3.c:119119: void sqlite3AddReturning(Parse *, ExprList *): Assertion `pParse->bReturning==0' failed.
Aborted (core dumped)

Issue 2 - Heap Buffer Overflow

Input: test3.sql

CREATE  TABLE  IF  NOT  EXISTS dbstat(pagetypeREAL);
CREATE  TEMPORARY  TABLE  IF  NOT  EXISTS dbstat AS  SELECT  ALL dbstat.*, *, *, *, *, *, dbstat.*, *, *, *, *, *, *, *, *, *, *, *, *, *, '%Y-%m-%d', *, *, *, *, * FROM (dbstat AS dbstat NOT  INDEXED , sqlite_schema AS dbstat NOT  INDEXED ) GROUP  BY  FALSE  HAVING  FALSE  ORDER  BY 5e200 NULLS  FIRST;
ALTER  TABLE dbstat ADD  COLUMN pagetype CONSTRAINT constr1 GENERATED  ALWAYS  AS ( RAISE ( IGNORE )->> FALSE  OR  FALSE );
SELECT * FROM dbstat;

Output:

=================================================================
==9780==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x6140000001d4 at pc 0x0000005998c7 bp 0x7ffdcf9d2e60 sp 0x7ffdcf9d2e58
READ of size 1 at 0x6140000001d4 thread T0
    #0 0x5998c6 in shell_error_context /home/rocky/sqlite3/shell.c:17931:13
    #1 0x5a1fee in save_err_msg /home/rocky/sqlite3/shell.c:18020:14
    #2 0x5608fa in shell_exec /home/rocky/sqlite3/shell.c:19237:21
    #3 0x5aba57 in runOneSqlLine /home/rocky/sqlite3/shell.c:26470:8
    #4 0x56229e in process_input /home/rocky/sqlite3/shell.c:26636:17
    #5 0x53dd8d in main /home/rocky/sqlite3/shell.c:27541:12
    #6 0x7f0c1f997eaf in __libc_start_call_main (/lib64/libc.so.6+0x3feaf) (BuildId: 82f7ae28e16376aa97cc3bf50b40ab2d1043924a)
    #7 0x7f0c1f997f5f in __libc_start_main@GLIBC_2.2.5 (/lib64/libc.so.6+0x3ff5f) (BuildId: 82f7ae28e16376aa97cc3bf50b40ab2d1043924a)
    #8 0x427614 in _start (/home/rocky/sqlite3/sqlite3+0x427614) (BuildId: bee988bfc215b67422a317b2b26ed8fcdadbf751)

0x6140000001d4 is located 0 bytes to the right of 404-byte region [0x614000000040,0x6140000001d4)
allocated by thread T0 here:
    #0 0x4da52f in __interceptor_realloc.part.0 asan_malloc_linux.cpp.o
    #1 0x561e29 in process_input /home/rocky/sqlite3/shell.c:26619:14

SUMMARY: AddressSanitizer: heap-buffer-overflow /home/rocky/sqlite3/shell.c:17931:13 in shell_error_context
Shadow bytes around the buggy address:
  0x0c287fff7fe0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c287fff7ff0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c287fff8000: fa fa fa fa fa fa fa fa 00 00 00 00 00 00 00 00
  0x0c287fff8010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c287fff8020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x0c287fff8030: 00 00 00 00 00 00 00 00 00 00[04]fa fa fa fa fa
  0x0c287fff8040: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
  0x0c287fff8050: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c287fff8060: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c287fff8070: fd fd fd fd fd fd fd fd fd fd fa fa fa fa fa fa
  0x0c287fff8080: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==9780==ABORTING

Input: test4.sql

CREATE  TEMPORARY  TABLE t1 AS  SELECT  ALL *, *, *, *, *, *, X'53514C697465', X'53514C697465', *, * FROM (temp.sqlite_schema) WHERE X'53514C697465';
ALTER  TABLE t1 ADD path COLLATE BINARY GENERATED  ALWAYS  AS (path->path* FALSE ) VIRTUAL ;
SELECT * FROM t1;

Output:

=================================================================
==9782==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x611000000279 at pc 0x0000005998c7 bp 0x7ffdbcc32360 sp 0x7ffdbcc32358
READ of size 1 at 0x611000000279 thread T0
    #0 0x5998c6 in shell_error_context /home/rocky/sqlite3/shell.c:17931:13
    #1 0x5a1fee in save_err_msg /home/rocky/sqlite3/shell.c:18020:14
    #2 0x5608fa in shell_exec /home/rocky/sqlite3/shell.c:19237:21
    #3 0x5aba57 in runOneSqlLine /home/rocky/sqlite3/shell.c:26470:8
    #4 0x56229e in process_input /home/rocky/sqlite3/shell.c:26636:17
    #5 0x53dd8d in main /home/rocky/sqlite3/shell.c:27541:12
    #6 0x7fe1dd1a4eaf in __libc_start_call_main (/lib64/libc.so.6+0x3feaf) (BuildId: 82f7ae28e16376aa97cc3bf50b40ab2d1043924a)
    #7 0x7fe1dd1a4f5f in __libc_start_main@GLIBC_2.2.5 (/lib64/libc.so.6+0x3ff5f) (BuildId: 82f7ae28e16376aa97cc3bf50b40ab2d1043924a)
    #8 0x427614 in _start (/home/rocky/sqlite3/sqlite3+0x427614) (BuildId: bee988bfc215b67422a317b2b26ed8fcdadbf751)

0x611000000279 is located 0 bytes to the right of 249-byte region [0x611000000180,0x611000000279)
allocated by thread T0 here:
    #0 0x4da52f in __interceptor_realloc.part.0 asan_malloc_linux.cpp.o
    #1 0x561e29 in process_input /home/rocky/sqlite3/shell.c:26619:14

SUMMARY: AddressSanitizer: heap-buffer-overflow /home/rocky/sqlite3/shell.c:17931:13 in shell_error_context
Shadow bytes around the buggy address:
  0x0c227fff7ff0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c227fff8000: fa fa fa fa fa fa fa fa 00 00 00 00 00 00 00 00
  0x0c227fff8010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c227fff8020: 00 00 00 fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c227fff8030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x0c227fff8040: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00[01]
  0x0c227fff8050: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
  0x0c227fff8060: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c227fff8070: fd fd fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c227fff8080: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c227fff8090: fd fd fd fd fd fd fd fd fd fd fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==9782==ABORTING

Input: test5.sql

CREATE  TEMPORARY  TABLE t2 AS  WITH  RECURSIVE sqlite_temp_schema AS ( VALUES (X'53514C697465')) SELECT  ALL *, *, *, *, *, *, *, *, * FROM (temp.sqlite_temp_schema), ( VALUES ( NOT  CURRENT_TIME ,  CURRENT_TIME )) AS t2;
ALTER  TABLE t2 ADD  COLUMN c0 CONSTRAINT constr1 AS (rootpage BETWEEN (rootpage) NOT  NULL  AND (((rootpage) NOT  IN (((rootpage) NOT  NULL ) NOT  NULL )) NOT  NULL ) NOTNULL ->>(rootpage) NOT  NULL );
SELECT * FROM t2;

Output:

=================================================================
==9814==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x613000000182 at pc 0x0000005998c7 bp 0x7fffaf8827a0 sp 0x7fffaf882798
READ of size 1 at 0x613000000182 thread T0
    #0 0x5998c6 in shell_error_context /home/rocky/sqlite3/shell.c:17931:13
    #1 0x5a1fee in save_err_msg /home/rocky/sqlite3/shell.c:18020:14
    #2 0x5608fa in shell_exec /home/rocky/sqlite3/shell.c:19237:21
    #3 0x5aba57 in runOneSqlLine /home/rocky/sqlite3/shell.c:26470:8
    #4 0x56229e in process_input /home/rocky/sqlite3/shell.c:26636:17
    #5 0x53dd8d in main /home/rocky/sqlite3/shell.c:27541:12
    #6 0x7ff999edaeaf in __libc_start_call_main (/lib64/libc.so.6+0x3feaf) (BuildId: 82f7ae28e16376aa97cc3bf50b40ab2d1043924a)
    #7 0x7ff999edaf5f in __libc_start_main@GLIBC_2.2.5 (/lib64/libc.so.6+0x3ff5f) (BuildId: 82f7ae28e16376aa97cc3bf50b40ab2d1043924a)
    #8 0x427614 in _start (/home/rocky/sqlite3/sqlite3+0x427614) (BuildId: bee988bfc215b67422a317b2b26ed8fcdadbf751)

0x613000000182 is located 0 bytes to the right of 322-byte region [0x613000000040,0x613000000182)
allocated by thread T0 here:
    #0 0x4da52f in __interceptor_realloc.part.0 asan_malloc_linux.cpp.o
    #1 0x561e29 in process_input /home/rocky/sqlite3/shell.c:26619:14

SUMMARY: AddressSanitizer: heap-buffer-overflow /home/rocky/sqlite3/shell.c:17931:13 in shell_error_context
Shadow bytes around the buggy address:
  0x0c267fff7fe0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c267fff7ff0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c267fff8000: fa fa fa fa fa fa fa fa 00 00 00 00 00 00 00 00
  0x0c267fff8010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c267fff8020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x0c267fff8030:[02]fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c267fff8040: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c267fff8050: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c267fff8060: 00 00 00 00 00 00 00 00 00 fa fa fa fa fa fa fa
  0x0c267fff8070: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
  0x0c267fff8080: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==9814==ABORTING

Input: test6.sql

CREATE  TEMPORARY  TABLE t1 AS  SELECT  ALL *, *, *, *, *, *, X'53514C697465', X'53514C697465', *, * FROM (temp.sqlite_schema) WHERE X'53514C697465';
ALTER  TABLE t1 ADD path COLLATE BINARY GENERATED  ALWAYS  AS (path->>X'53514C697465') VIRTUAL ;
SELECT * FROM t1;

Output:

=================================================================
==9818==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x611000000279 at pc 0x0000005998c7 bp 0x7ffdda2d12c0 sp 0x7ffdda2d12b8
READ of size 1 at 0x611000000279 thread T0
    #0 0x5998c6 in shell_error_context /home/rocky/sqlite3/shell.c:17931:13
    #1 0x5a1fee in save_err_msg /home/rocky/sqlite3/shell.c:18020:14
    #2 0x5608fa in shell_exec /home/rocky/sqlite3/shell.c:19237:21
    #3 0x5aba57 in runOneSqlLine /home/rocky/sqlite3/shell.c:26470:8
    #4 0x56229e in process_input /home/rocky/sqlite3/shell.c:26636:17
    #5 0x53dd8d in main /home/rocky/sqlite3/shell.c:27541:12
    #6 0x7f2463b69eaf in __libc_start_call_main (/lib64/libc.so.6+0x3feaf) (BuildId: 82f7ae28e16376aa97cc3bf50b40ab2d1043924a)
    #7 0x7f2463b69f5f in __libc_start_main@GLIBC_2.2.5 (/lib64/libc.so.6+0x3ff5f) (BuildId: 82f7ae28e16376aa97cc3bf50b40ab2d1043924a)
    #8 0x427614 in _start (/home/rocky/sqlite3/sqlite3+0x427614) (BuildId: bee988bfc215b67422a317b2b26ed8fcdadbf751)

0x611000000279 is located 0 bytes to the right of 249-byte region [0x611000000180,0x611000000279)
allocated by thread T0 here:
    #0 0x4da52f in __interceptor_realloc.part.0 asan_malloc_linux.cpp.o
    #1 0x561e29 in process_input /home/rocky/sqlite3/shell.c:26619:14

SUMMARY: AddressSanitizer: heap-buffer-overflow /home/rocky/sqlite3/shell.c:17931:13 in shell_error_context
Shadow bytes around the buggy address:
  0x0c227fff7ff0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c227fff8000: fa fa fa fa fa fa fa fa 00 00 00 00 00 00 00 00
  0x0c227fff8010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c227fff8020: 00 00 00 fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c227fff8030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x0c227fff8040: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00[01]
  0x0c227fff8050: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
  0x0c227fff8060: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c227fff8070: fd fd fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c227fff8080: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c227fff8090: fd fd fd fd fd fd fd fd fd fd fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==9818==ABORTING

Issue 3 - Segmentation Fault

Input: test7.sql

EXPLAIN  CREATE  TABLE t2 AS  SELECT  DISTINCT ':memory:', 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7 ORDER  BY '%J%j%w%s', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', '%J%j%w%s', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 42e-300, 'unixepoch', 'unixepoch', 'unixepoch' LIMIT 0xda;

Output:

AddressSanitizer:DEADLYSIGNAL
=================================================================
==10076==ERROR: AddressSanitizer: SEGV on unknown address (pc 0x000000713c3c bp 0x7ffeb1e76290 sp 0x7ffeb1e76140 T0)
==10076==The signal is caused by a READ memory access.
==10076==Hint: this fault was caused by a dereference of a high value address (see register values below).  Disassemble the provided pc to learn which register was used.
    #0 0x713c3c in sqlite3VdbeDisplayP4 /home/rocky/sqlite3/sqlite3.c:84362:44
    #1 0x5cbcac in sqlite3VdbeList /home/rocky/sqlite3/sqlite3.c:84888:19
    #2 0x5cbcac in sqlite3Step /home/rocky/sqlite3/sqlite3.c:88654:10
    #3 0x5cbcac in sqlite3_step /home/rocky/sqlite3/sqlite3.c:88720:16
    #4 0x5a3c67 in explain_data_prepare /home/rocky/sqlite3/shell.c:18416:26
    #5 0x5602c1 in shell_exec /home/rocky/sqlite3/shell.c:19320:11
    #6 0x5aba57 in runOneSqlLine /home/rocky/sqlite3/shell.c:26470:8
    #7 0x56229e in process_input /home/rocky/sqlite3/shell.c:26636:17
    #8 0x53dd8d in main /home/rocky/sqlite3/shell.c:27541:12
    #9 0x7fd0f1d12eaf in __libc_start_call_main (/lib64/libc.so.6+0x3feaf) (BuildId: 82f7ae28e16376aa97cc3bf50b40ab2d1043924a)
    #10 0x7fd0f1d12f5f in __libc_start_main@GLIBC_2.2.5 (/lib64/libc.so.6+0x3ff5f) (BuildId: 82f7ae28e16376aa97cc3bf50b40ab2d1043924a)
    #11 0x427614 in _start (/home/rocky/sqlite3/sqlite3+0x427614) (BuildId: bee988bfc215b67422a317b2b26ed8fcdadbf751)

AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV /home/rocky/sqlite3/sqlite3.c:84362:44 in sqlite3VdbeDisplayP4
==10076==ABORTING

Issue 4 - Heap Buffer Overflow

This issue seems to occur when using more than 64 arguments to "SELECT DISTINCT" and "ORDER BY"

Input: test8.sql

 CREATE  TABLE t0 AS  SELECT  DISTINCT 0xda, 'lit0', 'lit0', 'lit0', 'lit0', 'lit0', 'lit0', 'lit0', 'lit0', 'lit0', 'lit0', 'lit0', 'lit0', 'lit0', 'lit0', 'lit0', 'lit0', 'lit0', 'lit0', 'lit0', 'lit0', 'lit0', 'lit0', 'lit0', 'lit0', 'lit0', 'lit0', 'lit0', 'lit0', 'lit0', 'lit0', 'lit0', 'lit0', 'lit0', 'lit0', 'lit0', 'lit0', 'lit0', 'lit0', 'lit0', 'lit0', 'lit0', 'lit0', 'lit0', 'lit0', 0xda-0xda-42e-300, 'lit0', 'lit0', 'lit0', 'lit0', 'lit0', 'lit0', 'lit0', 'lit0', 'lit0', 'lit0', 'lit0', 'lit0', 'lit0', 'lit0', 'lit0', 'lit0', 'lit0', 'lit0', 'lit0' ORDER  BY '%%', '%%', '%%', '%%', '%%', '%%', '%%', '%%', '%%', '%%', '%%', '%%', '%%', '%%', '%%', '%Y-%m-%d', '%%', '%%', '%%', '%%', '%%', '%%', '%%', '%%', '%%', '%%', 'lit0', '%%', '%%', '%%', '%%', '%%', '%%', '%%', '%%', '%%', '%%', '%%', '%%', '%%', 'auto', '%%', '%%', '%%', '%%', '%%', '%%', '%%', '%%', '%%', '%%', '%%', ':memory:', '%%', '%%', '%%', '%%', '%%', '%%', '%%', '%%', '%%', '%%', '';

Output:

=================================================================
==10091==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x6160000017f8 at pc 0x0000004d952a bp 0x7ffccf4e2930 sp 0x7ffccf4e20f0
READ of size 524312 at 0x6160000017f8 thread T0
    #0 0x4d9529 in __asan_memcpy (/home/rocky/sqlite3/sqlite3+0x4d9529) (BuildId: bee988bfc215b67422a317b2b26ed8fcdadbf751)
    #1 0x6c72c3 in sqlite3VdbeSorterInit /home/rocky/sqlite3/sqlite3.c:100803:5
    #2 0x6c72c3 in sqlite3VdbeExec /home/rocky/sqlite3/sqlite3.c:94838:8
    #3 0x5ca99c in sqlite3Step /home/rocky/sqlite3/sqlite3.c:88659:10
    #4 0x5ca99c in sqlite3_step /home/rocky/sqlite3/sqlite3.c:88720:16
    #5 0x5a473f in exec_prepared_stmt /home/rocky/sqlite3/shell.c:19008:8
    #6 0x5603ef in shell_exec /home/rocky/sqlite3/shell.c:19325:7
    #7 0x5aba57 in runOneSqlLine /home/rocky/sqlite3/shell.c:26470:8
    #8 0x56229e in process_input /home/rocky/sqlite3/shell.c:26636:17
    #9 0x53dd8d in main /home/rocky/sqlite3/shell.c:27541:12
    #10 0x7f1f9d1dfeaf in __libc_start_call_main (/lib64/libc.so.6+0x3feaf) (BuildId: 82f7ae28e16376aa97cc3bf50b40ab2d1043924a)
    #11 0x7f1f9d1dff5f in __libc_start_main@GLIBC_2.2.5 (/lib64/libc.so.6+0x3ff5f) (BuildId: 82f7ae28e16376aa97cc3bf50b40ab2d1043924a)
    #12 0x427614 in _start (/home/rocky/sqlite3/sqlite3+0x427614) (BuildId: bee988bfc215b67422a317b2b26ed8fcdadbf751)

0x6160000017f8 is located 0 bytes to the right of 632-byte region [0x616000001580,0x6160000017f8)
allocated by thread T0 here:
    #0 0x4db327 in __interceptor_malloc (/home/rocky/sqlite3/sqlite3+0x4db327) (BuildId: bee988bfc215b67422a317b2b26ed8fcdadbf751)
    #1 0x9e9566 in sqlite3MemMalloc /home/rocky/sqlite3/sqlite3.c:25656:7

SUMMARY: AddressSanitizer: heap-buffer-overflow (/home/rocky/sqlite3/sqlite3+0x4d9529) (BuildId: bee988bfc215b67422a317b2b26ed8fcdadbf751) in __asan_memcpy
Shadow bytes around the buggy address:
  0x0c2c7fff82a0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c2c7fff82b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c2c7fff82c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c2c7fff82d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c2c7fff82e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x0c2c7fff82f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00[fa]
  0x0c2c7fff8300: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c2c7fff8310: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c2c7fff8320: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c2c7fff8330: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c2c7fff8340: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==10091==ABORTING

I think this is the same issue as above, but results in an assertion violation.

Input: test9.sql

CREATE  TABLE t2 AS  SELECT  DISTINCT ':memory:', 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 0.0*7/0, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7 ORDER  BY '%J%j%w%s', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', '%J%j%w%s', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 'unixepoch', 42e-300, 'unixepoch', 'unixepoch', 'unixepoch' LIMIT 0xda;

Output:

sqlite3: sqlite3.c:93237: int sqlite3VdbeExec(Vdbe *): Assertion `p2<(u32)pC->nField || (pC->eCurType==CURTYPE_PSEUDO && pC->seekResult==0)' failed.
Aborted (core dumped)

Best regards,

Maik Betka

(2) By Richard Hipp (drh) on 2023-03-16 20:36:27 in reply to 1.5 [link] [source]

Broken up into 9 separate tickets for tracking:

(3) By Richard Hipp (drh) on 2023-03-17 00:53:04 in reply to 1.5 [link] [source]

The first two problems stem from an incorrect assert() statement. That assert() statement has now been corrected. This has zero impact on production code, since assert() is disabled by default on production code.

The next four problems could not be reproduced. I tried many variations on ASAN and Valgrind, but was never able to generate a buffer overflow. The buffer in question is the command-line input buffer on the CLI. This is a CLI-only feature - it is not part of the core SQLite library. So regardless of whether or not this is a real problem, it does not affect the core SQLite library. Some extra space is now allocated on the end of the input buffer to guard against off-by-one errors that we might not have detected.

The last three reports are a single real problem, which is now fixed. It is a memory error in the SQLite core that affects production builds. SQLite versions 3.39.0 through 3.41.1 are affected. A new patch release that contains a fix for this problem will come out soon.

(6.1) By Maik (maikbe) on 2023-03-17 13:40:52 edited from 6.0 in reply to 3 [link] [source]

Deleted

(8.1) By Maik (maikbe) on 2023-03-17 13:51:26 edited from 8.0 in reply to 6.1 [link] [source]

Hi Richard,

thank you for the replies. I also cannot reproduce that one heap buffer overflow when using GCC with ASAN. When using AFL++, however, I can reliably get the heap buffer overflow, even after the recent update that adds extra space to the buffer.

This let's me question whether AFL++ is doing something wrong or the other compilers (or ASAN) miss something.

I don't know whether you'd still like to reproduce the bug, but I've prepared a Dockerfile and minimal steps below to compile SQLite3 with it.

vim Dockerfile

FROM rockylinux:9

# epel installation and 'crb enable' is needed for 're2c'
RUN dnf update -y && dnf install -y epel-release && crb enable && dnf install -y \
  # for user interaction, downloads, and password management
  vim wget passwd \
  # AFL++ dependencies
  python3-devel automake cmake git flex bison glib2-devel pixman-devel python3-setuptools gtk3-devel lld llvm llvm-devel clang \
  # SQLite dependencies
  tcl 

# create the 'rocky' group and user, replace it with your current user
RUN groupadd --gid 1000 rocky \
  && useradd --uid 1000 --gid 1000 --home-dir /home/rocky --create-home --no-user-group --shell /bin/bash rocky \
  # for the 'rocky and 'root' users, delete their passwords and lock the accounts
  && passwd --delete rocky \
  && passwd --lock rocky \
  && passwd --delete root \
  && passwd --lock root

USER rocky
WORKDIR /home/rocky

RUN git clone https://github.com/AFLplusplus/AFLplusplus/ && cd AFLplusplus/ && make all
RUN wget https://www.sqlite.org/src/tarball/sqlite.tar.gz && tar -xzf sqlite.tar.gz

Build:

sudo docker build -t sqlite3-asan-test:latest .

Starting the container:

sudo docker run -it sqlite3-asan-test:latest

Compile SQLite3 with afl-cc:

cd sqlite

./configure

make sqlite3.c

CC=~/AFLplusplus/afl-cc \
CXX=~/AFLplusplus/afl-c++ \
AFL_CC_COMPILER=LLVM \
AFL_LLVM_INSTRUMENT=PCGUARD \
AFL_USE_ASAN=1 \
~/AFLplusplus/afl-cc -O3 \
  -DSQLITE_DEBUG \
  -DSQLITE_MAX_EXPR_DEPTH=0 \
  -DSQLITE_THREADSAFE=0 \
  -DSQLITE_ENABLE_UPDATE_DELETE_LIMIT \
  -DSQLITE_OMIT_LOAD_EXTENSION \
  -DSQLITE_OMIT_JSON \
  -DSQLITE_OMIT_DEPRECATED \
  -I. shell.c sqlite3.c -o sqlite3

Next would be to test the examples again.

Best regards,

Maik

(12) By Maik (maikbe) on 2023-03-19 22:21:00 in reply to 3 [link] [source]

Hi Richard,

I was testing the second issue, i.e., the heap buffer overflow. Now I was able to reliably reproduce it on the current version with clang 14.0.6 and gcc 11.3.1. Maybe you want to check on that version again? On the one I've initially submitted I was only able to reproduce it with AFL++. Now I also get it with gcc and clang.

Fossil info:

$ fossil info
project-name: SQLite
repository:   /home/rocky/sqlite3/sqlite.fossil
local-root:   /home/rocky/sqlite3/
config-db:    /home/rocky/.fossil
project-code: 2ab58778c2967968b94284e989e43dc11791f548
checkout:     02ac2297abee6af64c8df230b42b07f21cff4565 2023-03-18 16:12:27 UTC
parent:       0910b1925e97f7ae4dae86894c9e2f54273c8511 2023-03-17 19:18:17 UTC
tags:         trunk
comment:      Avoid a buffer overread in fts3 that could occur when processing a corrupt record. (user: dan)
check-ins:    28532

Build:

CC=gcc ./configure
CC=gcc make sqlite3.c
gcc -fsanitize=address -O0 -DSQLITE_DEBUG -DSQLITE_MAX_EXPR_DEPTH=0 -DSQLITE_THREADSAFE=0 -DSQLITE_ENABLE_UPDATE_DELETE_LIMIT -DSQLITE_OMIT_LOAD_EXTENSION -DSQLITE_OMIT_JSON -DSQLITE_OMIT_DEPRECATED -I. shell.c sqlite3.c -o sqlite3

Test:

CREATE  TABLE  IF  NOT  EXISTS dbstat(pagetypeREAL);
CREATE  TEMPORARY  TABLE  IF  NOT  EXISTS dbstat AS  SELECT  ALL dbstat.*, *, *, *, *, *, dbstat.*, *, *, *, *, *, *, *, *, *, *, *, *, *, '%Y-%m-%d', *, *, *, *, * FROM (dbstat AS dbstat NOT  INDEXED , sqlite_schema AS dbstat NOT  INDEXED ) GROUP  BY  FALSE  HAVING  FALSE  ORDER  BY 5e200 NULLS  FIRST;
ALTER  TABLE dbstat ADD  COLUMN pagetype CONSTRAINT constr1 GENERATED  ALWAYS  AS ( RAISE ( IGNORE )->> FALSE  OR  FALSE );
SELECT * FROM dbstat;

Execution:

cat test3.sql | ./sqlite3

Output:

$ cat test3.sql | ./sqlite3
=================================================================
==58462==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x6140000001d8 at pc 0x000000434a22 bp 0x7ffc305cb270 sp 0x7ffc305cb260
READ of size 1 at 0x6140000001d8 thread T0
    #0 0x434a21 in shell_error_context (/home/rocky/sqlite3/sqlite3+0x434a21)
    #1 0x4353d6 in save_err_msg (/home/rocky/sqlite3/sqlite3+0x4353d6)
    #2 0x43d69e in shell_exec (/home/rocky/sqlite3/sqlite3+0x43d69e)
    #3 0x463625 in runOneSqlLine (/home/rocky/sqlite3/sqlite3+0x463625)
    #4 0x464392 in process_input (/home/rocky/sqlite3/sqlite3+0x464392)
    #5 0x467b04 in main (/home/rocky/sqlite3/sqlite3+0x467b04)
    #6 0x7fe45d46feaf in __libc_start_call_main (/lib64/libc.so.6+0x3feaf)
    #7 0x7fe45d46ff5f in __libc_start_main_alias_1 (/lib64/libc.so.6+0x3ff5f)
    #8 0x403704 in _start (/home/rocky/sqlite3/sqlite3+0x403704)

0x6140000001d8 is located 0 bytes to the right of 408-byte region [0x614000000040,0x6140000001d8)
allocated by thread T0 here:
    #0 0x7fe45d6edd98 in __interceptor_realloc (/lib64/libasan.so.6+0xb4d98)
    #1 0x4640b9 in process_input (/home/rocky/sqlite3/sqlite3+0x4640b9)
    #2 0x467b04 in main (/home/rocky/sqlite3/sqlite3+0x467b04)
    #3 0x7fe45d46feaf in __libc_start_call_main (/lib64/libc.so.6+0x3feaf)

SUMMARY: AddressSanitizer: heap-buffer-overflow (/home/rocky/sqlite3/sqlite3+0x434a21) in shell_error_context
Shadow bytes around the buggy address:
  0x0c287fff7fe0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c287fff7ff0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c287fff8000: fa fa fa fa fa fa fa fa 00 00 00 00 00 00 00 00
  0x0c287fff8010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c287fff8020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x0c287fff8030: 00 00 00 00 00 00 00 00 00 00 00[fa]fa fa fa fa
  0x0c287fff8040: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
  0x0c287fff8050: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c287fff8060: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c287fff8070: fd fd fd fd fd fd fd fd fd fd fa fa fa fa fa fa
  0x0c287fff8080: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==58462==ABORTING

Cheers,

Maik

(13) By Richard Hipp (drh) on 2023-03-19 23:20:26 in reply to 12 [link] [source]

The -DSQLITE_OMIT_JSON compile-time option seems to be necessary to get the failure to occur. I can repro the problem now.

(14) By Richard Hipp (drh) on 2023-03-20 02:00:44 in reply to 12 [link] [source]

Please try again with the latest trunk and/or branch-3.41 check-in.

(15) By Maik (maikbe) on 2023-03-20 06:20:38 in reply to 14 [link] [source]

The heap buffer overflow is gone on the latest trunk. I tested it with all provided examples.

Thank you very much for the investigation and the code fix!

Best regards,

Maik

(4) By jose isaias cabrera (jicman) on 2023-03-17 11:39:32 in reply to 1.5 [link] [source]

Perhaps, next time, break these into different posts. It's hard to keep track on multiple problems on one post. In my opinion. Thanks.

(5) By Richard Hipp (drh) on 2023-03-17 12:11:37 in reply to 4 [link] [source]

Amen.

In retrospect, I should not have created nine separate tickets - one ticket for each of the scripts provided in the original post. It would have been sufficient to do only three, since there were in fact only three distinct issues. But I didn't realize that at the time.

It would have been even better if the original poster (the "OP") had recognized that there were probably only three issues here, and posted each issue in a separate thread. The threads could reference each other. For example, the first thread could say "This is the first of three issues I want to report - the other two will following in separate forum threads." Then in the second thread say "This is the 2nd issue. The first was shown in <link-to-first-issue>". And so forth.

Do you have two or more repro cases and are unsure if they really are one and the same issue? Then just say so. "I think the following two scripts are different manifestations of the same issue since they have the same failure signature..."

(7) By Maik (maikbe) on 2023-03-17 13:10:10 in reply to 5 [link] [source]

Please excuse the hassle. I will split the issues next time.

Best regards, Maik

(9) By jose isaias cabrera (jicman) on 2023-03-17 13:51:41 in reply to 7 [link] [source]

Thank you for the report. We all learn a thing or two every day. :-)

(10) By Larry Brasfield (larrybr) on 2023-03-17 15:10:30 in reply to 7 [link] [source]

This is already apparent with close reading, but I want to be clear about it.

We appreciate well-formed and complete bug reports, which yours certainly was. And, given that the several reports shared common information, it was reasonable to combine them. They were combined with a clear distinction between the several result/expectation discrepancies that you found. (IOW, the "issues" were already split by section headings, "Issue X".)

All-in-all, I would say it was a stellar bug report. Any structural improvement creating separate yet linked reports would go to making it "stellar+".

(11) By Simon Slavin (slavin) on 2023-03-17 16:53:38 in reply to 7 [link] [source]

Please do not apologise for your excellent bug report, which has led to genuine bug-fixes in SQLite, and may lead to more once it is fully understood. It is top quality work which deserves respect.