SQLite Forum

Edit History Of A Forum Post
Login

(1) By Richard Hipp (drh) on 2020-06-04 03:34:15 [link]

It appears that the clang-11.0.0 compiler mis-compiles sqlite3.c version 3.32.1.
I have [checked in a change][0] to SQLite that appears to work around the
problem.  But there might be other bugs.  Therefore,
use clang-11.0.0 with caution and test your applications carefully!

[0]: https://www.sqlite.org/src/info/3c2bf8042ec46195

## Details:

[OSSFuzz][1] has been reported bug [23003][2] against SQLite.
I could not reproduce the problem on my desktop (Ubuntu with gcc-5.4.0)
so I followed the [OSSFuzz bug replication procedures][3] and discovered
what appears to be a problem with Clang-11.0.0 currently used by
OSSFuzz. 

The code that is miscompiled is [lines 345-347 of the src/utf.c source file][4], shown below:

~~~~
      c = pMem->flags;
      sqlite3VdbeMemRelease(pMem);
      pMem->flags = MEM_Str|MEM_Term|(c&(MEM_AffMask|MEM_Subtype));
~~~~

From the -S output, it looks like
Clang-11.0.0 is compiling these three lines as if there were written as:

~~~~
      sqlite3VdbeMemRelease(pMem);
      pMem->flags = MEM_Str|MEM_Term|(pMem->flags&(MEM_AffMask|MEM_Subtype));
~~~~

In other words, Clang seems to be assuming that the sqlite3VdbeMemRelease() function
does not change the value of pMem->flags.  But it does. My
work-around is to do the bit-twiddling of pMem->flags before the function
call instead of afterwards:

~~~~
      c = MEM_Str|MEM_Term|(pMem->flags&(MEM_AffMask|MEM_Subtype));
      sqlite3VdbeMemRelease(pMem);
      pMem->flags = c;
~~~~

## Compiler Version And Build Details:

OSSFuzz reports the compiler used as:

~~~~
    clang version 11.0.0 (https://github.com/llvm/llvm-project.git a6ae333a0c23fc9b0783ca45e2676abac00c6723)
    Target: x86_64-unknown-linux-gnu
    Thread model: posix
~~~~

The build script compiles SQLite thusly:

`clang -O1 -fno-omit-frame-pointer -gline-tables-only -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION -fsanitize=address -fsanitize-address-use-after-scope -fsanitize=fuzzer-no-link -DSQLITE_MAX_LENGTH=128000000 -DSQLITE_MAX_SQL_LENGTH=128000000 -DSQLITE_MAX_MEMORY=25000000 -DSQLITE_PRINTF_PRECISION_LIMIT=1048576 -DSQLITE_DEBUG=1 -DSQLITE_MAX_PAGE_COUNT=16384 -O1 -g -I. -c -O1 -g ./sqlite3.c -o sqlite3.o`





[1]: https://github.com/google/oss-fuzz
[2]: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=23003
[3]: https://google.github.io/oss-fuzz/advanced-topics/reproducing/
[4]: https://www.sqlite.org/src/artifact/d7a61c1dfda?ln=345-347

(2) By Richard Hipp (drh) on 2020-06-04 10:29:29 [link]

It appears that the clang-11.0.0 compiler mis-compiles sqlite3.c version 3.32.1.
I have [checked in a change][0] to SQLite that appears to work around the
problem.  But there might be other bugs.  Therefore,
use clang-11.0.0 with caution and test your applications carefully!

[0]: https://www.sqlite.org/src/info/3c2bf8042ec46195

UPDATE:  I learn (from a [thread on HN][hn]) that the compiler in question is
pre-release.

[hn]: https://news.ycombinator.com/item?id=23412482

## Details:

[OSSFuzz][1] has been reporting bug [23003][2] against SQLite.
I could not reproduce the problem on my desktop (Ubuntu with gcc-5.4.0)
so I followed the [OSSFuzz bug replication procedures][3] and discovered
what appears to be a problem with Clang-11.0.0 currently used by
OSSFuzz. 

The code that is miscompiled is [lines 345-347 of the src/utf.c source file][4], shown below:

~~~~
      c = pMem->flags;
      sqlite3VdbeMemRelease(pMem);
      pMem->flags = MEM_Str|MEM_Term|(c&(MEM_AffMask|MEM_Subtype));
~~~~

From the -S output, it looks like
Clang-11.0.0 is compiling these three lines as if there were written as:

~~~~
      sqlite3VdbeMemRelease(pMem);
      pMem->flags = MEM_Str|MEM_Term|(pMem->flags&(MEM_AffMask|MEM_Subtype));
~~~~

In other words, Clang seems to be assuming that the sqlite3VdbeMemRelease() function
does not change the value of pMem->flags.  But it does. My
work-around is to do the bit-twiddling of pMem->flags before the function
call instead of afterwards:

~~~~
      c = MEM_Str|MEM_Term|(pMem->flags&(MEM_AffMask|MEM_Subtype));
      sqlite3VdbeMemRelease(pMem);
      pMem->flags = c;
~~~~

## Compiler Version And Build Details:

OSSFuzz reports the compiler used as:

~~~~
    clang version 11.0.0 (https://github.com/llvm/llvm-project.git a6ae333a0c23fc9b0783ca45e2676abac00c6723)
    Target: x86_64-unknown-linux-gnu
    Thread model: posix
~~~~

The build script compiles SQLite thusly:

`clang -O1 -fno-omit-frame-pointer -gline-tables-only -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION -fsanitize=address -fsanitize-address-use-after-scope -fsanitize=fuzzer-no-link -DSQLITE_MAX_LENGTH=128000000 -DSQLITE_MAX_SQL_LENGTH=128000000 -DSQLITE_MAX_MEMORY=25000000 -DSQLITE_PRINTF_PRECISION_LIMIT=1048576 -DSQLITE_DEBUG=1 -DSQLITE_MAX_PAGE_COUNT=16384 -O1 -g -I. -c -O1 -g ./sqlite3.c -o sqlite3.o`





[1]: https://github.com/google/oss-fuzz
[2]: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=23003
[3]: https://google.github.io/oss-fuzz/advanced-topics/reproducing/
[4]: https://www.sqlite.org/src/artifact/d7a61c1dfda?ln=345-347

(3) By Richard Hipp (drh) on 2020-06-04 12:20:24 [link]

It appears that the clang-11.0.0 compiler mis-compiles sqlite3.c version 3.32.1.
I have [checked in a change][0] to SQLite that appears to work around the
problem.  But there might be other bugs.  Therefore,
use clang-11.0.0 with caution and test your applications carefully!

[0]: https://www.sqlite.org/src/info/3c2bf8042ec46195

UPDATE:  I learn (from a [thread on HN][hn]) that the compiler in question is
pre-release.

[hn]: https://news.ycombinator.com/item?id=23412482

UPDATE-2:  The LLVM devs apparently saw this post on [HN][hn] and
quickly isolated the problem. [They have a ticket][l1].  Great work, y'all!

[l1]: https://bugs.llvm.org/show_bug.cgi?id=46195

## Details:

[OSSFuzz][1] has been reporting bug [23003][2] against SQLite.
I could not reproduce the problem on my desktop (Ubuntu with gcc-5.4.0)
so I followed the [OSSFuzz bug replication procedures][3] and discovered
what appears to be a problem with Clang-11.0.0 currently used by
OSSFuzz. 

The code that is miscompiled is [lines 345-347 of the src/utf.c source file][4], shown below:

~~~~
      c = pMem->flags;
      sqlite3VdbeMemRelease(pMem);
      pMem->flags = MEM_Str|MEM_Term|(c&(MEM_AffMask|MEM_Subtype));
~~~~

From the -S output, it looks like
Clang-11.0.0 is compiling these three lines as if there were written as:

~~~~
      sqlite3VdbeMemRelease(pMem);
      pMem->flags = MEM_Str|MEM_Term|(pMem->flags&(MEM_AffMask|MEM_Subtype));
~~~~

In other words, Clang seems to be assuming that the sqlite3VdbeMemRelease() function
does not change the value of pMem->flags.  But it does. My
work-around is to do the bit-twiddling of pMem->flags before the function
call instead of afterwards:

~~~~
      c = MEM_Str|MEM_Term|(pMem->flags&(MEM_AffMask|MEM_Subtype));
      sqlite3VdbeMemRelease(pMem);
      pMem->flags = c;
~~~~

## Compiler Version And Build Details:

OSSFuzz reports the compiler used as:

~~~~
    clang version 11.0.0 (https://github.com/llvm/llvm-project.git a6ae333a0c23fc9b0783ca45e2676abac00c6723)
    Target: x86_64-unknown-linux-gnu
    Thread model: posix
~~~~

The build script compiles SQLite thusly:

`clang -O1 -fno-omit-frame-pointer -gline-tables-only -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION -fsanitize=address -fsanitize-address-use-after-scope -fsanitize=fuzzer-no-link -DSQLITE_MAX_LENGTH=128000000 -DSQLITE_MAX_SQL_LENGTH=128000000 -DSQLITE_MAX_MEMORY=25000000 -DSQLITE_PRINTF_PRECISION_LIMIT=1048576 -DSQLITE_DEBUG=1 -DSQLITE_MAX_PAGE_COUNT=16384 -O1 -g -I. -c -O1 -g ./sqlite3.c -o sqlite3.o`





[1]: https://github.com/google/oss-fuzz
[2]: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=23003
[3]: https://google.github.io/oss-fuzz/advanced-topics/reproducing/
[4]: https://www.sqlite.org/src/artifact/d7a61c1dfda?ln=345-347