SQLite Forum

Inconsistent memory stats of sqlite3 with different compilers
Login

Inconsistent memory stats of sqlite3 with different compilers

(1) By Shaohua (shaohua.li) on 2021-06-10 08:53:31 [link] [source]

Hi there,

I found that sqlite3 -stats compiled with gcc11 and clang12 would emit different memory stats on some crafted inputs.

  • SQLite version: 3.36.0

  • Compiler args: -DSQLITE_MAX_LENGTH=128000000 \ -DSQLITE_MAX_SQL_LENGTH=128000000 \ -DSQLITE_MAX_MEMORY=25000000 \ -DSQLITE_PRINTF_PRECISION_LIMIT=1048576 \ -DSQLITE_DEBUG=1 \ -DSQLITE_MAX_PAGE_COUNT=16384 (I used these args for avoiding creating large databases and irrelevant OOMs)

  • Compilers: gcc11 and clang12

  • Reproduce: run the following command with different compilers compiled binaries. Then compare the two stats.txt.

    sqlite3 -stats < diff.sql | xargs -i echo {} > stats.txt

You can download diff.sql from this link: diff.sql

(2) By Richard Hipp (drh) on 2021-06-10 14:51:42 in reply to 1 [link] [source]

The "diff.sql" is fuzzer output. What are you doing? Are you working on something useful, or are you just trying to break things?

(3) By Simon Slavin (slavin) on 2021-06-11 14:27:26 in reply to 1 updated by 3.1 [link] [source]

This is a common result of how the compilers allocate memory and optimise memory use.  Small differences in memory use between compilers, even between different versions of the same compiler, do not indicate any fault.

You are especially likely to see this using <code>-DSSQLITE_DEBUG=1</code> because it includes many <code>assert</code>ions and the compiler doesn't expect to see <code>assert</code> in production code.

If one compiler uses 2,048 bytes more than another, then you *might* have found a bug in the compiler.

(3.1) By Simon Slavin (slavin) on 2021-06-12 09:33:13 edited from 3.0 in reply to 1 updated by 3.2 [link] [source]

This is a common result of how the compilers allocate memory and optimise memory use.  Small differences in memory use between compilers, even between different versions of the same compiler, do not indicate any fault.

You are especially likely to see this using <code>-DSSQLITE_DEBUG=1</code> because it includes many <code>assert</code>ions and the compiler doesn't expect to see <code>assert</code> in production code.

If one compiler uses more than 1,024 bytes more than another, then you *might* have found a bug in the compiler.

(3.2) By Simon Slavin (slavin) on 2021-06-12 09:34:55 edited from 3.1 in reply to 1 updated by 3.3 [source]

Small differences are  a common result of how the compilers allocate memory and optimise memory use.  Small differences in memory use between compilers, even between different versions of the same compiler, do not indicate any fault.

You are especially likely to see this using <code>-DSSQLITE_DEBUG=1</code> because it includes many <code>assert</code>ions and the compiler doesn't expect to see <code>assert</code> in production code.

If one difference is more than 1,024 bytes, you *might* have found a bug in one of the compilers.

(3.3) By Simon Slavin (slavin) on 2021-06-12 09:38:58 edited from 3.2 in reply to 1 [link] [source]

Small differences are a common result of how the compilers allocate memory and optimise memory use. Small differences in memory-use between compilers, even between different versions of the same compiler, do not indicate any fault.

You are especially likely to see this using -DSSQLITE_DEBUG=1 because it includes many assertions and the compiler doesn't expect to see assert in production code, so it doesn't bother optimising the code around it.

If one difference is more than 1,024 bytes, you might have found a bug in one of the compilers.

If you want to explore these differences, you might set compiler options to turn optimisation off on both of them, and see whether there's still a difference in the results they generate.