Reproducible builds
(1) By Nuno Cruces (ncruces) on 2025-02-27 12:52:22 [source]
Hi!
Is it expected/acceptable that the output of the tool/mkpragmatab.tcl script is platform dependent?
I don't know exactly when this started happening, if it's caused by the recent autoconf to autosetup change, but I'm getting a different amalgamation generated on at least macOS and Linux.
I can workaround this, if it is expected, I'm just asking, as I personally value reproducibility.
For those curious, apparently a "map" gets constructed differently, causing the following diff (with corresponding changes on usage, which cancel out the change, so everything works out). Maybe because different TCL implementations get used?
/* 30 */ "enc",
/* 31 */ "narg",
/* 32 */ "flags",
- /* 33 */ "tbl", /* Used by: stats */
- /* 34 */ "idx",
- /* 35 */ "wdth",
- /* 36 */ "hght",
- /* 37 */ "flgs",
- /* 38 */ "seq", /* Used by: index_list */
- /* 39 */ "name",
- /* 40 */ "unique",
- /* 41 */ "origin",
- /* 42 */ "partial",
+ /* 33 */ "seq", /* Used by: index_list */
+ /* 34 */ "name",
+ /* 35 */ "unique",
+ /* 36 */ "origin",
+ /* 37 */ "partial",
+ /* 38 */ "tbl", /* Used by: stats */
+ /* 39 */ "idx",
+ /* 40 */ "wdth",
+ /* 41 */ "hght",
+ /* 42 */ "flgs",
/* 43 */ "table", /* Used by: foreign_key_check */
/* 44 */ "rowid",
/* 45 */ "parent",
/* 46 */ "fkid",
+ /* 47 */ "busy", /* Used by: wal_checkpoint */
+ /* 48 */ "log",
+ /* 49 */ "checkpointed",
+ /* 50 */ "seq", /* Used by: database_list */
+ /* 51 */ "name",
+ /* 52 */ "file",
/* index_info reuses 21 */
- /* 47 */ "seq", /* Used by: database_list */
- /* 48 */ "name",
- /* 49 */ "file",
- /* 50 */ "busy", /* Used by: wal_checkpoint */
- /* 51 */ "log",
- /* 52 */ "checkpointed",
- /* collation_list reuses 38 */
/* 53 */ "database", /* Used by: lock_status */
/* 54 */ "status",
- /* 55 */ "cache_size", /* Used by: default_cache_size */
+ /* collation_list reuses 33 */
/* module_list pragma_list reuses 9 */
- /* 56 */ "timeout", /* Used by: busy_timeout */
+ /* 55 */ "timeout", /* Used by: busy_timeout */
+ /* 56 */ "cache_size", /* Used by: default_cache_size */
};
/* Definitions of all built-in pragmas */
(2) By Richard Hipp (drh) on 2025-02-27 12:56:23 in reply to 1 [link] [source]
The "sqlite3.c" and "sqlite3.h" files should be byte-for-byte identical regardless of the build platform.
Your theory that it depends on the TCL implementation seems like a likely explanation. Probably there needs to be an "lsort" someplace in that TCL script.
(3) By Richard Hipp (drh) on 2025-02-27 13:03:11 in reply to 1 [link] [source]
I'm having trouble recreating the problem. Can you give a precise description of how you managed to generate different pragma.h files?
(4) By Donal Fellows (dkfellows) on 2025-02-27 15:06:33 in reply to 1 [link] [source]
This appears to be due to some sort of change to the behaviour of lsort in relation to two items that are technically equal. That would be a serious regression, as the sort algorithm is supposed to be stable (and the input data should be equivalent in all cases, as it comes from processing a string in order)!
What version of Tcl is in use in the various cases that vary from one platform to another? If someone's decided to override the guts of Tcl to "use qsort() because that's good enough" then I'll be very upset, but if it's due to using a non-standard Tcl that has gone that route then basically all bets are off (and you'll need to do a bit of detective work to identify who to report the bug to).
(5) By Richard Hipp (drh) on 2025-02-27 15:11:32 in reply to 4 [link] [source]
The "jimsh0" TCL-clone that is used now (so as to not have a build dependency on TCL - the jimsh0 sources are entirely in-tree) does appear to use qsort.
(6) By Donal Fellows (dkfellows) on 2025-02-27 15:15:36 in reply to 5 [link] [source]
Well, that could well do it; qsort() has no guarantee of order when dealing with equal keys. End of my panic. :-)
I'm guessing it depends on what libc chooses to do, and that would explain why it's awkward to reproduce as it could even use a random number to select behaviour. I guess the implementation needs to use the original order of keys as a secondary key for the compare-equal case. That would at least be relatively easy to implement.
(7) By Nuno Cruces (ncruces) on 2025-02-27 15:37:37 in reply to 3 [link] [source]
I'm sorry, it'll take me a bit to reproduce the issue, esp. because it originally shows up in me checking out a branch and building the amalgamation from that. But now that I know it's an issue you want fixed, I'll get back to you eventually.
(8) By Richard Hipp (drh) on 2025-02-27 15:41:11 in reply to 4 [link] [source]
My proposed patch is shown below. Does this address the problem you see, Donal?
I was trying to test it when I came across another issue which needs to take priority so I'll get back to this later today...
CHANGED tool/mkpragmatab.tcl
--- tool/mkpragmatab.tcl
+++ tool/mkpragmatab.tcl
@@ -526,14 +526,17 @@
puts $fd [format {#define PragFlg_%-10s 0x%02x /* %s */} \
$f $fv $flagMeaning($f)]
set fv [expr {$fv*2}]
}
-# Sort the column lists so that longer column lists occur first
+# Sort the column lists so that longer column lists occur first.
+# In the event of a tie, sort column lists lexicographically.
#
proc colscmp {a b} {
- return [expr {[llength $b] - [llength $a]}]
+ set rc [expr {[llength $b] - [llength $a]}]
+ if {$rc} {return $rc}
+ return [string compare $a $b]
}
set cols_list [lsort -command colscmp $cols_list]
# Generate the array of column names used by pragmas that act like
# queries.
(9) By Richard Hipp (drh) on 2025-02-27 16:09:53 in reply to 1 [link] [source]
A change to try to address this has been checked in. Please try the latest code and report back whether or not this solved the problem. Thanks for the report. Thanks to Donal for spotting this issue!
(10) By Nuno Cruces (ncruces) on 2025-02-27 23:27:22 in reply to 9 [link] [source]
That fixes it, thanks.
I tried that checkin and trunk (f50c2148). Both builds reproduced across platforms. Thanks again!