We have started to see the test suite fail after version 3.51.3 in Gentoo Linux package. The issue is tracked in our downstream bug report: https://bugs.gentoo.org/971341, we did some digging but have not been able to discover the root cause of this issue:

>>> Test phase: dev-db/sqlite-3.51.3
 * abi_x86_64.amd64: running multilib-minimal_abi_src_test
make tclextension 
echo '#ifndef USE_SYSTEM_SQLITE' >tclsqlite3.c
cat sqlite3.c >>tclsqlite3.c
echo '#endif /* USE_SYSTEM_SQLITE */' >>tclsqlite3.c

(...)


make -Onone test 
/usr/bin/tclsh8.6 /var/tmp/portage/dev-db/sqlite-3.51.3/work/sqlite-src-3510300-abi_x86_64.amd64/tool/srctree-check.tcl
x86_64-pc-linux-gnu-gcc -Os -pipe -march=native -fno-diagnostics-color -o srcck1 /var/tmp/portage/dev-db/sqlite-3.51.3/work/sqlite-src-3510300-abi_x86_64.amd64/tool/srcck1.c
./srcck1 sqlite3.c
/usr/bin/tclsh8.6 /var/tmp/portage/dev-db/sqlite-3.51.3/work/sqlite-src-3510300-abi_x86_64.amd64/test/testrunner.tcl mdevtest 
Cannot find tcl package sqlite3: Trying to build it now...
Unable to build the SQLite tcl extension
Cannot find a working instance of the SQLite tcl extension.
Run "make tclextension" or "make testfixture" and try again...
make: *** [/var/tmp/portage/dev-db/sqlite-3.51.3/work/sqlite-src-3510300-abi_x86_64.amd64/main.mk:1855: devtest] Error 1

Initially I could not reproduce this issue (or could not hit it in every environment I've tried), but based on the error message adding make tclextension command prior to running the test suite seemed like logical thing to do, but that didn't resolve the issue. I can now consistently reproduce the issue (both with 3.51.3 and 3.53.0) when building sqlite with icu support (--with-icu-config).

Fellow dev sam@gentoo.org has done some debugging in https://bugs.gentoo.org/971341#c2 through https://bugs.gentoo.org/971341#c10 (I can paste the details if needed but won't do yet to keep the post concise). Complete build log is available at https://bugs.gentoo.org/attachment.cgi?id=957194. I'm also happy to test things out or provide any additional info if it is related to our sqlite build recipe.

Thanks in advance!

I couldn't see the build log on the link provided. Do we need special permissions for that?

I'm not the OP, but it downloads fine for me.

https://bugs.gentoo.org/attachment.cgi?id=957194

It's an xz compressed file.

Checking for ICU support...
Checking for pkg-config...2.5.1
Checking for icu-io ...78.2
Enabling ICU support with flags: -licuio -licui18n -licuuc

... -shared -fPIC -Wl,-soname,libsqlite3.51.3.so -DUSE_TCL_STUBS tclsqlite3.c -o libsqlite3.51.3.so -ltclstub8.6

no -licuio -licui18n -licuuc?

We link the CLI shell ("sqlite3" binary) and libsqlite3.so with ICU. We don't link, and have never linked (insofar as my source tree archeology can determine), tclsqlite3 with ICU. The copy of libsqlite3.so for the tcl extension is distinctly different from the canonical libsqlite3.so, and does not get the same link flags.

This is arguably a bug but begs the question: has this ever worked for you are are you trying something new and just now finding that it doesn't work? If it indeed worked for you at some point then "citation needed": which version(s) it did work, so that i can diff those against the versions i've looked at so far.

For sanity's sake:

$ ./configure --with-icu-config --prefix=$HOME --with-tcl=$HOME/tcl/8.6 --all
...
$ m tclextension-install
installing /home/stephan/tcl/8.6/lib/tcl8.6/sqlite3.54.0/pkgIndex.tcl
installing /home/stephan/tcl/8.6/lib/tcl8.6/sqlite3.54.0/libsqlite3.54.0.so

$ ~/tcl/8.6/bin/tclsh8.6 
% package require sqlite3
couldn't load file "/home/stephan/tcl/8.6/lib/tcl8.6/sqlite3.54.0/libsqlite3.54.0.so": /home/stephan/tcl/8.6/lib/tcl8.6/sqlite3.54.0/libsqlite3.54.0.so: undefined symbol: ubrk_next_74

# Now repeat all of that without the --with-icu-config flag:

$ ~/tcl/8.6/bin/tclsh8.6 
% package require sqlite3
3.54.0

That's not to say we can't/shouldn't make the tcl extension work with ICU, but i would be terribly surprised if it ever did with an unpatched canonical source tree.

That's not to say we can't/shouldn't make the tcl extension work with ICU,

A patch for this is now checked in but is not yet on trunk, pending testing across more systems than this one workstation:

$ ./configure --with-icu-config --prefix=$HOME --with-tcl=$HOME/tcl/8.6 && m tclextension-install
...

$ ldd /home/stephan/tcl/8.6/lib/tcl8.6/sqlite3.54.0/libsqlite3.54.0.so
...
	libicui18n.so.74 => /lib/x86_64-linux-gnu/libicui18n.so.74 (0x000070810d400000)
	libicuuc.so.74 => /lib/x86_64-linux-gnu/libicuuc.so.74 (0x000070810d000000)
	libicudata.so.74 => /lib/x86_64-linux-gnu/libicudata.so.74 (0x000070810aa00000)

$ ~/tcl/8.6/bin/tclsh8.6 
% package require sqlite3
3.54.0

It's not yet decided whether this will/should be merged into version 3.53. We have a policy of "bug fixes only" in release branches and this isn't strictly a bug fix.

Cannot find a working instance of the SQLite tcl extension

Can you please try this again against version src:404fde8bac or newer, and let us know if that resolves the issue? The tcl extension now includes ICU on platforms for which --with-icu-config works.

I can confirm that https://sqlite.org/src/vpatch?from=46a288e43e4c48fc&to=404fde8bacc3f052 applied to version 3.53.0 fixes the issue. I'll ask one more person on our side to test it out to be on the safe side.

To summarize my last comment from the Gentoo bug report: the original build log from the bug was from a system configured to run multilib and the failure occured during testing of the 32-bit build.

The latest known good version (3.51.2) passes successfully both for 64 and 32 bit builds. With version 3.53.0 64-bit build passed while 32-bit still fails (with the patch applied):

make -j12 -l12 -Onone test 
/usr/bin/tclsh8.6 /var/tmp/portage/dev-db/sqlite-3.53.0/work/sqlite-src-3530000-abi_x86_32.x86/tool/srctree-check.tcl
x86_64-pc-linux-gnu-gcc -march=native -O2 -pipe -mfpmath=sse -o srcck1 /var/tmp/portage/dev-db/sqlite-3.53.0/work/sqlite-src-3530000-abi_x86_32.x86/tool/srcck1.c
./srcck1 sqlite3.c
/usr/bin/tclsh8.6 /var/tmp/portage/dev-db/sqlite-3.53.0/work/sqlite-src-3530000-abi_x86_32.x86/test/testrunner.tcl mdevtest 
Cannot find tcl package sqlite3: Trying to build it now...
Unable to build the SQLite tcl extension
Cannot find a working instance of the SQLite tcl extension.
Run "make tclextension" or "make testfixture" and try again...
make: *** [/var/tmp/portage/dev-db/sqlite-3.53.0/work/sqlite-src-3530000-abi_x86_32.x86/main.mk:1847: devtest] Error 1

Did you try with the latest trunk check-in? Did the changes that Stephan made to the ./configure and Makefile help?

Did the changes that Stephan made to the ./configure and Makefile help?

He notes "with the patch applied". "With multilib" means nothing to me, though, and is presumably part of the problem. i'm currently at a loss and don't have any 32-bit environments set up to try to reproduce this with, but getting a VM up and figuring out gentoo is on my list of TODOs.

@OP just to confirm:

1. it works fine without --with-icu-config? (To reiterate: our tcl build has never utilized that, which makes me suspect that ICU is a red herring.)
2. the exact same build option previously worked at some point before?
3. You are 100% certain that tcl is picking up the correct library?
4. What on earth is multilib? Our build knows of no such thing.

(typing on a phone, please pardon brevity and typos)

... system configured to run multilib and the failure occured during testing of the 32-bit build.

After having looked up "multilib", i make no promises except that this is very unlikely to be resolved without active support from someone familiar with an affected environment.

What does one need, in terms of a Gentoo installation, to reproduce this? Even better, rather than having to install a VM from scratch¹, would be a pre-built VM or ssh access to an affected system. If you can arrange access to either of those, please contact us via support @ this domain.

The latest known good version (3.51.2) passes successfully both for 64 and 32 bit builds. With version 3.53.0 64-bit build passed while 32-bit still fails (with the patch applied):

Our diffs between those versions:

• Full diff https://wasm-testing.sqlite.org/tmp/etilqs_ea924f432ced5910.html
• Only Unix-relevant build files https://wasm-testing.sqlite.org/tmp/etilqs_85bc4c178aa398ad.html

Reveal that how the tcl extension is build differs slightly, but not (to my eyes) materially, between those versions, and certainly not in any way which should trigger an ICU-related linking error because neither of those versions propagate the ICU build settings to tcl.

1. ^{^} Edit: tried that this morning but Gentoo installation, like Oracle Db administration, requires the user to commit to making it a Lifestyle Choice, placing it far out of my current reach bandwidth-wise.

After having looked up "multilib"

Apologies for not making it clear in the beginning, I was not aware that the term is not widely used outside of Gentoo; in short it's a feature that allows users to compile applications for different ABIs on a single system (can be useful for some desktop apps that need 32-bit libraries, e.g. wine, steam, etc.). Not all packages have support for this, but sqlite does given how many other packages depend on it.

What does one need, in terms of a Gentoo installation, to reproduce this?

It's not necessary to setup a full VM if you don't care about testing boot/kernel stuff, although since recently we do provide a complete iso image: https://www.gentoo.org/downloads/amd64/#media

For simpler compile tests, a chroot/systemd-nspawn or Docker container is enough. I can provide a Dockerfile that can be used to quickly reproduce this issue:

FROM gentoo/stage3:systemd-20260413

RUN emerge-webrsync

RUN CORES=$(nproc) && \
    echo "MAKEOPTS=\"-j${CORES}\"" >> /etc/portage/make.conf

RUN echo 'ABI_X86="32 64"' >> /etc/portage/make.conf

# install all dependencies
RUN emerge --onlydeps --with-test-deps --jobs=4 dev-db/sqlite

# good version
RUN FEATURES="test" emerge '=dev-db/sqlite-3.51.2'

# bad version
RUN ACCEPT_KEYWORDS="~amd64" FEATURES="test" emerge '=dev-db/sqlite-3.51.3' || true

WORKDIR /var/tmp/portage/dev-db/sqlite-3.51.3/work/sqlite-src-3510300-abi_x86_32.x86

CMD ["/bin/bash"]

You can build the image with docker build -t gentoo-sqlite -f Dockerfile . --progress=plain and run it with docker run -it gentoo-sqlite that should take you to the working build directory.

I tried to look into this more and it seems that this is a separate issue from the previous one and is not connected to the ICU compile flag as it can be reproduced without it:

/var/tmp/portage/dev-db/sqlite-3.51.3/work/sqlite-src-3510300-abi_x86_32.x86 # tclsh
% set dir [pwd]
/var/tmp/portage/dev-db/sqlite-3.51.3/work/sqlite-src-3510300-abi_x86_32.x86
% global dir
% source pkgIndex.tcl
% package require sqlite3
couldn't load file "/var/tmp/portage/dev-db/sqlite-3.51.3/work/sqlite-src-3510300-abi_x86_32.x86/libsqlite3.51.3.so": /var/tmp/portage/dev-db/sqlite-3.51.3/work/sqlite-src-3510300-abi_x86_32.x86/libsqlite3.51.3.so: wrong ELF class: ELFCLASS32

# file libsqlite3.51.3.so
libsqlite3.51.3.so: ELF 32-bit LSB shared object, Intel i386, version 1 (SYSV), dynamically linked, BuildID[sha1]=b43e408c79ae536f759a8a6ddd8861eb40b8dd1e, not stripped

# file $(which tclsh8.6)
/usr/sbin/tclsh8.6: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=21e0358e6633251bf7c9f6ccae89d4e57366649d, for GNU/Linux 3.2.0, stripped

It seems like 64-bit process is trying to load a 32-bit library. That does not happen in 3.51.2, my best guess is because we used the old test target test: srctree-check fuzztest sourcetest $(TESTPROGS) tcltest and the new one uses tclsh to drive the test suite.

Right. If you want to run the tests with -m32, then you have three choices:

• Also build and install tclsh using -m32

• First run "sudo make tclextension-install" without -m32, so that there is a 64-bit version of the SQLite TCL extension available to tclsh, and then do your build.

• Instead of just "make test", run "make testfixture test"

I don't think we, the SQLite devs, want to support running tests for a 32-bit build of SQLite on a 64-bit platform using 64-bit tooling (tclsh). If you want to do that, fine; the last two suggests above should work to solve the issue. That this worked for you before without work-arounds was a happy accident.

I believe that the third suggestion above will work, but I cannot promise that it will continue to work in the future. We won't deliberately try to break it, but in our unending quest to improve the testing (and hence reliability) of SQLite, it might accidentally get broken in the future, and it is not something that we test.

Instead of just "make test", run "make testfixture test"

This resulted in the same error unfortunately.

I don't think we, the SQLite devs, want to support running tests for a 32-bit build of SQLite on a 64-bit platform using 64-bit tooling (tclsh)

I'd say that's perfectly reasonable, and it makes sense to have to use 32-bit tclsh for the 32-bit build. In Gentoo's case specifically, I'm not sure how much work would be to properly add this change into the tcl package itself, but we can still continue to call the old test targets (srctree-check fuzztest sourcetest $(TESTPROGS) tcltest) when testing sqlite for non-native ABI, so that's probably the route I'm going to take to fix this in our package.

In any case I'd say there's nothing actionable here on your end anymore. I appreciate the detailed explanations and responsiveness of the Sqlite team as always. Thank you once again :)