Good evening, folks,

As Richard just posted in the "announce" list for the 3.47.0 release:

During the next release cycle, we are planning to do an overhaul of the "configure; make" system for SQLite. This is intended to be an improvement. We will try really, really hard not to break anything. But because we don't know all the creative ways that people are currently using "configure; make" now, our changes will be impossible to test thoroughly. We would appreciate your help in trying out the new "configure; make" found in periodic pre-release snapshots that we will make available. Please do not wait until the next official release to discover that we have accidentally broken your particular build, and then complain. Better to find these problems early.

To elaborate on that...

That work is currently going on in the autosetup branch, and ← that link has a description of the motivation for this migration, as well as the perceived benefits and down-sides.

For those of you who simply run "./configure && make", very little, if anything, will change. The parts of the build which most folks use day-to-day are already migrated to the newer system, but there are the obligatory pending TODOs and FIXMEs, mostly summarized at the top of Makefile.in.

This change will require some adaptation on some folks' automated build processes. i.e. some build-level breakage is unavoidable. We are making every effort to avoid gratuitous breakage, though.

Please alert us to any new breakage here in the forum.

Sidebar: these changes only apply to platforms which use the "configure" process, i.e. Unix-like environments. Vanilla Windows builds are unaffected and we will continue to provide static makefiles for Windows.

I need to know the various -DHAVE_ values because I am compiling SQLite with Python's build system not SQLite's. Currently I do it by running configure and parsing the generated Makefile and looking for the DEFS = line.

Please include a way of getting them. I'm happy to change my code.

Currently I do it by running configure and parsing the generated Makefile and looking for the DEFS = line.

FWIW, we're currently doing something similar in the ext/wasm build and i hate that ;).

In the off chance that you don't already know this: more reliable than parsing the makefile is using GNU make's -n and -p flags, like so:

$ make -pn | grep -e '^SHELL_OPT'
SHELL_OPT = -DSQLITE_DQS=0 -DSQLITE_ENABLE_FTS4 -DSQLITE_ENABLE_RTREE -DSQLITE_ENABLE_EXPLAIN_COMMENTS -DSQLITE_ENABLE_UNKNOWN_SQL_FUNCTION -DSQLITE_ENABLE_STMTVTAB -DSQLITE_ENABLE_DBPAGE_VTAB -DSQLITE_ENABLE_DBSTAT_VTAB -DSQLITE_ENABLE_BYTECODE_VTAB -DSQLITE_ENABLE_OFFSET_SQL_FUNC -DSQLITE_STRICT_SUBTYPE=1

• -p = print the make "database"
• -n = don't build anything

BSD make has -n but not -p, but i only this moment learned that BSD make has something arguably more interesting than -p:

$ bmake -V SHELL_OPT
-DSQLITE_DQS=0 -DSQLITE_ENABLE_FTS4 -DSQLITE_ENABLE_RTREE -DSQLITE_ENABLE_EXPLAIN_COMMENTS -DSQLITE_ENABLE_UNKNOWN_SQL_FUNCTION -DSQLITE_ENABLE_STMTVTAB -DSQLITE_ENABLE_DBPAGE_VTAB -DSQLITE_ENABLE_DBSTAT_VTAB -DSQLITE_ENABLE_BYTECODE_VTAB -DSQLITE_ENABLE_OFFSET_SQL_FUNC -DSQLITE_STRICT_SUBTYPE=1

Please include a way of getting them.

Which var(s) are you looking for? We don't have any named "DEFS" (unless they're part of the Windows build, but those aren't affected by these changes).

... is using GNU make's ...

I don't run make at any point, nor do I know or have any control over what make is present. My stuff is packaged by third parties on many different Linuxes, BSD, Android etc, although I only do this when SQLite is directly bundled into my library.

I do the extraction using Python code.

We don't have any named "DEFS"

You do, and I've been using them for over a decade! Here is what is in Makefile after running configure on Linux

DEFS = -DPACKAGE_NAME=\"sqlite\" -DPACKAGE_TARNAME=\"sqlite\" -DPACKAGE_VERSION=\"3.47.0\" -DPACKAGE_STRING=\"sqlite\ 3.47.0\" -DPACKAGE_BUGREPORT=\"http://www.sqlite.org\" -DPACKAGE_URL=\"\" -DPACKAGE=\"sqlite\" -DVERSION=\"3.47.0\" -DHAVE_STDIO_H=1 -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 -DHAVE_INTTYPES_H=1 -DHAVE_STDINT_H=1 -DHAVE_STRINGS_H=1 -DHAVE_SYS_STAT_H=1 -DHAVE_SYS_TYPES_H=1 -DHAVE_UNISTD_H=1 -DSTDC_HEADERS=1 -DHAVE_DLFCN_H=1 -DLT_OBJDIR=\".libs/\" -DHAVE_FDATASYNC=1 -DHAVE_USLEEP=1 -DHAVE_LOCALTIME_R=1 -DHAVE_GMTIME_R=1 -DHAVE_DECL_STRERROR_R=1 -DHAVE_STRERROR_R=1 -DHAVE_READLINE_READLINE_H=1 -DHAVE_READLINE=1 -DHAVE_POSIX_FALLOCATE=1 -DHAVE_ZLIB_H=1

I create a file in the same directory as sqlite3 named sqlite3config.h and then #include that in the build before #include "sqlite3.c". The above turns into this:

    /* This file was generated by parsing how configure altered the Makefile
    which isn't used when building python extensions.  It is specific to the
    machine and developer components on which it was run. */
    
#define HAVE_STDIO_H 1
#define HAVE_STDLIB_H 1
#define HAVE_STRING_H 1
#define HAVE_INTTYPES_H 1
#define HAVE_STDINT_H 1
#define HAVE_STRINGS_H 1
#define HAVE_SYS_STAT_H 1
#define HAVE_SYS_TYPES_H 1
#define HAVE_UNISTD_H 1
#define HAVE_DLFCN_H 1
#define HAVE_FDATASYNC 1
#define HAVE_USLEEP 1
#define HAVE_LOCALTIME_R 1
#define HAVE_GMTIME_R 1
#define HAVE_DECL_STRERROR_R 1
#define HAVE_STRERROR_R 1
#define HAVE_READLINE_READLINE_H 1
#define HAVE_READLINE 1
#define HAVE_POSIX_FALLOCATE 1
#define HAVE_ZLIB_H 1

You do, and I've been using them for over a decade! Here is what is in Makefile after running configure on Linux

That's not what i'm seeing from the current trunk:

$ uname -a
Linux nuc 6.8.0-41-generic ...

$ ./configure --enable-all
...

$ grep -w DEFS Makefile
<no output>

$ grep -w DEFS * 2>/dev/null
config.log:DEFS='-DHAVE_CONFIG_H'
config.status:S["DEFS"]="-DHAVE_CONFIG_H"
configure:DEFS
configure:# confdefs.h avoids OS command line length limits that DEFS can exceed.
configure:DEFS=-DHAVE_CONFIG_H

That's not what i'm seeing from the current trunk:

Correct. It is what you get from the release downloads like 3.47.0 which matters in this case.

During development when I work with fossil I keep a copy of the generated file around.

Correct. It is what you get from the release downloads like 3.47.0 which matters in this case.

Those artifacts are generated by the autotools, so will de facto be lost in this port, but i have noted a TODO for some way to extract those or, similarly, specifically emit them to their own file(s) as part of the configure step, perhaps even as python/tcl/shell/json/whatever code.

That works for me. JSON preferred. Other formats like KEY = VALUE are rife with problems such as determining what are numbers, what are strings, unintentionally converting no to a boolean etc.

Steve Bennett here (author of autosetup)

make-config-header can generate a nice easy-to parse header file that can be used independently of the make-based build.

make-config-header can generate a nice easy-to parse header file that can be used independently of the make-based build.

We're using that to generate sqlite_cfg.h and a debug-only dump of all of the build defines, so it would not be a far stretch for us to do the same to cover Roger's case. Ideally i'd rather emit that as something portable, so will probably fork a copy of make-config-header and dump that all out as JSON (and maybe other formats, like tcl, shell, and python, simply because it would be trivial to do at that point).

As Richard just posted in the "announce" list for the 3.47.0 release:

As of moments ago, that is now in the SQLite trunk. There is some cleanup to be done before the next release but, by and large, the build works as before. A few ./configure flags had to be renamed for $REASONS (described in the new auto.def file), a few still need to be implemented, and a few are no longer relevant, but enough of the functionality is in place to do day-to-day project maintenance.

Vanilla Windows builds are unaffected and we will continue to provide static makefiles for Windows.

That, it turns out, wasn't quite true: Richard managed to get JimTCL building with MSVC, which means that folks using that build environment can now build from the canonical sources without requiring a separate TCL installation. Canonical TCL is still needed (on all platforms) for the testing tools, but the in-tree copy of JimTCL is sufficient for the numerous code-generation tasks which formerly required a canonical TCL installation.

On cygwin under Windows 8.1 I get the following:

....
rm tsrc/sqlite.h.in tsrc/parse.y
./jimsh /home/mark/src/sqlite-trunk/tool/vdbe-compress.tcl  <tsrc/vdbe.c >vdbe.new
mv vdbe.new tsrc/vdbe.c
cp fts5.c fts5.h tsrc
touch .target_source
make: *** No rule to make target 'src-verify', needed by 'sqlite3.c'.  Stop.

make: *** No rule to make target 'src-verify', needed by 'sqlite3.c'. Stop.

That's now fixed. Thank you for the report.

Hi Mark.

Do you do any cross-compilation from cygwin to Windows to create a standalone Windows sqlite3.exe?

Do you do any cross-compilation from cygwin to Windows to create a standalone Windows sqlite3.exe?

FWIW, if this new build fixes the "immediate exit" you're seeing, it would be a inexplicable miracle, as there's nothing we know about in the build process which accounts for that.

Re. cross-compilation in general: the new tools support it out of the box, but that aspect of the build is not yet tested, simply for lack of readily-available-to-me cross-compilation options. It is most definitely a high-priority todo, though.

FWIW, if this new build fixes the "immediate exit" you're seeing, it would be a inexplicable miracle...

The inexplicable miracle has not happened. :-) This is with the latest check-in (e5f1a01f):

$  ./configure --enable-all --prefix=/usr --disable-tcl && make install && i686-w64-mingw32-gcc -shared -DFOR_WIN10 -DSQLITE_ENABLE_FTS5 -static-libgcc sqlite3.c -o sqlite3.dll && x86_64-w64-mingw32-gcc -DSQLITE_ENABLE_FTS5 -DFOR_WIN10 shell.c -o sqlite3.exe sqlite3.c
Host System...x86_64-pc-cygwin
Build System...x86_64-pc-cygwin
C compiler... cc
C++ compiler... c++
Build C compiler...cc
Checking for stdlib.h...ok
srcdir = /home/E608313/b/sqlite/SQLite-e5f1a01f
top_srcdir = /home/E608313/b/sqlite/SQLite-e5f1a01f
Configuring SQLite version 3.48.0
Looking for install ... /usr/bin/install
Checking for sys/types.h...ok
Checking if -D_FILE_OFFSET_BITS=64 is needed...no
...
[clip]
...
-e5f1a01f/ext/rtree/geopoly.c /home/E608313/b/sqlite/SQLite-e5f1a01f/ext/session/sqlite3session.c /home/E608313/b/sqlite/SQLite-e5f1a01f/ext/session/sqlite3session.h /home/E608313/b/sqlite/SQLite-e5f1a01f/ext/rbu/sqlite3rbu.h /home/E608313/b/sqlite/SQLite-e5f1a01f/ext/rbu/sqlite3rbu.c /home/E608313/b/sqlite/SQLite-e5f1a01f/ext/misc/stmt.c keywordhash.h opcodes.c opcodes.h parse.c parse.h sqlite_cfg.h shell.c sqlite3.h tsrc
rm tsrc/sqlite.h.in tsrc/parse.y
./jimsh /home/E608313/b/sqlite/SQLite-e5f1a01f/tool/vdbe-compress.tcl  <tsrc/vdbe.c >vdbe.new
mv vdbe.new tsrc/vdbe.c
cp fts5.c fts5.h tsrc
touch .target_source
cc -g -o src-verify.exe /home/E608313/b/sqlite/SQLite-e5f1a01f/tool/src-verify.c
./jimsh /home/E608313/b/sqlite/SQLite-e5f1a01f/tool/mksqlite3c.tcl --linemacros=0
cp tsrc/sqlite3ext.h .
cp /home/E608313/b/sqlite/SQLite-e5f1a01f/ext/session/sqlite3session.h .
cc   -DSQLITE_ENABLE_FTS4 -DSQLITE_ENABLE_FTS5 -DSQLITE_ENABLE_GEOPOLY -DSQLITE_ENABLE_MATH_FUNCTIONS -DSQLITE_ENABLE_PREUPDATE_HOOK -DSQLITE_ENABLE_RTREE -DSQLITE_ENABLE_SESSION -DSQLITE_THREADSAFE=1   -DNDEBUG -D_HAVE_SQLITE_CONFIG_H -DBUILD_sqlite -I/usr/include -I. -I/home/E608313/b/sqlite/SQLite-e5f1a01f/src -I/home/E608313/b/sqlite/SQLite-e5f1a01f/ext/rtree -I/home/E608313/b/sqlite/SQLite-e5f1a01f/ext/icu -I/home/E608313/b/sqlite/SQLite-e5f1a01f/ext/fts3 -I/home/E608313/b/sqlite/SQLite-e5f1a01f/ext/session -I/home/E608313/b/sqlite/SQLite-e5f1a01f/ext/misc -DSQLITE_TEMP_STORE=1 -c sqlite3.c
cc   -DSQLITE_ENABLE_FTS4 -DSQLITE_ENABLE_FTS5 -DSQLITE_ENABLE_GEOPOLY -DSQLITE_ENABLE_MATH_FUNCTIONS -DSQLITE_ENABLE_PREUPDATE_HOOK -DSQLITE_ENABLE_RTREE -DSQLITE_ENABLE_SESSION -DSQLITE_THREADSAFE=1   -DNDEBUG -D_HAVE_SQLITE_CONFIG_H -DBUILD_sqlite -I/usr/include -I. -I/home/E608313/b/sqlite/SQLite-e5f1a01f/src -I/home/E608313/b/sqlite/SQLite-e5f1a01f/ext/rtree -I/home/E608313/b/sqlite/SQLite-e5f1a01f/ext/icu -I/home/E608313/b/sqlite/SQLite-e5f1a01f/ext/fts3 -I/home/E608313/b/sqlite/SQLite-e5f1a01f/ext/session -I/home/E608313/b/sqlite/SQLite-e5f1a01f/ext/misc  -shared -o libsqlite3.dll sqlite3.o -Wl,-rpath,/usr/lib    -lz
/usr/bin/install libsqlite3.dll "/usr/lib"
Setting up SO symlinks...
lrwxrwxrwx 1 E608313 Domain Users      16 Oct 30 16:12 libsqlite3.dll -> libsqlite3.dll.3
lrwxrwxrwx 1 E608313 Domain Users      21 Oct 30 16:12 libsqlite3.dll.3 -> libsqlite3.dll.3.48.0
-rwxr-xr-x 1 E608313 Domain Users 1797689 Oct 30 16:12 libsqlite3.dll.3.48.0
ar cr  libsqlite3.lib sqlite3.o
/usr/bin/install -m 0644 libsqlite3.lib "/usr/lib"
/usr/bin/install -m 0644 sqlite3.h "/home/E608313/b/sqlite/SQLite-e5f1a01f/src/sqlite3ext.h" "/usr/include"
cc   sqlite3.o   -o sqlite3
/usr/lib/gcc/x86_64-pc-cygwin/11/../../../../x86_64-pc-cygwin/bin/ld: /usr/lib/gcc/x86_64-pc-cygwin/11/../../../../lib/libcygwin.a(libcmain.o): in function `main':
/usr/src/debug/cygwin-3.5.3-1/winsup/cygwin/lib/libcmain.c:37:(.text.startup+0x7b): undefined reference to `WinMain'
collect2: error: ld returned 1 exit status
make: *** [<builtin>: sqlite3] Error 1

Which is not completing the make piece. I've been a bit busy with other stuff, so I am just working with the released v3.47.0 Windows downloadable version for now.

cc sqlite3.o -o sqlite3 ... undefined reference to `WinMain

This part is definitely wrong:

cc   sqlite3.o   -o sqlite3

sqlite3.o is the library amalgamation and cannot compile into an executable. Why it's trying to run that rule is currently a mystery to me. It's apparently an implicit rule, and it's definitely not one we've set up (ours all have a slew of flags).

Please post the result of this command after running configure:

$ grep '^[TB]\.exe' Makefile

That will tell us whether the build thinks that cygwin is expecting to see a ".exe" suffix on executables, According to the configure script, i'm expecting the above to say:

B.exe = .exe
T.exe = .exe

But perhaps it should not for cygwin? As a quick test, you can edit both of those lines in the generated Makefile to remove the ".exe" part, and see if that eliminates the build error. If it does, it's a simple config detection change.

As that OS not available in this household i'll be unable to reproduce this myself but...

I've been a bit busy with other stuff, so I am just working with the released v3.47.0 Windows downloadable version for now.

... any changes you can can suggest to get it building on cygwin would be very welcomed!

It fails to detect readline in /usr/local (various macOS versions, x86_64 and arm64).

It fails to detect readline in /usr/local (various macOS versions, x86_64 and arm64).

The readline detection is currently very primitive. The historical one was rather involved and porting it from shell to tcl has, so far, exceeded my ambitions. It is a definite TODO, though, and will be done in the near future.

Trunk also fails to detect readline in /usr under Cygwin, so almost Neanderthal then? :-)

A more serious issue is that the CLI built without libreadline has a problem parsing input after it receives terminal escape codes:

SQLite version 3.48.0 2024-10-26 04:31:04
Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.
sqlite>.q
]  ...> .q
]  ...> .q
]  ...> ;
]  ...> ;
]  ...> Program interrupted.
[1] ~/src/sqlite-trunk$

Note the lack of space above between sqlite> and the first .q, where I hit the left arrow key. From that point on the parser apparently fails to recognise any kind of new/end of line.

It would seem that the terminal is still configured in raw mode? Would it not make sense in this case to keep it cooked (normal) and let the shell send input line-at-a-time?

Trunk also fails to detect readline in /usr under Cygwin, so almost Neanderthal then?

We're aware of the current readline detection limitations. It currently only works reliably when either (A) pkg-config info for readline works on that platform or (B) both readline.h and libreadline.so/a can be found and used without any extra flags.

Proper detection of readline is difficult because, for one, it uses different client-side link flags even on different Linux systems. (Some want -lncurses, some don't. OpenBSD not only wants different linker flags (which its pkg-config won't tell us) but puts readline.h in some truly obscure place. etc. etc. etc.)

Bringing the readline detection to feature parity with autotools' detection of it is an ongoing thing.

It would seem that the terminal is still configured in raw mode? Would it not make sense in this case to keep it cooked (normal) and let the shell send input line-at-a-time?

That behavior is unrelated to this port and is a Windows-ism, so i won't personally be able to help you with that. If you'll open up a new thread about that, Larry or Richard may be able to assist with it. You can almost certainly reproduce the same behavior in 3.47 by configuring with --disable-readline.

It fails to detect readline in /usr/local (various macOS versions, x86_64 and arm64).

Similarly, neither pkg-config nor compile-with-defaults detection is working on OpenBSD (both x64 and ARM). As a brute-force measure we just added some new flags:

$ ./configure --help | grep readl
  --disable-readline               Disable readline support
  --with-readline-ldflags=         Readline LDFLAGS, e.g. -lreadline -lncurses
  --with-readline-cflags=          Readline CFLAGS, e.g. -I/path/to/includes
  --with-readline-header=          Full path to readline.h, from which --with-readline-cflags will be derived

Edit: these have been reverted back to their legacy names:

  --disable-readline               Disable readline support
  --with-readline-lib=LDFLAGS      Readline LDFLAGS, e.g. -lreadline -lncurses
  --with-readline-inc=CFLAGS       Readline CFLAGS, e.g. -I/path/to/includes
  --with-readline-header=PATH      Full path to readline.h, from which --with-readline-inc will be derived

because, it turns out, those had the same semantics as the new flags.

e.g. on OpenBSD on a Raspberry Pi:

./configure --with-readline-lib='-lreadline -lcurses' --with-readline-header=/usr/include/readline/readline.h

which is (on this system) equivalent to:

$ ./configure --with-readline-lib='-lreadline -lcurses' --with-readline-cflags=-I/usr/include

Improving the automatic detection of readline is a definite todo, and will be incremental, but it's proven to be a genuine beast because of its varying linker flags.

Edit: as far as readline detection/specification goes, the new configure script is believed to have reached parity with the legacy configure script. That is to say: it works on my systems where it previously failed and now uses the same lib/header search logic as the legacy configure script. The readline alternative --with-linenoise also works but --enable-editline has not yet been ported over.

Yep, works as expected now, picking up /usr/local/lib/libreadline.dylib and /usr/lib/libncurses.dylib

Thank you!

Again under Cygwin on Windows 8.1, using the trunk from about 10 minutes ago. The configure script warns about the following:

Checking for a suitable tcl...
Checking for tclsh9.0...no
Checking for tclsh8.6.../usr/bin/tclsh8.6
Using tclsh: /usr/bin/tclsh8.6
WARNING: /usr/bin/tclsh8.6 is unable to recommand a tclConfig.sh
WARNING: Unable to run tests because no tclConfig.sh file could be located
Using tclsh: /usr/bin/tclsh8.6
Which TCL to use for code generation...

The is apparently a tclConfig.sh:

~$ apt-cyg filelist tcl | grep tclConfig
tcl: /usr/lib/tcl8.6/tclConfig.sh

But I don't know how tclsh is supposed to report on that. Anything I could run manually to see what it's actually saying?

Later, after successfully building libsqlite.dll it bombs out on tclsqlite.c:

cc  -DSQLITE_THREADSAFE=1 -DSQLITE_ENABLE_MATH_FUNCTIONS    -DUSE_TCL_STUBS=1  -I. -I/home/mark/src/sqlite-trunk/src -I/home/mark/src/sqlite-trunk/ext/rtree -I/home/mark/src/sqlite-trunk/ext/icu -I/home/mark/src/sqlite-trunk/ext/fts3 -I/home/mark/src/sqlite-trunk/ext/async -I/home/mark/src/sqlite-trunk/ext/session -I/home/mark/src/sqlite-trunk/ext/misc -I/home/mark/src/sqlite-trunk/ext/userauth \
        -c /home/mark/src/sqlite-trunk/src/tclsqlite.c
/home/mark/src/sqlite-trunk/src/tclsqlite.c:42:11: fatal error: tcl.h: No such file or directory
   42 | # include <tcl.h>          /* All normal cases */
      |           ^~~~~~~
compilation terminated.
make: *** [/home/mark/src/sqlite-trunk/main.mk:1227: tclsqlite.o] Error 1

From the Tcl wiki I found the following:

~/src/sqlite-trunk$ tclsh
% package require Tcl 8.5
set d [::tcl::pkgconfig get libdir,install]
puts [file join $d tclConfig.sh]
exit8.6.8
% /usr/lib
% /usr/lib/tclConfig.sh

So I suspect the issue lies not on the SQLite side. Some more details:

~/src/sqlite-trunk$ ll /usr/lib/tclConfig.sh
ls: cannot access '/usr/lib/tclConfig.sh': No such file or directory
[2] ~/src/sqlite-trunk$ ll /usr/lib/tcl8.6/tclConfig.sh
lrwxrwxrwx 1 mark None 15 Mar 25  2018 /usr/lib/tcl8.6/tclConfig.sh -> ../tclConfig.sh

It turns out that /usr/lib/tclConfig.sh is provided by the tcl-devel package. With that installed the configure generates no warnings and the sqlite3 CLI tool runs normally.

However I understood that the new process was intended to work without a (full?) local Tcl? Perhaps it could be made a bit more robust?

However I understood that the new process was intended to work without a (full?) local Tcl? Perhaps it could be made a bit more robust?

Thank you for the investigation!

What you're seeing is actually expected: it's warning you that it cannot run the test suite. The embedded copy of JimTCL is sufficient for the code generation scripts, but the test suite requires the canonical TCL interpreter because it uses sqlite/tcl bindings, and those require a much different binding API than JimTCL has.

Prior to this migration, the full TCL was also needed for the code generation.

Edit: that said, it should not have attempted:

Later, after successfully building libsqlite.dll it bombs out on tclsqlite.c:

if tclConfig.sh was not found. That will be fixed soon.

Here's a bug:

If I configure with --disable-tcl, then make install fails with this complaint: TCLLIBDIR is not set.

If I configure with --disable-tcl, then make install fails with this complaint: TCLLIBDIR is not set.

Thank you for the report. The --disable-tcl flag is not being honored properly but that will be fixed sometime today. (Edit: fixed!)

After working with same script for months / years

There have been some changes in the configure script. I also had the same problem, amongst others, which some are still not working for me.

I also had the same problem, amongst others, which some are still not working for me.

"Among others" is unhelpfully vague. We can't fix problems we're unaware of.

It appeared to work but echoed the line ... which I don't normally get.

Before the build rewrite change, the compilation line was definitely part of the output (just tested it and it was much more verbose), it just didn't have the "+" prefix in front of it. The + prefix is a side effect of a change made yesterday in order to enable static/hand-written makefiles to work without requiring TCL-related information from the configure script. We need that capability for certain special-purpose, out-of-tree makefiles, as well as to support other folks who need makefiles which aren't filtered by the configure script.

(Sidebar: if we could depend on everyone having GNU make we wouldn't have needed that change to implement that particular capability, for reasons described in Makefile.in (keyword: tclconfig.make), but we're limited to POSIX make compatibility, which forces some solutions to be rather hackier than they would otherwise be.)

If the change of output is alarming, please don't let it be. Practically every line of output from the makefile will change, largely because libtool was much more verbose than plain compiler/linker invocations are.

Just wondering why it suddenly failed to find the tclConfig.sh file.

That looks like it was a misunderstanding on my part when porting those pieces from the legacy build. My misunderstanding was that tclConfig was only used if it was explicitly requested, but that's now how the 3.47 build does it. You can expect to see a fix for that checked in today or, latest, tomorrow.

Edit: that's fixed now.

The compile is much faster than it was before the changes so thanks for that.

I want to resound the gratefulness of curmudgeon on the wonderful work of this autosetup. Thanks.

Thank you, curmudgeon and Jose, for your feedback!

The compile is much faster than it was before the changes so thanks for that.

That's actually an unintended side effect of the port. The legacy build uses, via GNU Autotools, libtool, a compiler/linker proxy which performs all sorts of platform-specific magic and, incidentally, also compiles things more often than they strictly need to be for reasons beyond my ken.

On my machine (a mid-range i3-based NUC clocked at 3ghz), a clean rebuild with a single-core make running using --enable-all takes approximately 3m20s with the legacy build and 2m15s with the new one (dropping to 30s when using -O0 instead of -O2!). With a -j4 build (up to 4 parallel "make" process), those times drop to 2m9s and 1m20s, respectively (and 18s with the new build using -O0).

The primary reason for the port was to gain control over the configure process, as opposed to the makefiles (which we already 95% controlled, except that libtool sat between us and the compilers/linkers and performed no small amount of voodoo which we don't understand).

My recent experience doesn't really support my last sentence.

It certainly does for me :).

All code seems to execute faster while the laptop's plugged in, something I didn't realise.

Modern CPUs have multiple power-consumption modes to trade off power use vs speed and it's not uncommon for laptops to throttle back into power-saving mode when unplugged.

It may well be faster but not to the extent I originally believed Stephen.

A minor nit about HAVE_FCHMOD. It doesn't look like the configure machinery explicitly checks for its existence. As part of the os unix WASI stuff there is this which unilaterally redefines it:

#else /* !SQLITE_WASI */
# ifndef HAVE_FCHMOD
#  define HAVE_FCHMOD
# endif
#endif /* SQLITE_WASI */

Please at least change it to:

#  define HAVE_FCHMOD 1

This is to be consistent with all other HAVE_ including those produced by autotools. I have another header that is defining HAVE_FCHMOD as 1 so the compiler warns about a redefinition mismatch.

/space/pydebug/include/python3.13d/pyconfig.h:352:9: warning: "HAVE_FCHMOD" redefined
  352 | #define HAVE_FCHMOD 1
      |         ^~~~~~~~~~~
sqlite3/sqlite3.c:38651:11: note: this is the location of the previous definition
38651 | #  define HAVE_FCHMOD
      |           ^~~~~~~~~~~

It doesn't look like the configure machinery explicitly checks for its existence.

Looking at the legacy build, it (curiously) didn't check/define that either.

It's surprising that this conflict shows up now rather than when that block was added (Feb. 2023).

# define HAVE_FCHMOD 1

That's definitely consistent with how the configuration tools would define it if they checked for it and that change is now in the trunk.

We'll also teach the configure script to check for that. Why that historically has not been done is a mystery - presumably that function really is on every platform for which os_unix.c works. (Edit: nevermind - os_unix.c has always required/assumed existence of fchmod(), and the block which forcibly defines it was added to reflect that status quo.)

Thanks for the fix, and I confirm it works.

It's surprising that this conflict shows up now rather than when that block was added (Feb. 2023).

It did show up back then, but I figured it made no semantic difference, and looked like the doing of configure in two different projects.

I also get a screen full of warnings anyway because there are function prototypes declared for a bunch of SQLite functions, but the function bodies are not defined as they are optional. I am the only one who notices because I define SQLITE_API as static. To repoduce:

gcc -DSQLITE_API=static -Wall -c sqlite3.c

Start of the warnings:

sqlite3.c:5214:24: warning: ‘sqlite3_column_database_name’ declared ‘static’ but never defined [-Wunused-function]
 5214 | SQLITE_API const char *sqlite3_column_database_name(sqlite3_stmt*,int);
      |                        ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
sqlite3.c:5215:24: warning: ‘sqlite3_column_database_name16’ declared ‘static’ but never defined [-Wunused-function]
 5215 | SQLITE_API const void *sqlite3_column_database_name16(sqlite3_stmt*,int);
      |                        ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
sqlite3.c:5216:24: warning: ‘sqlite3_column_table_name’ declared ‘static’ but never defined [-Wunused-function]
 5216 | SQLITE_API const char *sqlite3_column_table_name(sqlite3_stmt*,int);
      |                        ^~~~~~~~~~~~~~~~~~~~~~~~~
sqlite3.c:5217:24: warning: ‘sqlite3_column_table_name16’ declared ‘static’ but never defined [-Wunused-function]
 5217 | SQLITE_API const void *sqlite3_column_table_name16(sqlite3_stmt*,int);
      |                        ^~~~~~~~~~~~~~~~~~~~~~~~~~~
sqlite3.c:5218:24: warning: ‘sqlite3_column_origin_name’ declared ‘static’ but never defined [-Wunused-function]
 5218 | SQLITE_API const char *sqlite3_column_origin_name(sqlite3_stmt*,int);
      |                        ^~~~~~~~~~~~~~~~~~~~~~~~~~
sqlite3.c:5219:24: warning: ‘sqlite3_column_origin_name16’ declared ‘static’ but never defined [-Wunused-function]
 5219 | SQLITE_API const void *sqlite3_column_origin_name16(sqlite3_stmt*,int);
      |                        ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
sqlite3.c:6898:16: warning: ‘sqlite3_win32_set_directory’ declared ‘static’ but never defined [-Wunused-function]
 6898 | SQLITE_API int sqlite3_win32_set_directory(
      |                ^~~~~~~~~~~~~~~~~~~~~~~~~~~

I notice from this changeset and looking at auto.def that icu-config is being used to find ICU flags.

icu-config is no longer maintained:

Note: icu-config is deprecated, and no longer recommended for production use. Please use pkg-config files or other options.

You can use pkg-config --cflags icu-uc icu-i18n icu-io to get the cflags and pkg-config --libs icu-uc icu-i18n icu-io to get the -L and -l flags. Even the ICU extension readme says to use pkg-config!

In my own equivalent code I use pkg-config --exists first, and only fallback to icu-config if that fails.

As far as I can tell the new autosetup stuff doesn't use pkg-config to look for anything. My system has configs for readline and tcl itself, and arguably they are the preferred way of getting the necessary flags.

Interestingly, pkg-config is itself being replaced in major Linux distros with pkgconf, and while upstream has compatibility as an explicit goal, they forcefully decry bug compatibility with pkg-config.

I don't think that directly affects SQLite builds, but you never know...

I notice from this changeset and looking at auto.def that icu-config is being used to find ICU flags.

icu-config can be used to find the ICU flags, but only if it's explicitly invoked to do so. ICU requires explicit opt-in, and use of icu-config (as opposed to manually passing linker flags) requires specifically opting in to that configuration method. That was added because the OpenBSD test systems have that tool installed by default. The BSDs in general are averse to outward-facing changes, so it's a good bet that icu-config will still be there in 10+ years.

i will look into providing a pkg-config-based check as an option.

As far as I can tell the new autosetup stuff doesn't use pkg-config to look for anything.

(A) because the legacy build doesn't use pkg-config for anything and (B) because my (limited) experience with it has been highly inconsistent in terms of getting usable results. As a timely example of (B), a cursory check shows that pkg-config and icu-config give different results on the same system for icu:

$ icu-config --ldflags-libsonly
-licui18n -licuuc -licudata 
$ pkg-config --libs-only-l icu-uc
-licuuc -licudata

Which one is more correct, one can only guess. Edit: no longer true! The list pkg-config returns is not sufficient for linking the sqlite3 binary on either Linux or OpenBSD. The list icu-config returns is. Edit: it turns out that icu-io, by itself, provides all the necessary flags, so we'll go with that.

Autosetup actually has builtin support for pkg-config, but my early attempts at using it (a month or two ago) were disappointing due to no fault of autosetup's: the pkg-config binary was exiting with an inexplicable message when checking for readline on an ARM-based OpenBSD system.

Interestingly, pkg-config is itself being replaced in major Linux distros with pkgconf, ... I don't think that directly affects SQLite builds, but you never know...

Not yet, it doesn't, but you never do know!

it turns out that icu-io, by itself, provides all the necessary flags, so we'll go with that.

I can confirm that if you pass only icu-uc to pkg-config then you don't get enough libraries, but passing icu-uc icu-i18n does, as does icu-io by itself.

pkgconf is yet another implementation of the pkg-config "standard" with a different sweet spot. Fortunately invoking pkg-config is sufficient to pick up whatever the implementation is. It even installs under homebrew on MacOS.

The lsm1 extension isn't setup when running this. Reproducer after getting the latest fossil:

$ ./configure --quiet --all --disable-tcl
$ cd ext/lsm1/
$ make
# lines of compiling each file, and 3 warnings
shared -fPIC -o lsm.so lsm_ckpt.o lsm_file.o lsm_log.o lsm_main.o lsm_mem.o lsm_mutex.o lsm_shared.o lsm_sorted.o lsm_str.o lsm_tree.o lsm_unix.o lsm_win32.o lsm_varint.o lsm_vtab.o
make: shared: No such file or directory
make: [Makefile:49: lsm.so] Error 127 (ignored)

Yes the error line is executing shared. The Makefile is telling me I should have my own makefile? In any event I would expect that the new world of autosetup would appropriately configure that directory like it does with all the others. It would be nice if ext/misc was also configured with a Makefile, so folks can get the SQLite release, run configure and all of the source tree is then appropriately configured for compilation and building.

The lsm1 extension isn't setup when running this

It's not working in the legacy build, either. The makefile is an incomplete stub, intended to...

The Makefile is telling me I should have my own makefile?

That indeed seems to be the case. (This dir is new to me, as is this quirk.)

In any event I would expect that the new world of autosetup would appropriately configure that directory like it does with all the others. ... It would be nice if ext/misc was also configured with a Makefile, so folks can get the SQLite release, run configure and all of the source tree is then appropriately configured for compilation and building.

The ext/... dirs are not configured/touched by the legacy build process and the new build process takes that same hands-off approach. We're not terribly keen on the burden of maintaining makefiles which aren't used by the project or its primarily deliverables. Since we wouldn't be using them, they'd be particularly subject to bitrot and breakage as the top-level build evolves.

That said: i'll bring this up with Richard and check how he'd like to proceed.

I am still keen on getting a prototype SQLite 3.48 release - ie what the equivalent of the download page would be. I've used the second Source Code file (sqlite-autoconf-...) for a very long time.

If that will become an export of the fossil code, then it would include ext/lsm1 and ext/misc.

... burden of maintaining makefiles ...

I thought the whole point of autosetup was that it made that burden very light :)

... bitrot and breakage ...

Some of the misc extensions are documented here. LSM was originally part of the SQLite 4 explorations and is nicely externally documented here. If the extensions are not supported then they really should be deleted, and if they are supported then they should really be compiled by the project to detect bitrot and breakage in them!

If the extensions are not supported then they really should be deleted,

Now and then that happens, but it always requires a decision from Richard.

If the extensions are not supported then they really should be deleted, and if they are supported then they should really be compiled by the project to detect bitrot and breakage in them!

My bitrot/breakage comment refers to the hypothetical build files we would be maintaining (but not using), not the corresponding C files. They would impose new maintenance burdens regardless of which build platform we use, and that burden-to-value ratio is infinitely on the left side of that equation because we don't need those build files - they'd just be dead weight for the project at that point, 100% burden with no project-level benefits for us.

Many of the extensions are included in the CLI shell without requiring external build files, so they lack that particular maintenance burden and aren't readily subject to "breakage through neglect" as such hypothetical build files would be.