SQLite Forum

Announce: migration of the build system to autosetup
Login

Announce: migration of the build system to autosetup

(1) By Stephan Beal (stephan) on 2024-10-21 18:53:13 [source]

Good evening, folks,

As Richard just posted in the "announce" list for the 3.47.0 release:

During the next release cycle, we are planning to do an overhaul of the "configure; make" system for SQLite. This is intended to be an improvement. We will try really, really hard not to break anything. But because we don't know all the creative ways that people are currently using "configure; make" now, our changes will be impossible to test thoroughly. We would appreciate your help in trying out the new "configure; make" found in periodic pre-release snapshots that we will make available. Please do not wait until the next official release to discover that we have accidentally broken your particular build, and then complain. Better to find these problems early.

To elaborate on that...

That work is currently going on in the autosetup branch, and ← that link has a description of the motivation for this migration, as well as the perceived benefits and down-sides.

For those of you who simply run "./configure && make", very little, if anything, will change. The parts of the build which most folks use day-to-day are already migrated to the newer system, but there are the obligatory pending TODOs and FIXMEs, mostly summarized at the top of Makefile.in.

This change will require some adaptation on some folks' automated build processes. i.e. some build-level breakage is unavoidable. We are making every effort to avoid gratuitous breakage, though.

Please alert us to any new breakage here in the forum.

Sidebar: these changes only apply to platforms which use the "configure" process, i.e. Unix-like environments. Vanilla Windows builds are unaffected and we will continue to provide static makefiles for Windows.

(2) By Roger Binns (rogerbinns) on 2024-10-21 21:54:38 in reply to 1 [link] [source]

I need to know the various -DHAVE_ values because I am compiling SQLite with Python's build system not SQLite's. Currently I do it by running configure and parsing the generated Makefile and looking for the DEFS = line.

Please include a way of getting them. I'm happy to change my code.

(3) By Stephan Beal (stephan) on 2024-10-21 22:08:39 in reply to 2 [link] [source]

Currently I do it by running configure and parsing the generated Makefile and looking for the DEFS = line.

FWIW, we're currently doing something similar in the ext/wasm build and i hate that ;).

In the off chance that you don't already know this: more reliable than parsing the makefile is using GNU make's -n and -p flags, like so:

$ make -pn | grep -e '^SHELL_OPT'
SHELL_OPT = -DSQLITE_DQS=0 -DSQLITE_ENABLE_FTS4 -DSQLITE_ENABLE_RTREE -DSQLITE_ENABLE_EXPLAIN_COMMENTS -DSQLITE_ENABLE_UNKNOWN_SQL_FUNCTION -DSQLITE_ENABLE_STMTVTAB -DSQLITE_ENABLE_DBPAGE_VTAB -DSQLITE_ENABLE_DBSTAT_VTAB -DSQLITE_ENABLE_BYTECODE_VTAB -DSQLITE_ENABLE_OFFSET_SQL_FUNC -DSQLITE_STRICT_SUBTYPE=1
  • -p = print the make "database"
  • -n = don't build anything

BSD make has -n but not -p, but i only this moment learned that BSD make has something arguably more interesting than -p:

$ bmake -V SHELL_OPT
-DSQLITE_DQS=0 -DSQLITE_ENABLE_FTS4 -DSQLITE_ENABLE_RTREE -DSQLITE_ENABLE_EXPLAIN_COMMENTS -DSQLITE_ENABLE_UNKNOWN_SQL_FUNCTION -DSQLITE_ENABLE_STMTVTAB -DSQLITE_ENABLE_DBPAGE_VTAB -DSQLITE_ENABLE_DBSTAT_VTAB -DSQLITE_ENABLE_BYTECODE_VTAB -DSQLITE_ENABLE_OFFSET_SQL_FUNC -DSQLITE_STRICT_SUBTYPE=1

Please include a way of getting them.

Which var(s) are you looking for? We don't have any named "DEFS" (unless they're part of the Windows build, but those aren't affected by these changes).

(4) By Roger Binns (rogerbinns) on 2024-10-21 22:23:53 in reply to 3 [link] [source]

... is using GNU make's ...

I don't run make at any point, nor do I know or have any control over what make is present. My stuff is packaged by third parties on many different Linuxes, BSD, Android etc, although I only do this when SQLite is directly bundled into my library.

I do the extraction using Python code.

We don't have any named "DEFS"

You do, and I've been using them for over a decade! Here is what is in Makefile after running configure on Linux

DEFS = -DPACKAGE_NAME=\"sqlite\" -DPACKAGE_TARNAME=\"sqlite\" -DPACKAGE_VERSION=\"3.47.0\" -DPACKAGE_STRING=\"sqlite\ 3.47.0\" -DPACKAGE_BUGREPORT=\"http://www.sqlite.org\" -DPACKAGE_URL=\"\" -DPACKAGE=\"sqlite\" -DVERSION=\"3.47.0\" -DHAVE_STDIO_H=1 -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 -DHAVE_INTTYPES_H=1 -DHAVE_STDINT_H=1 -DHAVE_STRINGS_H=1 -DHAVE_SYS_STAT_H=1 -DHAVE_SYS_TYPES_H=1 -DHAVE_UNISTD_H=1 -DSTDC_HEADERS=1 -DHAVE_DLFCN_H=1 -DLT_OBJDIR=\".libs/\" -DHAVE_FDATASYNC=1 -DHAVE_USLEEP=1 -DHAVE_LOCALTIME_R=1 -DHAVE_GMTIME_R=1 -DHAVE_DECL_STRERROR_R=1 -DHAVE_STRERROR_R=1 -DHAVE_READLINE_READLINE_H=1 -DHAVE_READLINE=1 -DHAVE_POSIX_FALLOCATE=1 -DHAVE_ZLIB_H=1

I create a file in the same directory as sqlite3 named sqlite3config.h and then #include that in the build before #include "sqlite3.c". The above turns into this:


    /* This file was generated by parsing how configure altered the Makefile
    which isn't used when building python extensions.  It is specific to the
    machine and developer components on which it was run. */
    
#define HAVE_STDIO_H 1
#define HAVE_STDLIB_H 1
#define HAVE_STRING_H 1
#define HAVE_INTTYPES_H 1
#define HAVE_STDINT_H 1
#define HAVE_STRINGS_H 1
#define HAVE_SYS_STAT_H 1
#define HAVE_SYS_TYPES_H 1
#define HAVE_UNISTD_H 1
#define HAVE_DLFCN_H 1
#define HAVE_FDATASYNC 1
#define HAVE_USLEEP 1
#define HAVE_LOCALTIME_R 1
#define HAVE_GMTIME_R 1
#define HAVE_DECL_STRERROR_R 1
#define HAVE_STRERROR_R 1
#define HAVE_READLINE_READLINE_H 1
#define HAVE_READLINE 1
#define HAVE_POSIX_FALLOCATE 1
#define HAVE_ZLIB_H 1

(5) By Stephan Beal (stephan) on 2024-10-21 22:43:39 in reply to 4 [link] [source]

You do, and I've been using them for over a decade! Here is what is in Makefile after running configure on Linux

That's not what i'm seeing from the current trunk:

$ uname -a
Linux nuc 6.8.0-41-generic ...

$ ./configure --enable-all
...

$ grep -w DEFS Makefile
<no output>

$ grep -w DEFS * 2>/dev/null
config.log:DEFS='-DHAVE_CONFIG_H'
config.status:S["DEFS"]="-DHAVE_CONFIG_H"
configure:DEFS
configure:# confdefs.h avoids OS command line length limits that DEFS can exceed.
configure:DEFS=-DHAVE_CONFIG_H

(6) By Roger Binns (rogerbinns) on 2024-10-21 22:50:20 in reply to 5 [link] [source]

That's not what i'm seeing from the current trunk:

Correct. It is what you get from the release downloads like 3.47.0 which matters in this case.

During development when I work with fossil I keep a copy of the generated file around.

(7) By Stephan Beal (stephan) on 2024-10-21 22:58:53 in reply to 6 [link] [source]

Correct. It is what you get from the release downloads like 3.47.0 which matters in this case.

Those artifacts are generated by the autotools, so will de facto be lost in this port, but i have noted a TODO for some way to extract those or, similarly, specifically emit them to their own file(s) as part of the configure step, perhaps even as python/tcl/shell/json/whatever code.

(8) By Roger Binns (rogerbinns) on 2024-10-21 23:07:51 in reply to 7 [link] [source]

That works for me. JSON preferred. Other formats like KEY = VALUE are rife with problems such as determining what are numbers, what are strings, unintentionally converting no to a boolean etc.

(9) By anonymous on 2024-10-23 04:52:09 in reply to 4 [link] [source]

Steve Bennett here (author of autosetup)

make-config-header can generate a nice easy-to parse header file that can be used independently of the make-based build.

(10) By Stephan Beal (stephan) on 2024-10-23 05:23:09 in reply to 9 [link] [source]

make-config-header can generate a nice easy-to parse header file that can be used independently of the make-based build.

We're using that to generate sqlite_cfg.h and a debug-only dump of all of the build defines, so it would not be a far stretch for us to do the same to cover Roger's case. Ideally i'd rather emit that as something portable, so will probably fork a copy of make-config-header and dump that all out as JSON (and maybe other formats, like tcl, shell, and python, simply because it would be trivial to do at that point).