I was asked in one of the posts to check fixes in current trunk and after trying new autosetup I thought I would share some remarks from a packager point of view:

1. Could you please add x-libraries/x-includes to autosetup/system.tcl for compatibility with autoconf? Used by AC_PATH_X macro irrelevant to sqlite but some distributions have configure macros that set them for all projects.
2. Some directory problems:
   • -rpath location LDFLAGS.rpath ?= -Wl,-rpath -Wl,$(prefix)/lib refers to $(prefix)/lib instead of $(libdir)
   • install-dir.include refers to $(prefix)/include instead of $(includedir)
   • it would be go to have $(mandir) overridable (defaults to $(prefix)/share/man)
3. Looks like SONAME is not set. I see SH_SOPREFIX determination in autosetup/cc-shared.tcl but it doesn't seem to be used anywhere. Although judging by comments in main.mk I fear SONAME is not intended to be maintained anymore?

Good early morning and thank you for taking the time to provide feedback!

Could you please add x-libraries/x-includes to autosetup/system.tcl for compatibility with autoconf? Used by AC_PATH_X macro irrelevant to sqlite but some distributions have configure macros that set them for all projects.

i have zero idea what the X-... things are, and even less idea what AC_PATH_X is, nor is it clear why we should have autoconf compatibility for a feature this tree does not use. All of the files in ./autosetup/..., except for proj.tcl, are part of the upstream autosetup project, so any change requests for those are better sent to https://github.com/msteveb/autosetup. We will be updating from that tree as needed.

-rpath location LDFLAGS.rpath ?= -Wl,-rpath -Wl,$(prefix)/lib refers to $(prefix)/lib instead of $(libdir)

First, note the ?= part, which means you're seeing that in main.mk and it does not apply to the canonical build. As a rule, the X?=Y assignments in main.mk are there as a convenience for the sake of hand-written makefiles which import main.mk, but Makefile (via Makefile.in) defines all of those itself (via the configure script) before including main.mk.

Secondly, $prefix/lib == $libdir, $prefix/include == $includedir, and $prefix/share/man == $mandir, as we can see by looking at a debug dump from the configure script (generated using the --dump-defines flag):

#define prefix "/usr/local"
#define libdir "/usr/local/lib"
#define includedir "/usr/local/include"
#define mandir "/usr/local/share/man"

We very intentionally do not export those dozen-ish $XYZdir vars to the makefiles, nor has any compelling argument been made as to why we should, seeing as they all derive from $prefix. We use $prefix and $DESTDIR, and that's about it.

We'd need a more compelling justification beyond "it would be good" to justify adding that currently-unnecessary noise to the makefiles.

Do you have a build process which overrides, for this specific project, $mandir/$libdir/$includedir to be rooted someplace other than $prefix?

Looks like SONAME is not set. I see SH_SOPREFIX determination in autosetup/cc-shared.tcl but it doesn't seem to be used anywhere. Although judging by comments in main.mk I fear SONAME is not intended to be maintained anymore?

We don't explicitly use SONAME or SH_SOPREFIX, nor are we currently aware of any reason we should be using them. If you feel otherwise, please elaborate on why they're needed and what it should be set to. As it is, the shared libraries build, link, and run fine without them.

For the x-libraries/x-includes could you please sync with autosetup upstream at any convenient time before making 3.48.0 release so it includes https://github.com/msteveb/autosetup/pull/71?

As for directories and SONAME I don't really want to go into disputes what's better or not -- just wanted to highlight spotted differences. Feel free to make your choices, ultimately we'll make it work for us if needed. To give some examples though:

• my libdir is /usr/lib64, so is at Fedora, Debian would use /usr/lib/aarch64-linux-gnu and it would be correctly set by configure for installation. Yet -rpath would keep on using $(prefix)/lib since it does not reuse $(libdir)
• we will surely maintain SONAME at downstream and I suspect Fedora will continue to do so too (see their packaging rules). It is unfortunate that so many binaries exist now that require libsqlite3.so.0 and starting with 3.48.0 it's gonna be a mix of libsqlite3.so and libsqlite3.so.0

As for directories and SONAME I don't really want to go into disputes what's better or not -- just wanted to highlight spotted differences.

We don't use it because we're not aware of any utility in it, but if you can convince us otherwise, and show us where it should be applied, we will include it.

FWIW, in 25+ years of building my own shared libs, SONAME has never once been relevant, and that's the primary reason for my reluctance to add it here without an explanation of what genuine benefit it provides.

my libdir is /usr/lib64, so is at Fedora, Debian would use /usr/lib/aarch64-linux-gnu and it would be correctly set by configure for installation. Yet -rpath would keep on using $(prefix)/lib since it does not reuse $(libdir)

My reluctance to using $libdir, etc., in the makefiles is based on:

• We don't have any use for any of those numerous vars. They're just noise.
• Because we don't use them, it's not clear to us where they should be used if there is a genuine use for them which we're unaware of.

autosetup supports an undocumented feature of enabling overriding of all of the autotools-specific XYZdir vars as flags to the configure script (--libdir=..., etc.) but we do not expose those to the generated makefile because they have no apparent practical use for our build.

We would be happy to be taught otherwise, however. If you'd like to point out, for example, which instances of $(prefix)/... in main.mk "should" be one of the other $XYZdir vars, and why it should be so, i'll be happy to accommodate that.

It is unfortunate that so many binaries exist now that require libsqlite3.so.0

The .0 suffix is a libtool'ism which simply doesn't apply anymore. None of us know where the 0.8.6 version number originally came from.

We experimented with a workaround in "make install" which would automatically re-link libsqlite3.so.0 (only if found) to the new naming scheme. We have it in the code history but it's not in the current trunk. If you can convince us that it would be useful, we can bring that back.

My reluctance to using $libdir, etc., in the makefiles is based on:

The current trunk introduces the ability to override those via the configure script because it's proven necessary for package maintainers who put libs someplace other than $prefix/lib:

$ ./configure --libdir=/foo --prefix=/usr
$ make install

will put everything under its $prefix-derived default dirs except for the libraries, which will go in /foo. Similarly, the -rpath flag, which has to be calculated at configure-time, also now honors any --libdir=X override. Conversely, passing a different libdir value to make will not change the -rpath flag because the exact formulation of that flag is compiler-dependent and we have to do that calculation at configure-time.

FWIW, in 25+ years of building my own shared libs, SONAME has never once been relevant, and that's the primary reason for my reluctance to add it here without an explanation of what genuine benefit it provides.

SONAME is used by the dynamic library loader to locate the correct file to provide the dependent symbols. The usual process is along the lines of:

1. A tool is compiled with a library dependency via a command along the lines of $CC foo.c -o foo -lssl
2. The compiler finds libssl.so on its search path, and adds an entry like "NEEDED libssl.so.10" to the Dynamic Section of the resulting ELF binary, where "libssl.so.10" here is derived from the SONAME found in libssl.so
3. When the foo binary is launched, the loader looks for "libssl.so.10" to satisfy the library dependency

A consequence here is that when the SONAME changes, previously compiled binaries need to be rebuilt to use the new library. The usual convention for dynamic libraries on linux is that the SONAME includes only the major version, and that binary compatibility is maintained within a major version. Ie. Changing the SONAME is reserved for ABI breaking changes. This convention leads to an environment where as long as a library can be located on the system we get the behaviour we expect from the dependent library, and we don't accidentally get different semantics from eg. a newer or older version.

I'm not sure where this convention came from, possibly libtool was an influence but it is very common on linux. It also enables multiple versions of a library to co-exist to satisfy different dependees when the ABI does change, however most distros seem to try to stick to one version of a library.

ie. the main impact of changing the SONAME here would be that every binary that has been linked so far against libsqlite.so.0 will not accept the new libsqlite.so as satisfying its dependency.

Adding libsqlite.so.0 as a symlink would probably be sufficient -- as long as the filename matches I don't think the loader verifies that the SONAME is the one that was requested. However it would be surprising to have the unversioned libsqlite.so as the canonical version, usually that is a symlink and on many distros is only included with the -dev/-devel package.

SONAME is used by...

Thank you for that detailed explanation!

Is it correct to say that there is no SONAME embedded in the lib then the linker will simply use whatever libsqlite3.so there is?

If so, i opine that we don't need to set SONAME, as there are no binary-incompatible versions of libsqlite3.so (and none are ever expected - binary backwards compatibility is a hard requirement for libsqlite3). However...

The current lib installation rules leave behind what, to my eyes, looks like a conventional Unix-esque DLL installation:

libsqlite3.so       -> libsqlite3.so.3
libsqlite3.so.3     -> libsqlite3.so.3.48.0
libsqlite3.so.3.48.0

My current (mis?)understanding is that we can appease the legacy use of SONAME=libsqlite3.so.0 by simply providing a compatibility symlink of:

libsqlite3.so.0 -> libsqlite3.so.3

Is that correct? Is it misled?