Hello SQLite Team,

We are using BerkeleyDB of db-18.1.32 in our product. We support our product on more than 16 platforms. We are trying to add support for the new platform Linux-PPC-64 Little Endian. As of now, we were supporting Linux-PPC-64 Big Endian.

So, we are getting some issues related to BerkeleyDB for this new platform Linux-PPC-64 Little Endian.

First of all, when we tried configuring the standalone build of db-18.1.32, we got the below error :

checking build system type... ../../dist/../lang/sql/sqlite/config.guess: unable to guess system type

This script, last modified 2007-07-22, has failed to recognize the operating system you are using. uname -m = ppc64le uname -r = 4.18.0-147.el8.ppc64le uname -s = Linux configure: error: cannot guess build type; you must specify one

In order to solve above error we added below code in the file - db-18.1.32/lang/sql/sqlite/config.guess Line931 ppc64le:Linux::) Line932 echo powerpc64-unknown-linux-${LIBC} Line933 exit ;;

After this change, configuration and build was successful. But, when we used this newly built binary in our project, we started getting the below error:

outsidein/Vendor/Berkeley_DB/db-18.1.32/lib/linux-ppc-64/release/libdb-18.1.so: error adding symbols: File in wrong format collect2: error: ld returned 1 exit status

Information for the Linux-PPC Machine where we are building : uname -ai Linux 4.18.0-147.el8.ppc64le #1 SMP Thu Sep 26 15:44:34 UTC 2019 ppc64le ppc64le ppc64le GNU/Linux

/usr/bin/gcc --version >gcc (GCC) 8.3.1 20190507 (Red Hat 8.3.1-4)

/usr/bin/g++ --version >g++ (GCC) 8.3.1 20190507 (Red Hat 8.3.1-4)

On contacting BerkeleyDB tea, they mentioned that there might be a problem with SQLite. Can you please guide us on whether the changes we made are correct and sufficient enough or do we need to make some other changes to incorporate the new platform Linux ppc64le(little-endian).

I think the SQLite source code assumes that PPC is always big-endian. See sqliteInt.h lines 904-906.

Perhaps if you add -DSQLITE_BYTEORDER=1234 to your Linux-PPC-64 Little Endian build, you can get it to work that way.

Talk about obscure platforms: We have enough trouble finding big-ending PPC machines for testing. Where did you find a little-ending PPC machine that runs Linux?

I've tried to force SQLITE_BYTEORDER to 1234 but it fails to compile http3server: https://bugzilla.mozilla.org/show_bug.cgi?id=1757271#c8

(Corrected code example.)

This is buggy code, it's not correct to infer from ppc that the platform is big-endian. Indeed, POWER can change endian at runtime, though Linux doesn't (really) support this.

It's a reasonable default under the circumstances. Also, see last comment below.

The simplest thing to do is to just remove the "defined(ppc)" expression.

I think that's not quite right either.

There's no performance penalty I can see as the manual "swizzles" from SQL_BYTEORDER=0 are compiled away above -O1.

Compiling away those swizzles would be incorrect if the endianness changed at runtime. So I think this argues for not letting the swizzler figure it out and requiring that SQLITE_BYTEORDER have a defined value.

Alternatively, there is a little runtime trickery that can be used: /* Index into a u32, ms-byte first through ls-byte last. */ union swizz { u32 asULong; u8 asBytes[4]; } byteSwizz = { 0x03020100 }; // /* Read a u32 from big-endian in-memory sequence. */ u32 read(u8 *pb) { int i = 0; union swizz rv = {0}; while( i<4 ){ rv.asBytes[byteSwizz.asBytes[i]] = pb[i]; ++i; } return rv.asULong; }

to clarify, the endianness of the processor is not going to change during the execution of any given program. an operating system on ppc64 boots into a specific endianness, and it remains that way. the little endian and big endian modes have their own target triples, and a binary compiled for one mode is only valid for that mode - the same as with mips, or arm.

it is also certainly incorrect to infer that __ppc__ is always big endian. even the old 32 bit chips could run in little endian modes, but this was only used in embedded scenarios to my knowledge, and support was temporarily dropped in the transition to 64 bit. for 64 bit chips, it was reintroduced in POWER8, a decade ago, and many operating systems targeting it today only support the little endian mode (ubuntu, fedora, chimera linux, and alpine linux come to mind; gentoo and freebsd support both endiannesses).

removing the check for __ppc__ ensures the expected behavior regardless of the endianness (i have tested with both), and i believe it is a suitable way to resolve this.

it is also certainly incorrect to infer that ppc is always big endian.

I see no such inference, nor have I made such.

removing the check for ppc ensures the expected behavior regardless of the endianness (i have tested with both), and i believe it is a suitable way to resolve this.

Does it not work for you to use "#define SQLITE_BYTEORDER 1234"?

When you say "i have tested with both", do mean to claim that you have verified that the suggested change continues to yield the same results across the range of platforms using PPC and compilers targeting it?

(Added via edit:) An even stronger (and skepticism-worthy) claim would be that those same results occur across that combination for builds from reasonably likely build scripts and make files. Will you go that far too?

i have not done exhaustive tests across all possible ppc platforms; i did run make test on 64-bit little endian powerpc on linux with gcc/clang, and on 32-bit big endian powerpc on Mac OS X with gcc7 only insofar as it runs (it dies towards the end on a tcl error, but everything that runs passes, same as without the change). on little endian, i also tested with Alpine Linux's distro build scripts for sqlite, installing it as my system sqlite with no obvious ill effects. i'm preparing to run the test suite on 64-bit big endian.

for 64-bit little endian with clang, to run make test, i did also have to also patch out some inline assembly related to hwtime that fails to assemble for ppc64(le). Chimera Linux also patches this out. the test results are unaffected by this, since hwtime is already stubbed out in the gcc build due to it not defining __ppc__. the correct fix for this would probably be to add a check for __powerpc64__ there.

although it would take some time, i would be happy to build the project and run the full test suites, collecting logs from start to finish to provide to you, both with and without this change, on each of:

• alpine linux, 64-bit, little endian, POWER9 (gcc13, clang16)
• gentoo, 64-bit, big endian, POWER9 (gcc, clang)
• gentoo, 64-bit, little endian POWER9 (gcc, clang)
• gentoo, 32-bit, big endian, POWER9 (gcc, clang)
• Mac OS X 10.5, 32-bit, big endian, PowerPC G4 (buggy apple gcc4, gcc7.3 from tigerbrew)

on the 64-bit systems, i can also apply this change to Firefox on my system in place of the patch that Alpine Linux and Chimera Linux are currently carrying, and confirm that it works as expected. that patch changes the same portion of code in a slightly different way, and also #ifdef 0's the hwtime code. currently, if compiled on ppc64le with clang, without aforementioned patches, Firefox does not work properly because of this very issue.

if necessary, i may also be able to run tests on 64-bit AIX (gcc) and FreeBSD (clang), for big and both endians respectively, including testing with Firefox on FreeBSD, but both of these will take some time. i cannot run AIX directly but i think i can get access to a box; i can run FreeBSD but i'll need to set up VMs for it. i will note that FreeBSD uses -DSQLITE_BYTEORDER=1234 for their system sqlite to work around this. they also build Firefox with their system sqlite rather than Firefox's vendored version, which most Linux distros do not do. this does result in a working Firefox for them.

if you want to be particularly picky, i can bring out my G5 and test on that, big-endian only, with 32/64 bit Gentoo and 32-bit NetBSD, but that machine is a pain to deal with on account of faulty video hardware. testing on Power MachTen and BeOS is also within my grasp, though i doubt you're interested in those results (frankly, i would be very surprised to see this compile at all with BeOS's mwcc)

i will note that FreeBSD uses -DSQLITE_BYTEORDER=1234

Does that work for you?

i may also be able to run tests ...

That looks like a lot of work.

For what it's worth, I would see your suggestion of eliminating the dependence upon defined(ppc) as having aesthetic and technical merit if there were standardized macros for detecting "correct" endianness.

However, I see significant risk flowing from such a solution to your problem, risk which others not (yet) participating in the discussion would bear. Unless all clients building for PPC are using a compiler which defines BYTE_ORDER and ORDER_LITTLE_ENDIAN preprocessor symbols, (which I understand is a gcc-specific innovation), they are going to end up relying on the runtime endianness detection. And, depending on what trickery is being done with the chameleon-like PPC endianness, that may not be right either.

Making the call regarding hazard-fraught code changes is above my pay grade, so probably Richard would have to decide it. (He may already have.) He is a very pragmatic person, who hesitates to risk breakage for existing users based on code aesthetic considerations. Where there is already an override available for those who need it, (defining SQLITE_BYTEORDER), I am hard-put to argue for the change, especially when verifying it is a non-trivial task.

As I study that endianness-setting preprocessor code and the ways of doing such established elsewhere, I see no good way to argue that a more reliable solution exists. I grant that the present scheme necessitates use of SQLITE_BYTEORDER for the little-endian PPC target, but I do not see a solution that can be argued, on a technical basis, to avoid that drawback without risking breakage for folks who do not see it coming.

modern IBM POWER machines supporting little endian (POWER8 and up) can be gotten from IBM (expensively $$$$$), or on the used market (less expensively, $$$). there are also other vendors that make machines with these processors ($$$$). my own such machine is a Blackbird workstation from Raptor Computing Systems. these machines support booting into either endianness, and you can also set up virtual machines in either endianness and with a different featureset - so, one machine could do testing for 64-bit big-endian, 64-bit little-endian, 32-bit big-endian, and hypothetically, 32-bit little-endian (good luck finding a distro to run on that - you'll have to build it yourself).

would a donation of such a POWER8 system be useful? if so, i might consider that in the future.

Thank you for the offer, but we don't have any use for a POWER8 system at this time.

I keep an old Mac iBook at hand for testing SQLite on big-endian processors. Performance on that 20-year-old iBook is sluggish at best, but it is sufficient for verifying that SQLite works on big-endian. The iBook is compact, uses little power, is easy to operate, and does not get in the way. I'm guessing that a POWER8 machine would not meet any of those criteria.

Trunk has now been updated so that it no longer assumes big-endian if the __ppc__ macro is defined. This changes works on every platform I have at hand, including the old iBook. Please verify that the latest trunk check-in works on your little-endian PPCs and report back.