SQLite Forum

Compiling without the amalgamation process

Compiling without the amalgamation process

(1) By anonymous on 2020-03-13 17:38:14 [link] [source]

Hi everyone,

I'm currently working on a university project whose aim is to modify the behaviour of the VDBE. My end goal is to replace parts of the sqlite3_vdbeexec() function with a custom implementation.

The thing is, so far, I have not found a way to compile each implementation file separately. That leads to very long compile times (since every make call kicks off the amalgamation process - which is already long - and consequently rebuilds everything, even though I only change the body of a function in the vdbe.c file).

I was wondering if there was a way to disable amalgamation totally and work with separate compilation? If so, how? Is it documented somewhere? I have been trying different workarounds I could come up with, but so far nothing really worked, and I think I'm just not looking in the right place.

Any suggestions are welcome.

Thanks a lot in advance,


(2) By Richard Hipp (drh) on 2020-03-13 17:45:54 in reply to 1 [link] [source]

Here's what you do:

  1. Make a copy of the "Makefile.linux-gcc" makefile. Call it whatever you like.

  2. Edit your copy of Makefile.linux-gcc to set up the compiler and system options as you like. This is not hard as there are not many options.

  3. make

(3) By anonymous on 2020-03-13 18:55:07 in reply to 2 [link] [source]

Thank you.

I've been trying to use that, but I'm not sure this is doing exactly what I'm looking for: when I execute make, it still generates sqlite3.c and sqlite3.h, and still tries to find sqlite3VdbeExec function in this file, where it is not.

Do you think I did something wrong?

(4) By Richard Hipp (drh) on 2020-03-13 19:09:59 in reply to 3 [source]

I just ran the experiment myself, and it did separate compilation for me.

Are you sure you are using Makefile.linux-gcc and not Makefile.in?

(5) By anonymous on 2020-03-13 19:27:40 in reply to 4 [link] [source]

Umh... you're right actually. I re-did it and it worked. I'm not sure exactly what I did wrong the first time. For reference, I ended up using

make -f Makefile.linux-gcc

Instead of just make.

I've got a follow-up question if that is alright. I'd like to write my implementation in C++ instead of C, is that possible? I naively tried to replace gcc by g++ in Makefile.linux-gcc but G++ is not as happy with the code as GCC is. The first error I have (of many) is the following, which I'd say comes from the more restrictive type system of C++:

shell.c:643:15: error: assigning to 'char *' from incompatible type 'void *'
      zLine = realloc(zLine, nLine);

The thing is as far I as can see, all SQLite code files do have the necessary

#ifdef __cplusplus
extern "C" {

So I don't know what's the cause of these issues.

Thanks again.

(6) By Keith Medcalf (kmedcalf) on 2020-03-13 22:42:52 in reply to 5 [link] [source]

g++ forces compilation with the C++ preprocessor irrespective of the file extension, and automagically adds the libstdc++.a (and maybe others) to the linkage.

In "plain english" that means that "g++ <other stuff>" is pretty much merely a magical incantation/syntactical sugar for "gcc -x c++ -lstdc++ <other stuff>"

To have g++ choose the compiler preprocessor based on the file extension you have to turn off the -x option and allow the file extension to determine the language preprocessor to use.

You do this by using "g++ -x none" which will revert to using the standard rules for determining which compiler to use but still include the libstdc++.a in the link step if this g++ invocation invokes the linker.

"extern C" merely controls external symbol name mangling.  Declarations which are "extern C" will use industry-standard C-style name mangling no matter what language preprocessor is being used -- so for example if you are using the C++ language preprocessor then C-style name mangling will be performed rather than C++ name mangling.  "extern C" does not affect the choice of language preprocessor used by the compiler set.

(7) By anonymous on 2020-03-14 15:57:46 in reply to 6 [link] [source]

Thanks a lot for that detailed answer. I understand better now why it wouldn't work.

I have replaced all definitions (TCC and BCC) in the Makefile from g++ (which used to be gcc before I changed it) to g++ -x none. Unfortunately that has not changed anything, and I'm still getting the same errors.

After running make clean, and re-running make -f Makefile.linux-gcc, the errors I get are the following (I guess they're the errors from the first targets):

../sqlite/tool/lemon.c:2286:40: error: assigning to 'Boolean' from incompatible type 'int'
          psp->prevrule->neverReduce = 1;
../sqlite/tool/lemon.c:2290:35: error: assigning to 'Boolean' from incompatible type 'int'
          psp->prevrule->noCode = 0;
480 warnings generated.
../sqlite/tool/lemon.c:2398:24: error: assigning to 'Boolean' from incompatible type 'int'
          rp->noCode = 1;
../sqlite/tool/lemon.c:3732:18: error: assigning to 'Boolean' from incompatible type 'int'
    rp->noCode = 1;
../sqlite/tool/lemon.c:3734:18: error: assigning to 'Boolean' from incompatible type 'int'
    rp->noCode = 0;
../sqlite/tool/lemon.c:3750:20: error: assigning to 'Boolean' from incompatible type 'int'
      rp->noCode = 0;
../sqlite/tool/lemon.c:3900:18: error: assigning to 'Boolean' from incompatible type 'int'
    rp->noCode = 0;
../sqlite/tool/lemon.c:4428:32: error: assigning to 'Boolean' from incompatible type 'int'
        ap->x.rp->doesReduce = 1;
../sqlite/tool/lemon.c:4743:28: error: assigning to 'Boolean' from incompatible type 'int'
        rp2->codeEmitted = 1;
../sqlite/tool/lemon.c:4748:23: error: assigning to 'Boolean' from incompatible type 'int'
    rp->codeEmitted = 1;

My diagnostic would be that lemon is simply not meant to be compiled with a C++ compiler, hence I should first compile it with gcc and only use a C++ compiler in a second time for the actual SQLite source.

So that's what I did: I changed g++ back to gcc and re-ran make. I get these errors again (but I expect it, as I'm not linking the C++ standard library / using GCC):  

212 warnings generated.
Undefined symbols for architecture x86_64:
  "std::__1::locale::use_facet(std::__1::locale::id&) const", referenced from:
      std::__1::ctype<char> const& std::__1::use_facet<std::__1::ctype<char> >(std::__1::locale const&) in libsqlite3.a(thomas.o)
  "std::__1::ios_base::getloc() const", referenced from:
      std::__1::basic_ios<char, std::__1::char_traits<char> >::widen(char) const in libsqlite3.a(thomas.o)
  "std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::__init(unsigned long, char)", referenced from:
      std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::basic_string(unsigned long, char) in libsqlite3.a(thomas.o)
  "std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::~basic_string()", referenced from:
      std::__1::ostreambuf_iterator<char, std::__1::char_traits<char> > std::__1::__pad_and_output<char, std::__1::char_traits<char> >(std::__1::ostreambuf_iterator<char, std::__1::char_traits<char> >, char const*, char const*, char const*, std::__1::ios_base&, char) in libsqlite3.a(thomas.o)
  "std::__1::basic_ostream<char, std::__1::char_traits<char> >::put(char)", referenced from:
      std::__1::basic_ostream<char, std::__1::char_traits<char> >& std::__1::endl<char, std::__1::char_traits<char> >(std::__1::basic_ostream<char, std::__1::char_traits<char> >&) in libsqlite3.a(thomas.o)

After changing back to g++:

BCC = g++ -x none -g -O0
TCC = g++ -x none -O0

I get several errors coming from compiling shell.c:

shell.c:817:12: error: assigning to 'char *' from incompatible type 'void *'
    p->z = realloc(p->z, p->nAlloc);
shell.c:868:16: warning: conversion from string literal to 'char *' is deprecated [-Wc++11-compat-deprecated-writable-strings]
  char *zDiv = "(";
shell.c:888:12: warning: conversion from string literal to 'char *' is deprecated [-Wc++11-compat-deprecated-writable-strings]
    zDiv = ",";
shell.c:1883:5: error: no matching function for call to 'SHA3Update'
    SHA3Update(&cx, sqlite3_value_blob(argv[0]), nByte);
shell.c:1793:13: note: candidate function not viable: cannot convert argument of incomplete type 'const void *' to 'const unsigned char *' for 2nd argument
static void SHA3Update(
shell.c:2037:34: error: cannot initialize a variable of type 'const unsigned char *' with an rvalue of type 'const void *'
            const unsigned char *z2 = sqlite3_column_blob(pStmt, i);
                                 ^    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

All the errors come from compiling the shell, though I don't know whether it is because it can't compile the shell or if it is because the shell is the first of several files that can't be compiled.

Once again any suggestions are welcome.

(8) By anonymous on 2020-03-14 16:07:01 in reply to 7 [link] [source]

Okay, so right now I got to change the main.mk a bit and finally got it to work.

So for future reference:

  • I have a first make pass where I don't change any file (to compile lemon, possibly among others)
  • Then, I edit Makefile.linux-gcc and replace all occurrences of gcc with gcc -x none
  • I change main.mk and add change the executable target by adding -lstdc++ at the end:
sqlite3$(EXE):	shell.c libsqlite3.a sqlite3.h
	$(TCCX) $(READLINE_FLAGS) -o sqlite3$(EXE) $(SHELL_OPT) \
		shell.c libsqlite3.a $(LIBREADLINE) $(TLIBS) $(THREADLIB) -lstdc++
  • I run make again using make -f Makefile.linux-gcc and get my sqlite3 executable as expected.

Thanks everyone for your support, you were of great help.

(9) By Keith Medcalf (kmedcalf) on 2020-03-14 16:43:17 in reply to 8 [link] [source]

I should think that you do not need to change "gcc" to "gcc -x none". As far as I know the default for the gcc driver is -x none. g++ is the one that changes the defaults.

Also, I believe your other issues are because the language level defaults are probably also changed to incompatible settings when using the g++ command to compile c code because using g++ as the driver implies that you want C++ compatibility, not standard C.

You really should only need to change the linker steps to include the additional C++ libraries (and of course to use your new .cpp source file).

(10) By anonymous on 2020-03-17 08:33:04 in reply to 9 [link] [source]

Thank you, I'll try without.