SQLite Forum

Compiling without the amalgamation process

Compiling without the amalgamation process

(1) By anonymous on 2020-03-13 17:38:14 [link]

Hi everyone,

I'm currently working on a university project whose aim is to modify the behaviour of the VDBE. My end goal is to replace parts of the `sqlite3_vdbeexec()` function with a custom implementation.

The thing is, so far, I have not found a way to compile each implementation file separately. That leads to very long compile times (since every `make` call kicks off the amalgamation process - which is already long - and consequently rebuilds everything, even though I only change the body of a function in the `vdbe.c` file). 

I was wondering if there was a way to disable amalgamation totally and work with separate compilation? If so, how? Is it documented somewhere? I have been trying different workarounds I could come up with, but so far nothing really worked, and I think I'm just not looking in the right place.

Any suggestions are welcome.

Thanks a lot in advance,


(2) By Richard Hipp (drh) on 2020-03-13 17:45:54 in reply to 1 [link]

Here's what you do:

  1.  Make a copy of the "Makefile.linux-gcc" makefile.  Call it whatever
      you like.

  2.  Edit your copy of Makefile.linux-gcc to set up the compiler and
      system options as you like.  This is not hard as there are not many

  3.  `make`

(3) By anonymous on 2020-03-13 18:55:07 in reply to 2

Thank you. 

I've been trying to use that, but I'm not sure this is doing exactly what I'm looking for: when I execute `make`, it still generates `sqlite3.c` and `sqlite3.h`, and still tries to find `sqlite3VdbeExec` function in this file, where it is not.

Do you think I did something wrong?

(4) By Richard Hipp (drh) on 2020-03-13 19:09:59 in reply to 3 [link]

I just ran the experiment myself, and it did separate compilation for me.

Are you sure you are using `Makefile.linux-gcc` and not `Makefile.in`?

(5) By anonymous on 2020-03-13 19:27:40 in reply to 4 [link]

Umh... you're right actually. I re-did it and it worked. I'm not sure exactly what I did wrong the first time. For reference, I ended up using 

make -f Makefile.linux-gcc

Instead of just `make`. 

I've got a follow-up question if that is alright. I'd like to write my implementation in C++ instead of C, is that possible? I naively tried to replace `gcc` by `g++` in `Makefile.linux-gcc` but G++ is not as happy with the code as GCC is. The first error I have (of many) is the following, which I'd say comes from the more restrictive type system of C++:

shell.c:643:15: error: assigning to 'char *' from incompatible type 'void *'
      zLine = realloc(zLine, nLine);

The thing is as far I as can see, all SQLite code files do have the necessary 

#ifdef __cplusplus
extern "C" {

So I don't know what's the cause of these issues.

Thanks again.

(6) By Keith Medcalf (kmedcalf) on 2020-03-13 22:42:52 in reply to 5 [link]

g++ forces compilation with the C++ preprocessor irrespective of the file extension, and automagically adds the libstdc++.a (and maybe others) to the linkage.

In "plain english" that means that "g++ <other stuff>" is pretty much merely a magical incantation/syntactical sugar for "gcc -x c++ -lstdc++ <other stuff>"

To have g++ choose the compiler preprocessor based on the file extension you have to turn off the -x option and allow the file extension to determine the language preprocessor to use.

You do this by using "g++ -x none" which will revert to using the standard rules for determining which compiler to use but still include the libstdc++.a in the link step if this g++ invocation invokes the linker.

"extern C" merely controls external symbol name mangling.  Declarations which are "extern C" will use industry-standard C-style name mangling no matter what language preprocessor is being used -- so for example if you are using the C++ language preprocessor then C-style name mangling will be performed rather than C++ name mangling.  "extern C" does not affect the choice of language preprocessor used by the compiler set.

(7) By anonymous on 2020-03-14 15:57:46 in reply to 6 [link]

Thanks a lot for that detailed answer. I understand better now why it wouldn't work.

I have replaced all definitions (`TCC` and `BCC`) in the Makefile from `g++` (which used to be `gcc` before I changed it) to `g++ -x none`. Unfortunately that has not changed anything, and I'm still getting the same errors.

After running `make clean`, and re-running `make -f Makefile.linux-gcc`, the errors I get are the following (I guess they're the errors from the first targets):

../sqlite/tool/lemon.c:2286:40: error: assigning to 'Boolean' from incompatible type 'int'
          psp->prevrule->neverReduce = 1;
../sqlite/tool/lemon.c:2290:35: error: assigning to 'Boolean' from incompatible type 'int'
          psp->prevrule->noCode = 0;
480 warnings generated.
../sqlite/tool/lemon.c:2398:24: error: assigning to 'Boolean' from incompatible type 'int'
          rp->noCode = 1;
../sqlite/tool/lemon.c:3732:18: error: assigning to 'Boolean' from incompatible type 'int'
    rp->noCode = 1;
../sqlite/tool/lemon.c:3734:18: error: assigning to 'Boolean' from incompatible type 'int'
    rp->noCode = 0;
../sqlite/tool/lemon.c:3750:20: error: assigning to 'Boolean' from incompatible type 'int'
      rp->noCode = 0;
../sqlite/tool/lemon.c:3900:18: error: assigning to 'Boolean' from incompatible type 'int'
    rp->noCode = 0;
../sqlite/tool/lemon.c:4428:32: error: assigning to 'Boolean' from incompatible type 'int'
        ap->x.rp->doesReduce = 1;
../sqlite/tool/lemon.c:4743:28: error: assigning to 'Boolean' from incompatible type 'int'
        rp2->codeEmitted = 1;
../sqlite/tool/lemon.c:4748:23: error: assigning to 'Boolean' from incompatible type 'int'
    rp->codeEmitted = 1;

My diagnostic would be that `lemon` is simply not meant to be compiled with a C++ compiler, hence I should first compile it with `gcc` and only use a C++ compiler in a second time for the actual SQLite source.

So that's what I did: I changed `g++` back to `gcc` and re-ran `make`. I get these errors again (but I expect it, as I'm not linking the C++ standard library / using GCC):
212 warnings generated.
Undefined symbols for architecture x86_64:
  "std::__1::locale::use_facet(std::__1::locale::id&) const", referenced from:
      std::__1::ctype<char> const& std::__1::use_facet<std::__1::ctype<char> >(std::__1::locale const&) in libsqlite3.a(thomas.o)
  "std::__1::ios_base::getloc() const", referenced from:
      std::__1::basic_ios<char, std::__1::char_traits<char> >::widen(char) const in libsqlite3.a(thomas.o)
  "std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::__init(unsigned long, char)", referenced from:
      std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::basic_string(unsigned long, char) in libsqlite3.a(thomas.o)
  "std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::~basic_string()", referenced from:
      std::__1::ostreambuf_iterator<char, std::__1::char_traits<char> > std::__1::__pad_and_output<char, std::__1::char_traits<char> >(std::__1::ostreambuf_iterator<char, std::__1::char_traits<char> >, char const*, char const*, char const*, std::__1::ios_base&, char) in libsqlite3.a(thomas.o)
  "std::__1::basic_ostream<char, std::__1::char_traits<char> >::put(char)", referenced from:
      std::__1::basic_ostream<char, std::__1::char_traits<char> >& std::__1::endl<char, std::__1::char_traits<char> >(std::__1::basic_ostream<char, std::__1::char_traits<char> >&) in libsqlite3.a(thomas.o)

After changing back to `g++`:

BCC = g++ -x none -g -O0
TCC = g++ -x none -O0

I get several errors coming from compiling `shell.c`:

shell.c:817:12: error: assigning to 'char *' from incompatible type 'void *'
    p->z = realloc(p->z, p->nAlloc);
shell.c:868:16: warning: conversion from string literal to 'char *' is deprecated [-Wc++11-compat-deprecated-writable-strings]
  char *zDiv = "(";
shell.c:888:12: warning: conversion from string literal to 'char *' is deprecated [-Wc++11-compat-deprecated-writable-strings]
    zDiv = ",";
shell.c:1883:5: error: no matching function for call to 'SHA3Update'
    SHA3Update(&cx, sqlite3_value_blob(argv[0]), nByte);
shell.c:1793:13: note: candidate function not viable: cannot convert argument of incomplete type 'const void *' to 'const unsigned char *' for 2nd argument
static void SHA3Update(
shell.c:2037:34: error: cannot initialize a variable of type 'const unsigned char *' with an rvalue of type 'const void *'
            const unsigned char *z2 = sqlite3_column_blob(pStmt, i);
                                 ^    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

All the errors come from compiling the shell, though I don't know whether it is because it can't compile the shell or if it is because the shell is the first of several files that can't be compiled. 

Once again any suggestions are welcome.

(8) By anonymous on 2020-03-14 16:07:01 in reply to 7 [link]

Okay, so right now I got to change the `main.mk` a bit and finally got it to work. 

So for future reference: 

* I have a first `make` pass where I don't change any file (to compile `lemon`, possibly among others)
* Then, I edit `Makefile.linux-gcc` and replace all occurrences of `gcc` with `gcc -x none`
* I change `main.mk` and add change the executable target by adding `-lstdc++` at the end:

sqlite3$(EXE):	shell.c libsqlite3.a sqlite3.h
	$(TCCX) $(READLINE_FLAGS) -o sqlite3$(EXE) $(SHELL_OPT) \
		shell.c libsqlite3.a $(LIBREADLINE) $(TLIBS) $(THREADLIB) -lstdc++

* I run make again using `make -f Makefile.linux-gcc` and get my `sqlite3` executable as expected.

Thanks everyone for your support, you were of great help.

(9) By Keith Medcalf (kmedcalf) on 2020-03-14 16:43:17 in reply to 8 [link]

I should think that you do not need to change "gcc" to "gcc -x none".  As far as I know the default for the gcc driver is -x none.  g++ is the one that changes the defaults.

Also, I believe your other issues are because the language level defaults are probably also changed to incompatible settings when using the g++ command to compile c code because using g++ as the driver implies that you want C++ compatibility, not standard C.

You really should only need to change the linker steps to include the additional C++ libraries (and of course to use your new .cpp source file).

(10) By anonymous on 2020-03-17 08:33:04 in reply to 9 [link]

Thank you, I'll try without.