SQLite Forum

lsm1 compression
Login

lsm1 compression

(1) By baraherself on 2020-09-03 05:48:39

Hello!

Many thanks for the lsm1 extension! I'm prototyping an application using the lsm1 extension that will benefit from compression. I have code working for small test ingests, but not for a larger test ingest. Help?

Details... 

I found this project, for the sqlite4 lsm:

https://github.com/thoughtpolice/sqlite4_lsm_lz4

Based on that, I've been working with this build directory, which works fine for small test ingests:

https://github.com/galgeek/sqlite-lsm1-lz4

A larger ingest without compression creates an 11 GB .lsm file.  Attempts to do the same ingest using compression fail after ~5-10 minutes, with this error:

`Assertion failed: (iPg>=fsFirstPageOnBlock(pFS, 1)), function fsPageGet, file lsm_file.c, line 1555.`

https://www.sqlite.org/cgi/src/file?name=ext%2Flsm1%2Flsm_file.c&ln=1555

I've tried commenting out the two lines of assertions there.

Then, retrying the larger ingest, again after ~5-10 minutes, the app fails with this error:

`Assertion failed: ((rc==LSM_OK && (p || (pnSpace && *pnSpace))) || (rc!=LSM_OK && p==0)), function fsPageGet, file lsm_file.c, line 1632.`

https://www.sqlite.org/cgi/src/file?name=ext%2Flsm1%2Flsm_file.c&ln=1630-1632

I've commented out this assertion, too, and tried again.

The latest error:  `segmentation fault`

some detail from the MacOs Problem Report:
```
Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0   lsm.so                        	0x000000010d83ce41 pageGetKey + 17
1   lsm.so                        	0x000000010d83e41c pageGetBtreeKey + 332
2   lsm.so                        	0x000000010d84033a seekInBtree + 378
3   lsm.so                        	0x000000010d83b92f sortedWork + 5599
4   lsm.so                        	0x000000010d839c18 doLsmWork + 392
5   lsm.so                        	0x000000010d83a24e lsmSortedAutoWork + 302
6   lsm.so                        	0x000000010d832583 doWriteOp + 323
7   lsm.so                        	0x000000010d8488e6 lsm1Update + 2934
8   libsqlite3.0.dylib            	0x000000010d653ab0 sqlite3VdbeExec + 8425
9   libsqlite3.0.dylib            	0x000000010d62e256 sqlite3_step + 391
10  _sqlite3.cpython-38-darwin.so 	0x000000010d61974f pysqlite_step + 31
11  _sqlite3.cpython-38-darwin.so 	0x000000010d616dfe _pysqlite_query_execute + 952
12  org.python.python             	0x000000010be3718a cfunction_call_varargs + 319
13  _sqlite3.cpython-38-darwin.so 	0x000000010d615b3d pysqlite_connection_executemany + 74
14  org.python.python             	0x000000010be3e012 method_vectorcall_VARARGS + 270
```

(2) By Dan Kennedy (dan) on 2020-09-04 08:13:55 in reply to 1 [link]

I'm not sure what version those github repositories are using. Do you get the same problem with the LSM code here?

[](https://sqlite.org/src/dir?ci=tip&name=ext/lsm1)

Dan.

(3) By baraherself on 2020-09-04 18:23:45 in reply to 2 [link]

I'm pretty sure that's the code that I'm using.

I created the build dir Tuesday, September 1:

```
fossil clone https://www.sqlite.org/src sqlite.fossil
mkdir sqlite3 ; cd sqlite3
fossil open ../sqlite.fossil
```

I copied code from a clone of [https://github.com/thoughtpolice/sqlite4_lsm_lz4](https://github.com/thoughtpolice/sqlite4_lsm_lz4):

```
sqlite4_lsm_lz4 % cp -p *.c *.h lz4/*.c lz4/*.h ../sqlite3/ext/lsm1
sqlite4_lsm_lz4 % ls *.c *.h lz4/*.c lz4/*.h
lsm-lz4.c		lsm-test.c		lz4/lz4.h		lz4/lz4_encoder.h
lsm-lz4.h		lz4/lz4.c		lz4/lz4_decoder.h
```

I edited the Makefile:
[https://github.com/galgeek/sqlite-lsm1-lz4/commit/3aae5c28eff92f7a7ff85b75b91b105239ed0de9](https://github.com/galgeek/sqlite-lsm1-lz4/commit/3aae5c28eff92f7a7ff85b75b91b105239ed0de9)

I edited vtab.c:
[https://github.com/galgeek/sqlite-lsm1-lz4/commit/4ed20a55afcd70f3595438e2d3f7d8b47f89cb59](https://github.com/galgeek/sqlite-lsm1-lz4/commit/4ed20a55afcd70f3595438e2d3f7d8b47f89cb59)

(5) By Dan Kennedy (dan) on 2020-09-05 14:59:13 in reply to 3 [link]

How can the problem be reproduced?

If I checkout this:

[](https://github.com/galgeek/sqlite-lsm1-lz4)

is there something to run that will produce the crash?

Thanks,
Dan.

(6) By baraherself on 2020-09-05 20:18:28 in reply to 5 [link]

The lsm datastore was created using a Python webapp (built using sanic) running under Python 3.8. The webapp logs these version numbers:

```
[2020-09-05 12:24:57 -0700] [95053] [INFO] sqlite3.version is 2.6.0
[2020-09-05 12:24:57 -0700] [95053] [INFO] sqlite3.sqlite_version is 3.33.0

```

lsm.so was built from `SQLite version 3.34.0 2020-09-01 19:02:52`.

Maybe the version mismatch is the underlying issue?

Here's the webapp datastore code (we also use a standard sqlite table):
```
        self.conn = sqlite3.connect(":memory:")
        self.conn.create_function("REGEXP", 2, regex_match)
        self.conn.enable_load_extension(True)
        self.conn.load_extension("lsm.so")
        self.conn.execute('''CREATE VIRTUAL TABLE cdx USING lsm1 ('collections/%s/%s.lsm', key, TEXT, cdxline);''' % (name, name,) )
        self.replicator = sqlite3.connect("collections/%s/%s.replication.sqlite" % (name, name,))
```

I've tried to open the .lsm within the sqlite3 cli, but I'm missing something (`Error: lsm_open failed with 266`):
```
% ./sqlite3
SQLite version 3.34.0 2020-09-01 19:02:52
Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.
sqlite> .load ext/lsm1/lsm.so
sqlite> CREATE VIRTUAL TABLE cdx USING lsm1 ('/Users/baraherself/collections/lz4_weetest/lz4_weetest.lsm', key, TEXT, cdxline);
Error: lsm_open failed with 266
sqlite> 
```

I can provide webapp code, the ingested data, or the troubled .lsm, if helpful.

(7.1) By Dan Kennedy (dan) on 2020-09-06 09:58:10 edited from 7.0 in reply to 6 [link]

266 means a call to open() returned ENOENT. Typo in the directory name?

AFAICT you're not doing anything wrong. That assert() failing most likely indicates some problem in the LSM incremental merge code. But it might also be a problem in the compression wrappers, or something in the host app (i.e. heap corruption). Assuming it is an LSM bug it will be subtle - or it would have shown up in testing already.

To fix this I need some way to reproduce the problem locally. Are you able to create a C program, or a python script, or an SQL script for the shell tool (or anything really) that does that? 

We already have tests for "big databases that use compression" - so I suspect if I just try to go fishing for the bug it might take quite a while...

Cheers,

Dan.

(8) By baraherself on 2020-09-13 19:43:11 in reply to 7.1 [link]

Thanks, Dan!

With the correct path name for the lsm datastore, I can open it using the sqlite3 cli, and eventually hit an error inserting an additional line. So far, this has been a different error than my webapp has hit.

You may be able to reproduce...

by building lsm.so from a clone of [https://github.com/galgeek/sqlite-lsm1-lz4](https://github.com/galgeek/sqlite-lsm1-lz4)
(my build commands are in the README)

and

loading lsm.so, and the datastore, and inserting one or more lines, 
as below.  
(lsm datastore: [https://drive.google.com/file/d/1fJFz9vdXaE8ehBqX5ARUN5toBhKBFaZc/view?usp=sharing](https://drive.google.com/file/d/1fJFz9vdXaE8ehBqX5ARUN5toBhKBFaZc/view?usp=sharing))

```
baraherself@b-ma ~/D/sqlite3 > ./sqlite3
SQLite version 3.34.0 2020-09-01 19:02:52
Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.
sqlite> .load ext/lsm1/lsm.so
sqlite> CREATE VIRTUAL TABLE cdx USING lsm1 ('/Users/baraherself/Dev/cdxlite/collections/lz4_weetest/lz4_weetest.lsm', key, TEXT, cdxline);
sqlite> select count(*) from cdx;
8741534

# separately, I ran the python/sanic webapp, posting insertions, and again it failed with segmentation fault.
# here, the lsm store remains open, and inserts of several lines succeed...

sqlite> insert into cdx (key, cdxline) values ('ai,ak)/20200621000621','ai,ak)/ 20200621000621 http://ak.ai/ text/html 200 7OJCBOXLSJRUTLFALBSMSD5XHAWSMGO3 - - 5907 12823972 ARCHIVEIT-9810-DAILY-JOB1205229-SEED2290404-20200620230127543-00000-h3.warc.gz');
sqlite> insert into cdx (key, cdxline) values ('ai,ak)/20200621000622','ai,ak)/ 20200621000621 http://ak.ai/ text/html 200 7OJCBOXLSJRUTLFALBSMSD5XHAWSMGO3 - - 5907 12823972 ARCHIVEIT-9810-DAILY-JOB1205229-SEED2290404-20200620230127543-00000-h3.warc.gz');
sqlite> insert into cdx (key, cdxline) values ('ai,ak)/20200621000623','ai,ak)/ 20200621000621 http://ak.ai/ text/html 200 7OJCBOXLSJRUTLFALBSMSD5XHAWSMGO3 - - 5907 12823972 ARCHIVEIT-9810-DAILY-JOB1205229-SEED2290404-20200620230127543-00000-h3.warc.gz');
sqlite> select count(*) from cdx;
8741537

# let's try another... 

sqlite> insert into cdx (key, cdxline) values ('ai,ak)/20200621000625','ai,ak)/ 20200621000621 http://ak.ai/ text/html 200 7OJCBOXLSJRUTLFALBSMSD5XHAWSMGO3 - - 5907 12823972 ARCHIVEIT-9810-DAILY-JOB1205229-SEED2290404-20200620230127543-00000-h3.warc.gz');

# nope!  note: this is a new error, at a different location in the lsm code

Assertion failed: (aSpace[aSort[i].pShm->iShmid - iPrevShmid].pShm==0), function treeRepairList, file lsm_tree.c, line 1312.
fish: './sqlite3' terminated by signal SIGABRT (Abort)
```
I'm hoping to find some debugging time myself!

And checking my ability to reproduce, I eventually hit another error:
```
sqlite> insert into cdx (key, cdxline) values ('ai,ak)/20200621000639','ai,ak)/ 20200621000621 http://ak.ai/ text/html 200 7OJCBOXLSJRUTLFALBSMSD5XHAWSMGO3 - - 5907 12823972 ARCHIVEIT-9810-DAILY-JOB1205229-SEED2290404-20200620230127543-00000-h3.warc.gz');
Error: SQL logic error
sqlite3(12092,0x10d671dc0) malloc: Incorrect checksum for freed object 0x7f8936c06c60: probably modified after being freed.
Corrupt value: 0x1
sqlite3(12092,0x10d671dc0) malloc: *** set a breakpoint in malloc_error_break to debug
fish: './sqlite3' terminated by signal SIGABRT (Abort)
```

(9) By baraherself on 2020-09-13 20:15:19 in reply to 8 [link]

Here's a multi-line insert, for what it's worth.  
```
insert into cdx (key, cdxline) values ('ai,ak)/20200621000641','ai,ak)/ 20200621000621 http://ak.ai/ text/html 200 7OJCBOXLSJRUTLFALBSMSD5XHAWSMGO3 - - 5907 12823972 ARCHIVEIT-9810-DAILY-JOB1205229-SEED2290404-20200620230127543-00000-h3.warc.gz'), 
('ai,ak)/20200621000642','ai,ak)/ 20200621000621 http://ak.ai/ text/html 200 7OJCBOXLSJRUTLFALBSMSD5XHAWSMGO3 - - 5907 12823972 ARCHIVEIT-9810-DAILY-JOB1205229-SEED2290404-20200620230127543-00000-h3.warc.gz'), 
('ai,ak)/20200621000643','ai,ak)/ 20200621000621 http://ak.ai/ text/html 200 7OJCBOXLSJRUTLFALBSMSD5XHAWSMGO3 - - 5907 12823972 ARCHIVEIT-9810-DAILY-JOB1205229-SEED2290404-20200620230127543-00000-h3.warc.gz'), 
('ai,ak)/20200621000644','ai,ak)/ 20200621000621 http://ak.ai/ text/html 200 7OJCBOXLSJRUTLFALBSMSD5XHAWSMGO3 - - 5907 12823972 ARCHIVEIT-9810-DAILY-JOB1205229-SEED2290404-20200620230127543-00000-h3.warc.gz'), 
('ai,ak)/20200621000645','ai,ak)/ 20200621000621 http://ak.ai/ text/html 200 7OJCBOXLSJRUTLFALBSMSD5XHAWSMGO3 - - 5907 12823972 ARCHIVEIT-9810-DAILY-JOB1205229-SEED2290404-20200620230127543-00000-h3.warc.gz'), 
('ai,ak)/20200621000646','ai,ak)/ 20200621000621 http://ak.ai/ text/html 200 7OJCBOXLSJRUTLFALBSMSD5XHAWSMGO3 - - 5907 12823972 ARCHIVEIT-9810-DAILY-JOB1205229-SEED2290404-20200620230127543-00000-h3.warc.gz'), 
('ai,ak)/20200621000647','ai,ak)/ 20200621000621 http://ak.ai/ text/html 200 7OJCBOXLSJRUTLFALBSMSD5XHAWSMGO3 - - 5907 12823972 ARCHIVEIT-9810-DAILY-JOB1205229-SEED2290404-20200620230127543-00000-h3.warc.gz'),
('ai,ak)/20200621000648','ai,ak)/ 20200621000621 http://ak.ai/ text/html 200 7OJCBOXLSJRUTLFALBSMSD5XHAWSMGO3 - - 5907 12823972 ARCHIVEIT-9810-DAILY-JOB1205229-SEED2290404-20200620230127543-00000-h3.warc.gz'),
('ai,ak)/20200621000649','ai,ak)/ 20200621000621 http://ak.ai/ text/html 200 7OJCBOXLSJRUTLFALBSMSD5XHAWSMGO3 - - 5907 12823972 ARCHIVEIT-9810-DAILY-JOB1205229-SEED2290404-20200620230127543-00000-h3.warc.gz'),
('ai,ak)/20200621000650','ai,ak)/ 20200621000621 http://ak.ai/ text/html 200 7OJCBOXLSJRUTLFALBSMSD5XHAWSMGO3 - - 5907 12823972 ARCHIVEIT-9810-DAILY-JOB1205229-SEED2290404-20200620230127543-00000-h3.warc.gz');
```

(10) By Dan Kennedy (dan) on 2020-09-14 14:40:20 in reply to 8 [link]

Cannot reproduce here.

I copied your lz4 files into the ext/lsm1 directory of SQLite, applied the patch below to build the shell tool with lsm and lz4 support built-in, then ran the following multiple times:

<verbatim>
  CREATE VIRTUAL TABLE cdx USING lsm1 ('/home/dan/work/sqlite/bld4/lz4_weetest.lsm', key, TEXT, cdxline);
  insert into cdx (key, cdxline) values ('ai,ak)/20200621000621','ai,ak)/ 20200621000621 http://ak.ai/ text/html 200 7OJCBOXLSJRUTLFALBSMSD5XHAWSMGO3 - - 5907 12823972 ARCHIVEIT-9810-DAILY-JOB1205229-SEED2290404-20200620230127543-00000-h3.warc.gz');
  insert into cdx (key, cdxline) values ('ai,ak)/20200621000621','ai,ak)/ 20200621000621 http://ak.ai/ text/html 200 7OJCBOXLSJRUTLFALBSMSD5XHAWSMGO3 - - 5907 12823972 ARCHIVEIT-9810-DAILY-JOB1205229-SEED2290404-20200620230127543-00000-h3.warc.gz');
  insert into cdx (key, cdxline) values ('ai,ak)/20200621000622','ai,ak)/ 20200621000621 http://ak.ai/ text/html 200 7OJCBOXLSJRUTLFALBSMSD5XHAWSMGO3 - - 5907 12823972 ARCHIVEIT-9810-DAILY-JOB1205229-SEED2290404-20200620230127543-00000-h3.warc.gz');
  insert into cdx (key, cdxline) values ('ai,ak)/20200621000623','ai,ak)/ 20200621000621 http://ak.ai/ text/html 200 7OJCBOXLSJRUTLFALBSMSD5XHAWSMGO3 - - 5907 12823972 ARCHIVEIT-9810-DAILY-JOB1205229-SEED2290404-20200620230127543-00000-h3.warc.gz');
  insert into cdx (key, cdxline) values ('ai,ak)/20200621000625','ai,ak)/ 20200621000621 http://ak.ai/ text/html 200 7OJCBOXLSJRUTLFALBSMSD5XHAWSMGO3 - - 5907 12823972 ARCHIVEIT-9810-DAILY-JOB1205229-SEED2290404-20200620230127543-00000-h3.warc.gz');
  insert into cdx (key, cdxline) values ('ai,ak)/20200621000639','ai,ak)/ 20200621000621 http://ak.ai/ text/html 200 7OJCBOXLSJRUTLFALBSMSD5XHAWSMGO3 - - 5907 12823972 ARCHIVEIT-9810-DAILY-JOB1205229-SEED2290404-20200620230127543-00000-h3.warc.gz');
</verbatim>

but, no error. Not under valgrind or with asan either. Any ideas?

<verbatim>
Index: ext/lsm1/Makefile
==================================================================
--- ext/lsm1/Makefile
+++ ext/lsm1/Makefile
@@ -11,10 +11,11 @@
 
 LSMOBJ    = \
   lsm_ckpt.o \
   lsm_file.o \
   lsm_log.o \
+  lsm-lz4.o \
   lsm_main.o \
   lsm_mem.o \
   lsm_mutex.o \
   lsm_shared.o \
   lsm_sorted.o \
@@ -21,15 +22,20 @@
   lsm_str.o \
   lsm_tree.o \
   lsm_unix.o \
   lsm_win32.o \
   lsm_varint.o \
-  lsm_vtab.o
+  lsm_vtab.o \
+  lz4.o
 
 LSMHDR   = \
   $(LSMDIR)/lsm.h \
-  $(LSMDIR)/lsmInt.h
+  $(LSMDIR)/lsmInt.h \
+  $(LSMDIR)/lsm-lz4.h \
+  $(LSMDIR)/lz4.h \
+  $(LSMDIR)/lz4_decoder.h \
+  $(LSMDIR)/lz4_encoder.h
 
 LSMTESTSRC = $(LSMDIR)/lsm-test/lsmtest1.c $(LSMDIR)/lsm-test/lsmtest2.c     \
              $(LSMDIR)/lsm-test/lsmtest3.c $(LSMDIR)/lsm-test/lsmtest4.c     \
              $(LSMDIR)/lsm-test/lsmtest5.c $(LSMDIR)/lsm-test/lsmtest6.c     \
              $(LSMDIR)/lsm-test/lsmtest7.c $(LSMDIR)/lsm-test/lsmtest8.c     \
@@ -41,16 +47,17 @@
              $(LSMDIR)/lsm-test/lsmtest_util.c $(LSMDIR)/lsm-test/lsmtest_win32.c
 
 
 # all: lsm.so
 
-LSMOPTS += -fPIC -DLSM_MUTEX_PTHREADS=1 -I$(LSMDIR) -DHAVE_ZLIB
+LSMOPTS += -fPIC -DLSM_MUTEX_PTHREADS=1 -I$(LSMDIR) -DHAVE_ZLIB -DSQLITE_CORE
+LZ4_OPTS = -DUSE_LSM_LZ4_COMPRESSOR 
 
 lsm.so:	$(LSMOBJ)
 	$(TCCX) -shared -fPIC -o lsm.so $(LSMOBJ)
 
 %.o:	$(LSMDIR)/%.c $(LSMHDR) sqlite3.h
-	$(TCCX) $(LSMOPTS) -c $<
+	$(TCCX) $(LSMOPTS) $(LZ4_OPTS) -c $<
 	
 lsmtest$(EXE): $(LSMOBJ) $(LSMTESTSRC) $(LSMTESTHDR) sqlite3.o
 	# $(TCPPX) -c $(TOP)/lsm-test/lsmtest_tdb2.cc
-	$(TCCX) $(LSMOPTS) $(LSMTESTSRC) $(LSMOBJ) sqlite3.o -o lsmtest$(EXE) $(THREADLIB) -lz
+	$(TCCX) $(LSMOPTS) $(LZ4_OPTS) $(LSMTESTSRC) $(LSMOBJ) sqlite3.o -o lsmtest$(EXE) $(THREADLIB) -lz

Index: ext/lsm1/lsm_vtab.c
==================================================================
--- ext/lsm1/lsm_vtab.c
+++ ext/lsm1/lsm_vtab.c
@@ -90,10 +90,11 @@
 #include "sqlite3ext.h"
 SQLITE_EXTENSION_INIT1
 #include "lsm.h"
 #include <assert.h>
 #include <string.h>
+#include "lsm-lz4.h"
 
 /* Forward declaration of subclasses of virtual table objects */
 typedef struct lsm1_vtab lsm1_vtab;
 typedef struct lsm1_cursor lsm1_cursor;
 typedef struct lsm1_vblob lsm1_vblob;
@@ -263,10 +264,16 @@
   rc = lsm_new(0, &pNew->pDb);
   if( rc ){
     *pzErr = sqlite3_mprintf("lsm_new failed with error code %d",  rc);
     rc = SQLITE_ERROR;
     goto connect_failed;
+  }
+  rc = lsm_lz4_config(pNew->pDb);
+  if( rc ){
+    *pzErr = sqlite3_mprintf("lsm_lz4_config failed with error code %d",  rc);
+    rc = SQLITE_ERROR;
+    goto connect_failed;
   }
   zFilename = sqlite3_mprintf("%s", argv[3]);
   lsm1Dequote(zFilename);
   rc = lsm_open(pNew->pDb, zFilename);
   sqlite3_free(zFilename);

Index: main.mk
==================================================================
--- main.mk
+++ main.mk
@@ -48,10 +48,13 @@
 TCCX += -I$(TOP)/ext/async -I$(TOP)/ext/userauth
 TCCX += -I$(TOP)/ext/session
 TCCX += -I$(TOP)/ext/fts5
 THREADLIB += $(LIBS)
 
+LSMDIR=$(TOP)/ext/lsm1/
+include $(LSMDIR)/Makefile
+
 # Object files for the SQLite library.
 #
 LIBOBJ+= vdbe.o parse.o \
          alter.o analyze.o attach.o auth.o \
          backup.o bitvec.o btmutex.o btree.o build.o \
@@ -74,11 +77,11 @@
          table.o threads.o tokenize.o treeview.o trigger.o \
          update.o upsert.o userauth.o util.o vacuum.o \
          vdbeapi.o vdbeaux.o vdbeblob.o vdbemem.o vdbesort.o \
 	 vdbetrace.o vdbevtab.o \
          wal.o walker.o where.o wherecode.o whereexpr.o \
-         utf.o vtab.o window.o
+         utf.o vtab.o window.o $(LSMOBJ)
 
 LIBOBJ += sqlite3session.o
 
 # All of the source code files.
 #
@@ -549,11 +552,11 @@
 # This is the default Makefile target.  The objects listed here
 # are what get build when you type just "make" with no arguments.
 #
 all:	sqlite3.h libsqlite3.a sqlite3$(EXE)
 
-libsqlite3.a:	$(LIBOBJ)
+libsqlite3.a: $(LIBOBJ) 
 	$(AR) libsqlite3.a $(LIBOBJ)
 	$(RANLIB) libsqlite3.a
 
 sqlite3$(EXE):	shell.c libsqlite3.a sqlite3.h
 	$(TCCX) $(READLINE_FLAGS) -o sqlite3$(EXE) $(SHELL_OPT) \

Index: src/main.c
==================================================================
--- src/main.c
+++ src/main.c
@@ -3350,10 +3350,12 @@
 
   /* Load automatic extensions - extensions that have been registered
   ** using the sqlite3_automatic_extension() API.
   */
   if( rc==SQLITE_OK ){
+    int sqlite3_lsm_init(sqlite3 *, char **, const sqlite3_api_routines *);
+    sqlite3_lsm_init(db, 0, 0);
     sqlite3AutoLoadExtensions(db);
     rc = sqlite3_errcode(db);
     if( rc!=SQLITE_OK ){
       goto opendb_out;
     }
</verbatim>

(11) By baraherself on 2020-09-18 06:18:40 in reply to 10 [link]

Thank you, Dan!

I've applied your patch and attempted to build the shell tool with lsm and lz4 support built-in.  I've tried a couple of variations on build commands. (I've consulted [https://www.sqlite.org/howtocompile.html](https://www.sqlite.org/howtocompile.html) and the READMEs in the autoconf dir.)  In each case, however, I've gotten this error:

```  
Undefined symbols for architecture x86_64:
  "_sqlite3_lsm_init", referenced from:
      _openDatabase in sqlite3.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [libsqlite3.la] Error 1
```
Do you have any more tips for replicating your build?

I'm developing on MacOS 10.15.6. I could easily try a new clone of sqlite3, or a build on ubuntu, where the app we're prototyping is most likely to run.

Thank you again!

(12) By Dan Kennedy (dan) on 2020-09-18 08:15:39 in reply to 11 [link]

Are you using an autoconf build? 

The patch above only patches the legacy build system. I use a Makefile like the following:

<verbatim>
TOP=/home/dan/work/sqlite/t4

TCL_FLAGS=-L/home/dan/tcl/lib/ -I/home/dan/tcl/include/
LIBTCL=-ltcl8.6 -lm

READLINE_FLAGS = -DHAVE_READLINE=1 -I/usr/include/readline
LIBREADLINE = -lreadline

BCC = gcc
OPTS +=    -DSQLITE_ENABLE_FTS5
OPTS +=    -DHAVE_USLEEP=1
OPTS +=    -DLSM_DEBUG

TCC = gcc -fPIC -g -Wall -I. -DSQLITE_DEBUG -DSQLITE_NO_SYNC

NAWK = awk
AR     = ar cr
RANLIB = ranlib

THREADLIB = -lpthread -ldl -lz -lm

###############################################################
include $(TOP)/main.mk
</verbatim>

(13) By baraherself on 2020-09-20 22:19:21 in reply to 12 [link]

Thanks, Dan!

Using a Makefile like yours, and your edits, I've been able to build sqlite3 with built-in lsm and lz4. I've inserted several blocks of lines into my 8-million line .lsm file, with no errors.

Now on to sort out how to load such a build in our app...

(4.1) By baraherself on 2020-09-13 19:44:36 edited from 4.0 in reply to 1 [link]

Deleted