SQLite Forum

lsm1 compression
Login
Thanks, Dan!

With the correct path name for the lsm datastore, I can open it using the sqlite3 cli, and eventually hit an error inserting an additional line. So far, this has been a different error than my webapp has hit.

You may be able to reproduce...

by building lsm.so from a clone of [https://github.com/galgeek/sqlite-lsm1-lz4](https://github.com/galgeek/sqlite-lsm1-lz4)
(my build commands are in the README)

and

loading lsm.so, and the datastore, and inserting one or more lines, 
as below.  
(lsm datastore: [https://drive.google.com/file/d/1fJFz9vdXaE8ehBqX5ARUN5toBhKBFaZc/view?usp=sharing](https://drive.google.com/file/d/1fJFz9vdXaE8ehBqX5ARUN5toBhKBFaZc/view?usp=sharing))

```
baraherself@b-ma ~/D/sqlite3 > ./sqlite3
SQLite version 3.34.0 2020-09-01 19:02:52
Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.
sqlite> .load ext/lsm1/lsm.so
sqlite> CREATE VIRTUAL TABLE cdx USING lsm1 ('/Users/baraherself/Dev/cdxlite/collections/lz4_weetest/lz4_weetest.lsm', key, TEXT, cdxline);
sqlite> select count(*) from cdx;
8741534

# separately, I ran the python/sanic webapp, posting insertions, and again it failed with segmentation fault.
# here, the lsm store remains open, and inserts of several lines succeed...

sqlite> insert into cdx (key, cdxline) values ('ai,ak)/20200621000621','ai,ak)/ 20200621000621 http://ak.ai/ text/html 200 7OJCBOXLSJRUTLFALBSMSD5XHAWSMGO3 - - 5907 12823972 ARCHIVEIT-9810-DAILY-JOB1205229-SEED2290404-20200620230127543-00000-h3.warc.gz');
sqlite> insert into cdx (key, cdxline) values ('ai,ak)/20200621000622','ai,ak)/ 20200621000621 http://ak.ai/ text/html 200 7OJCBOXLSJRUTLFALBSMSD5XHAWSMGO3 - - 5907 12823972 ARCHIVEIT-9810-DAILY-JOB1205229-SEED2290404-20200620230127543-00000-h3.warc.gz');
sqlite> insert into cdx (key, cdxline) values ('ai,ak)/20200621000623','ai,ak)/ 20200621000621 http://ak.ai/ text/html 200 7OJCBOXLSJRUTLFALBSMSD5XHAWSMGO3 - - 5907 12823972 ARCHIVEIT-9810-DAILY-JOB1205229-SEED2290404-20200620230127543-00000-h3.warc.gz');
sqlite> select count(*) from cdx;
8741537

# let's try another... 

sqlite> insert into cdx (key, cdxline) values ('ai,ak)/20200621000625','ai,ak)/ 20200621000621 http://ak.ai/ text/html 200 7OJCBOXLSJRUTLFALBSMSD5XHAWSMGO3 - - 5907 12823972 ARCHIVEIT-9810-DAILY-JOB1205229-SEED2290404-20200620230127543-00000-h3.warc.gz');

# nope!  note: this is a new error, at a different location in the lsm code

Assertion failed: (aSpace[aSort[i].pShm->iShmid - iPrevShmid].pShm==0), function treeRepairList, file lsm_tree.c, line 1312.
fish: './sqlite3' terminated by signal SIGABRT (Abort)
```
I'm hoping to find some debugging time myself!

And checking my ability to reproduce, I eventually hit another error:
```
sqlite> insert into cdx (key, cdxline) values ('ai,ak)/20200621000639','ai,ak)/ 20200621000621 http://ak.ai/ text/html 200 7OJCBOXLSJRUTLFALBSMSD5XHAWSMGO3 - - 5907 12823972 ARCHIVEIT-9810-DAILY-JOB1205229-SEED2290404-20200620230127543-00000-h3.warc.gz');
Error: SQL logic error
sqlite3(12092,0x10d671dc0) malloc: Incorrect checksum for freed object 0x7f8936c06c60: probably modified after being freed.
Corrupt value: 0x1
sqlite3(12092,0x10d671dc0) malloc: *** set a breakpoint in malloc_error_break to debug
fish: './sqlite3' terminated by signal SIGABRT (Abort)
```