SQLite Forum

lsm1 compression
Login
Hello!

Many thanks for the lsm1 extension! I'm prototyping an application using the lsm1 extension that will benefit from compression. I have code working for small test ingests, but not for a larger test ingest. Help?

Details... 

I found this project, for the sqlite4 lsm:

https://github.com/thoughtpolice/sqlite4_lsm_lz4

Based on that, I've been working with this build directory, which works fine for small test ingests:

https://github.com/galgeek/sqlite-lsm1-lz4

A larger ingest without compression creates an 11 GB .lsm file.  Attempts to do the same ingest using compression fail after ~5-10 minutes, with this error:

`Assertion failed: (iPg>=fsFirstPageOnBlock(pFS, 1)), function fsPageGet, file lsm_file.c, line 1555.`

https://www.sqlite.org/cgi/src/file?name=ext%2Flsm1%2Flsm_file.c&ln=1555

I've tried commenting out the two lines of assertions there.

Then, retrying the larger ingest, again after ~5-10 minutes, the app fails with this error:

`Assertion failed: ((rc==LSM_OK && (p || (pnSpace && *pnSpace))) || (rc!=LSM_OK && p==0)), function fsPageGet, file lsm_file.c, line 1632.`

https://www.sqlite.org/cgi/src/file?name=ext%2Flsm1%2Flsm_file.c&ln=1630-1632

I've commented out this assertion, too, and tried again.

The latest error:  `segmentation fault`

some detail from the MacOs Problem Report:
```
Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0   lsm.so                        	0x000000010d83ce41 pageGetKey + 17
1   lsm.so                        	0x000000010d83e41c pageGetBtreeKey + 332
2   lsm.so                        	0x000000010d84033a seekInBtree + 378
3   lsm.so                        	0x000000010d83b92f sortedWork + 5599
4   lsm.so                        	0x000000010d839c18 doLsmWork + 392
5   lsm.so                        	0x000000010d83a24e lsmSortedAutoWork + 302
6   lsm.so                        	0x000000010d832583 doWriteOp + 323
7   lsm.so                        	0x000000010d8488e6 lsm1Update + 2934
8   libsqlite3.0.dylib            	0x000000010d653ab0 sqlite3VdbeExec + 8425
9   libsqlite3.0.dylib            	0x000000010d62e256 sqlite3_step + 391
10  _sqlite3.cpython-38-darwin.so 	0x000000010d61974f pysqlite_step + 31
11  _sqlite3.cpython-38-darwin.so 	0x000000010d616dfe _pysqlite_query_execute + 952
12  org.python.python             	0x000000010be3718a cfunction_call_varargs + 319
13  _sqlite3.cpython-38-darwin.so 	0x000000010d615b3d pysqlite_connection_executemany + 74
14  org.python.python             	0x000000010be3e012 method_vectorcall_VARARGS + 270
```