Cloud Backed SQLite

Hi,

I was running into trouble vacuuming...

Our current workflow is:

1. copy previousDb to a new db entry
2. make changes to the new db entry.
3. vacuum db if necessary (assume it happens in this case)
4. sqlite3_bcvfs_upload() (pragma bcv_upload also reproduces the error)
5. detach, "container has local changes" error.

I can prevent the "container has local changes" error if I add another step to the above workflow, 2a which calls upload again.

Taking a look at the logs in the case where I don't upload twice. I can see dirty blocks which aren't uploaded. block 21 of imodelblocks-3cfe9b0d-c19e-48dd-a67d-8fb78c971d4b/912c1c83ef5529214b66a4bd6fca9c5e28d250ac.bim is now dirty block C781CF07F97A083BFE2C9F512978F13D.bcv (block 21 in cache file)

Even though during the upload operation, there are many PUTs like: r3_7 [] PUT http://127.0.0.1:10000/devstoreaccount1/imodelblocks-3cfe9b0d-c19e-48dd-a67d-8fb78c971d4b/8A759AAC611FC20AAAFD84EC0503C066.bcv... There is not a corresponding PUT for block with blockId: C781CF07F97A083BFE2C9F512978F13D

This makes sense, probably because after vacuuming that block is no longer considered part of the database. As a result, there are local dirty blocks which no longer belong to a database and have never made it to cloud storage (due to the vacuum). I'll attach the log I am referring to.

Any other information you might need?

Is there some function we can call to clean these blocks up? I know theres the sqlite3_bcv_cleanup but that seems intended for blocks which have made it to cloud storage.

Thanks for posting the log file.

Hopefully fixed now. Can you try with the latest trunk?

https://sqlite.org/cloudsqlite/info/bf008eee6a6ebcd2

Thanks,

Dan.

Will do, thanks!

Hi,

Thanks for the fix, that worked.

I figured I'd make use of this ticket again, since its related to vacuuming. We're now seeing occasional SEGFAULTs with a SEGV_MAPERR while vacuuming. Sometimes they don't segfault when retrying these vacuum operations, sometimes they segfault again.

I'll attach a log where the vacuuming segfaulted on both attempts. We have a callstack:

bcvfsCopyBlock
(blockcachevfs.c:1032)

bcvfsMakeBlockWritable
(blockcachevfs.c:1380)

bcvfsReadWriteDatabase
(blockcachevfs.c:1479)

bcvfsReadWriteDatabaseWithRetry
(blockcachevfs.c:1557)

sqlite3OsWrite
(sqlite3.c:24708)

walCheckpoint
(sqlite3.c:63730)

sqlite3WalCheckpoint
(sqlite3.c:65512)

sqlite3PagerCheckpoint
(sqlite3.c:61294)

sqlite3BtreeCheckpoint
(sqlite3.c:77671)

sqlite3Checkpoint
(sqlite3.c:173107)

sqlite3_wal_checkpoint_v2
(sqlite3.c:173045)

sqlite3_wal_checkpoint
(sqlite3.c:173070)

sqlite3WalDefaultHook
(sqlite3.c:172937)

doWalCallbacks
(sqlite3.c:86557)

sqlite3Step
(sqlite3.c:86684)

sqlite3_step
(sqlite3.c:86728)

sqlite3_exec
(sqlite3.c:129404)

It looks like the segfault is occurring while trying to determine the number of trailing zeroes. We're still working on debugging a full coredump. Do you anticipate that will be necessary? Let me know if any other information is required as well.

Looks like that debugging code has "szBlk" where it should have "nCopy". In any case, it's not so useful now that the block truncation problem has been fixed so we'll just remove it:

https://sqlite.org/cloudsqlite/info/da6e6920a29c3b41

Hopefully working now.

Thanks,

Dan.