SQLite Forum

binary .dump format
Login
> it's not even theoretically possible 

Not really. Binary dumps typically just prefix arbitrary byte sequences with a length.  
Yes, that makes the dump *context-dependent*, and any missing or corrupt byte can make  
the whole dump corrupt, but they are ways to detect such cases too.

You could also encode (base16, base64, ascci85) small blobs in the main dump file,  
and offload larger blobs into a separate paged companion file. One could even use  
content addressable schemes (e.g SHA-3 like Fossil) for blobs, gaining blob dedup too.

For such a large DB, using a snapshot'ing file-system, like ZFS, would allow instant  
*shallow copies*, which would allow to compute a diff (via SQLite's own DB diff tool)  
where time is not as critical, and keeping only the diff for backups perhaps.