SQLite User Forum

Best way to take a backup with limited disk space?
Login

Best way to take a backup with limited disk space?

(1) By Simon Willison (simonw) on 2022-07-06 16:14:23 [link] [source]

As far as I can tell the recommended ways to take a snapshot backup of a SQLite database that may be accepting writes are either to use the backup API or to perform a VACUUM INTO against a new file.

Am I right in thinking that both of these options require having enough local disk space to create a (potentially shrunk) full copy of the existing database?

This seems like it would be a problem if you only have 10GB of disk space available and your main SQLite database it taking up 8GB already.

Are there any good strategies for performing backups if you don't have the space to create a whole additional copy? Streaming the backup over the network in some way, via SSH or similar?

(2) By Simon Willison (simonw) on 2022-07-06 16:16:31 in reply to 1 [source]

The main option I'm considering at the moment is https://litestream.io but I'm interested in hearing about other possibilities as well.

(3.1) By Keith Medcalf (kmedcalf) on 2022-07-06 16:34:22 edited from 3.0 in reply to 1 [link] [source]

Am I right in thinking that both of these options require having enough local disk space to create a (potentially shrunk) full copy of the existing database?

I don't quite get what you are meaning here. Clearly you need enough free disk space in the destination location to write the output file, which may equal but will not exceed the size of the file being copied. IMHO you should replace the word "local" with "destination free" in order to make a correct statement.

"Am I right in thinking that both of these options require having enough destination free disk space to create a (potentially shrunk) full copy of the existing database?"

The answer would be affirmative. The destination file may be up to the same size as the source file, but should not be larger. Unless, of course, the allocation units are different sizes on the different storage devices.

Since the target (where the backup/copy is written) of a backup or vacuum into is not subject to multiple access while being created, there is no reason that it cannot be written to a disk over yonder, across the street, on the next continent, or next universe over.

Of course, I would not recommend remote multiple access to the copy once it has been created, but I would expect that it would be created just fine.

(6) By Simon Willison (simonw) on 2022-07-06 19:59:50 in reply to 3.1 [link] [source]

My challenge is that the hosting environment I'm using here doesn't support attaching multiple volumes to a container - so I have a hard limit on the amount of local disk I can access. I'm trying to figure out a way to stream that backup over the network to another machine.

(4) By Stephan Beal (stephan) on 2022-07-06 16:36:22 in reply to 1 [link] [source]

Are there any good strategies for performing backups if you don't have the space to create a whole additional copy? Streaming the backup over the network in some way, via SSH or similar?

You can't, AFAIK, vacuum-into over a sequential-only stream (it requires random access), but you can "dump" to one with something along the lines of:

ssh the-db-host '/path/to/sqlite3 /path/to/mydb .dump | gzip -c' > mybackup.sql.gz

You can of course instead pipe it to:

ssh ... '... | gzip -c' | gzip -cd | sqlite3 mybackup.sqlite3

but that will be slower and tie up your remote db longer.

The "pv" tool is really nice for interactive use of such streams because it generates a human-readable transfer progress report. It's useless for unattended batch mode, though.

(7) By Simon Willison (simonw) on 2022-07-06 20:00:47 in reply to 4 [link] [source]

This looks like it might be the right solution for my particular challenge here, thanks!

(5) By anonymous on 2022-07-06 17:01:52 in reply to 1 [link] [source]

Why not nfs mount a network drive and use it as a backup destination, traditional nfs related caveats should not be an issue for this particular use case I assume

(8) By Simon Willison (simonw) on 2022-07-06 20:02:11 in reply to 5 [link] [source]

I'm working with containers here, but yeah maybe I can build myself a custom container which includes enough of a NFS client that I could do this - thanks for the tip!