SQLite Forum

can I insert an 8GB file into sqlite?
Login
> There are plenty of resumable, chunked transfer tools to deal with this

As long as you're willing to shut all conns to the DB down and leave them down while the transfer continues, sure.

Resumability doesn't let you start the app up, modify the DB some, shut it down, and resume the transfer. That'll almost guarantee a corrupt DB at the end of the transfer. It also doesn't let you copy the DB once, do some work on the DB, and then copy only the changes, as `zfs send` will, given suitable snapshots.

Indeed, snapshotting the DB under ZFS and then doing a `zfs send` is a much better plan than using userspace-level resumable transfer tools on a SQLite DB. You might lose the last few modifications in uncommitted transactions, but it's no worse than kill the app while it runs. The journal gets replayed, and the DB is back in action again.

> it's done all the time in several scientific "big data" disciplines that have a taste for colossal individual files.

Are these individual files random-access, modified internally as SQLite DBs are, or are they written once, then possibly appended to only?

> Node.js JIT-compiles your script into native machine code.

Ah, that explains why VSCode feels about as fast as 4.3 BSD `vi` over a 9600 bps VT220 connection, then.

JIT isn't magic. Dynamic languages have unavoidable costs, no matter how well optimized, which is why statically-compiled languages are still being created and used.