SQLite Forum

can I insert an 8GB file into sqlite?
Login
> > copying the database just means copying a single file, and insuring that the checksums match.

> This is fine in principle, but when the size of the DB gets so big that it takes days and days to transfer, what happens when there's an interruption?

There are plenty of resumable, chunked transfer tools to deal with this; consider aria2 and BitTorrent for example. For better or for worse, it's done all the time in several scientific "big data" disciplines that have a taste for colossal individual files. See also HDF5, not to be confused with HDFS =)

> > What I need is fast IO and fast http transfers. Both of which are possible in javascript using libraries that leverage lower level C code.

We should note that Node.js JIT-compiles your script into native machine code. It opens a pretty nice speed/abstraction "sweet spot" between static compiled executables and interpreted scripts, which has been a big contributor to its success.