SQLite Forum

Open In-Memory DB on existing memory
Login
Hi All,

<h2>Synopsis:</h2>
For one of our projects we receive a fully-formed SQLite DB that was previously written to disk, aka the "log file", then read as bytes into a stream and sent over TCP-IP to a Server which collects and stores all the logs from all the devices. ["Devices" here are technical field-tablets or loggers that use hard-to-update firmware and have limited resources]

At this point frequency was low, but becoming more and will increase with normal growth in future (hence this optimization question) and file-sizes are small - typically 4 to 64 MB in size - and is not foreseen to change much in future.


<h2>Problem:</h2>
Server-side, what I do currently, is persist the bytes to physical storage (just write to a file), then open the SQLite DB (via sqlite API from dynamic-loading libraries[1]) as the persisted file - all of which works perfectly well but are very slow. From this, data is translated and transferred to teh Server-side DB which serves aggregated data to web-facing elements.  The system is platform/OS independent and can be used on any, but the typical is a web-server running CentOS.

Since I don't have control over the server environments, I can't easily force mounting in-memory RAM-drives or such, but I can use file-space and memory space as needed in my process. This makes the current way orders of magnitude slower than the actual "work" suggests.

[1] Simply using .so or .dll so that updates to SQLite can be rolled in or rolled back (in case of breaking changes) without needing to update the server software.


<h2>Questions:</h2>
I can open a NEW in-memory DB easily, but is there a way to open it from a given (at least initially) set of bytes? Read-Only perhaps?

I've thought of obtaining the initial allocated memory from SQLite (after making a standard new in-memory DB) and then simply re-size the memory and fill with my own sqlite-file bytes, but would rather like to avoid the effort if it is an obvious folly.

- Can I even get the memory location/size of the DB bytes from SQLite?
- Are the byte-structures of In-Memory and File-based SQLIte DBs the same?
- Can this be done in some way I am missing?

I'm hoping there's some simple easy thing I'm just unaware of.  
Any ideas welcome.