Obtain information from DB header via PRAGMAs on truncated DBs
(1) By ddevienne on 2023-09-01 09:33:58 [link] [source]
I have lots of (potentially large) DBs in TAR files.
And I'd like to check the user_version of those DBs, which is part of the 100-byte header,
without having to untar potentially GBs of data.
So is there a way to make SQLite accept a truncated DB file,
for the limited purpose of issuing pragmas against the header?
Truncated to 100-bytes? Truncated to the max-page-size of 64KB?
That way the 1st page used for sqlite_master is always there,
and the full schema can be read (assuming small enough to fit in 64KB).
I know how to extract that value myself, if push comes to shove,
but if I can use the SQLite API for that somehow, that's be better.
I suspect it's not possible, but I'm hoping I'm wrong, thus my asking.
And maybe we could/should have a new public API to parse that header,
so that it's not re-invented all over the place (including my code)?
(2) By anonymous on 2023-09-01 10:45:32 in reply to 1 [link] [source]
Two quick experiments later...
CLI input:
.version .mode table .headers on pragma application_id; pragma data_version; pragma encoding; pragma freelist_count; pragma page_count; pragma page_size; pragma schema_version; pragma user_version;
Output from a database truncated to 64K:
SQLite 3.41.2 2023-03-22 11:56:21 0d1fc92f94cb6b76bffe3ec34d69cffde2924203304e8ffc4155597af0c191da zlib version 1.2.11 clang-12.0.0 Runtime error near line 4: database disk image is malformed (11) Runtime error near line 5: database disk image is malformed (11) Parse error near line 6: database disk image is malformed (11) Runtime error near line 7: database disk image is malformed (11) Parse error near line 8: database disk image is malformed (11) +-----------+ | page_size | +-----------+ | 4096 | +-----------+ Runtime error near line 10: database disk image is malformed (11) Runtime error near line 11: database disk image is malformed (11)
Output from a database truncated to 64K and then extended to the original size:
SQLite 3.41.2 2023-03-22 11:56:21 0d1fc92f94cb6b76bffe3ec34d69cffde2924203304e8ffc4155597af0c191da zlib version 1.2.11 clang-12.0.0 +----------------+ | application_id | +----------------+ | 0 | +----------------+ +--------------+ | data_version | +--------------+ | 1 | +--------------+ +----------+ | encoding | +----------+ | UTF-8 | +----------+ +----------------+ | freelist_count | +----------------+ | 10 | +----------------+ +------------+ | page_count | +------------+ | 689 | +------------+ +-----------+ | page_size | +-----------+ | 4096 | +-----------+ +----------------+ | schema_version | +----------------+ | 6 | +----------------+ +--------------+ | user_version | +--------------+ | 0 | +--------------+
[Insert the usual warnings about relying on undocumented behaviour here.]
(3) By Dan Kennedy (dan) on 2023-09-01 11:50:25 in reply to 1 [source]
I think if you open a db truncated to 64KiB, then use sqlite3_db_config() to set both SQLITE_DBCONFIG_TRUNCATED_SCHEMA and SQLITE_DBCONFIG_WRITABLE_SCHEMA, then SQLite will let you run PRAGMA statements to query header fields even though the file has been truncated.
Or, with the shell tool, add the "-unsafe-testing" option to the command line, then execute a "PRAGMA writable_schema=1" command. You should then be able to query a truncated db.
Dan.
(4.1) By ddevienne on 2023-09-01 15:59:57 edited from 4.0 in reply to 3 [link] [source]
Thanks Dan!
My (corporate) version of SQLite does NOT have SQLITE_DBCONFIG_TRUNCATED_SCHEMA (yet).
But SQLITE_DBCONFIG_WRITABLE_SCHEMA is enough in my case; apparently our schemas fit in 64KB.
Took a little while to reply, since I tested out the real thing,
i.e. partial untar (subset of files, truncated to 64KB) and running pragma user_version on those.
PS: I confirm SQLITE_CORRUPT is reported _step()'ing the pragma without it.
(5) By ddevienne on 2023-09-01 16:07:18 in reply to 3 [link] [source]
Actually Dan, SQLITE_DBCONFIG_TRUNCATED_SCHEMA is not even listed in the doc. Is it new and unreleased yet? Or somehow private?
(6) By Dan Kennedy (dan) on 2023-09-01 16:26:09 in reply to 5 [link] [source]
Sorry - should be "SQLITE_DBCONFIG_TRUSTED_SCHEMA".
Dan.
(7) By Aask (AAsk1902) on 2023-09-01 17:46:13 in reply to 1 [link] [source]
And I'd like to check the user_version of those DBs, which is part of the 100-byte header,without having to untar potentially GBs of data.
The user_version is an unsigned 4 bytes (32-bits) integer at bytes 60-63 inclusive.
I read (accessing the file/database in exclusive read-only mode) 32 bits starting at offset 60 (using C#) and converted to integer: the result is what pragma user_version returns.
(8) By ddevienne on 2023-09-01 19:12:48 in reply to 7 [link] [source]
Right. Thanks for the reminder. I know that, that's why I wrote
I know how to extract that value myself, if push comes to shove
And that integer is big-endian, to add to your description.