SQLite User Forum

Obtain information from DB header via PRAGMAs on truncated DBs
Login

Obtain information from DB header via PRAGMAs on truncated DBs

(1) By ddevienne on 2023-09-01 09:33:58 [link] [source]

I have lots of (potentially large) DBs in TAR files.

And I'd like to check the user_version of those DBs, which is part of the 100-byte header,
without having to untar potentially GBs of data.

So is there a way to make SQLite accept a truncated DB file,
for the limited purpose of issuing pragmas against the header?

Truncated to 100-bytes? Truncated to the max-page-size of 64KB?
That way the 1st page used for sqlite_master is always there,
and the full schema can be read (assuming small enough to fit in 64KB).

I know how to extract that value myself, if push comes to shove,
but if I can use the SQLite API for that somehow, that's be better.

I suspect it's not possible, but I'm hoping I'm wrong, thus my asking.

And maybe we could/should have a new public API to parse that header,
so that it's not re-invented all over the place (including my code)?

(2) By anonymous on 2023-09-01 10:45:32 in reply to 1 [link] [source]

Two quick experiments later...

CLI input:

.version
.mode table
.headers on
pragma application_id;
pragma data_version;
pragma encoding;
pragma freelist_count;
pragma page_count;
pragma page_size;
pragma schema_version;
pragma user_version;

Output from a database truncated to 64K:

SQLite 3.41.2 2023-03-22 11:56:21 0d1fc92f94cb6b76bffe3ec34d69cffde2924203304e8ffc4155597af0c191da
zlib version 1.2.11
clang-12.0.0
Runtime error near line 4: database disk image is malformed (11)
Runtime error near line 5: database disk image is malformed (11)
Parse error near line 6: database disk image is malformed (11)
Runtime error near line 7: database disk image is malformed (11)
Parse error near line 8: database disk image is malformed (11)
+-----------+
| page_size |
+-----------+
| 4096      |
+-----------+
Runtime error near line 10: database disk image is malformed (11)
Runtime error near line 11: database disk image is malformed (11)

Output from a database truncated to 64K and then extended to the original size:

SQLite 3.41.2 2023-03-22 11:56:21 0d1fc92f94cb6b76bffe3ec34d69cffde2924203304e8ffc4155597af0c191da
zlib version 1.2.11
clang-12.0.0
+----------------+
| application_id |
+----------------+
| 0              |
+----------------+
+--------------+
| data_version |
+--------------+
| 1            |
+--------------+
+----------+
| encoding |
+----------+
| UTF-8    |
+----------+
+----------------+
| freelist_count |
+----------------+
| 10             |
+----------------+
+------------+
| page_count |
+------------+
| 689        |
+------------+
+-----------+
| page_size |
+-----------+
| 4096      |
+-----------+
+----------------+
| schema_version |
+----------------+
| 6              |
+----------------+
+--------------+
| user_version |
+--------------+
| 0            |
+--------------+

[Insert the usual warnings about relying on undocumented behaviour here.]

(3) By Dan Kennedy (dan) on 2023-09-01 11:50:25 in reply to 1 [source]

I think if you open a db truncated to 64KiB, then use sqlite3_db_config() to set both SQLITE_DBCONFIG_TRUNCATED_SCHEMA and SQLITE_DBCONFIG_WRITABLE_SCHEMA, then SQLite will let you run PRAGMA statements to query header fields even though the file has been truncated.

Or, with the shell tool, add the "-unsafe-testing" option to the command line, then execute a "PRAGMA writable_schema=1" command. You should then be able to query a truncated db.

Dan.

(4.1) By ddevienne on 2023-09-01 15:59:57 edited from 4.0 in reply to 3 [link] [source]

Thanks Dan!

My (corporate) version of SQLite does NOT have SQLITE_DBCONFIG_TRUNCATED_SCHEMA (yet).
But SQLITE_DBCONFIG_WRITABLE_SCHEMA is enough in my case; apparently our schemas fit in 64KB.

Took a little while to reply, since I tested out the real thing,
i.e. partial untar (subset of files, truncated to 64KB) and running pragma user_version on those.

PS: I confirm SQLITE_CORRUPT is reported _step()'ing the pragma without it.

(5) By ddevienne on 2023-09-01 16:07:18 in reply to 3 [link] [source]

Actually Dan, SQLITE_DBCONFIG_TRUNCATED_SCHEMA is not even listed in the doc. Is it new and unreleased yet? Or somehow private?

(6) By Dan Kennedy (dan) on 2023-09-01 16:26:09 in reply to 5 [link] [source]

Sorry - should be "SQLITE_DBCONFIG_TRUSTED_SCHEMA".

Dan.

(7) By Aask (AAsk1902) on 2023-09-01 17:46:13 in reply to 1 [link] [source]

And I'd like to check the user_version of those DBs, which is part of the 100-byte header,without having to untar potentially GBs of data.

The user_version is an unsigned 4 bytes (32-bits) integer at bytes 60-63 inclusive.

I read (accessing the file/database in exclusive read-only mode) 32 bits starting at offset 60 (using C#) and converted to integer: the result is what pragma user_version returns.

(8) By ddevienne on 2023-09-01 19:12:48 in reply to 7 [link] [source]

Right. Thanks for the reminder. I know that, that's why I wrote

I know how to extract that value myself, if push comes to shove

And that integer is big-endian, to add to your description.