SQLite Forum

Error "database disk image is malformed" when loading an appendvfs db
Login

Error "database disk image is malformed" when loading an appendvfs db

(1) By anonymous on 2021-03-15 18:28:45 [link] [source]

Hello!

I'm trying to use the appendvfs to attach a database to a statically linked executable.

I appended the contents of appendvfs.c to sqlite3.c (and removed it from shell.c) in the amalgamation (v3.35.0) (and tweaked sqlite3.h to have the sqlite3_appendvfs_init prototype).

My code calls sqlite3_appendvfs_init(0,0,0) and feeds "apndvfs" as the zVfs parameter to sqlite3_open_v2.

Everything builds fine (Ubuntu 18.04, gcc 8.4.0) and the database is is correctly generated at the end of the executable. I see the "Start-Of-SQLite3-NNNNNNNN" trailer at the end, and the "SQLite format 3" db header at the proper offset.

If I chop off the first NNNNNNNN bytes of the resulting file, I get the database, as expected, and can load it in the sqlite3 shell.

But when I try to load the executable+database via apndvfs, I get the "database disk image is malformed" error: both from my application and from the sqlite3 shell. (When using the shell, I tried both ".open --append" and "sqlite3 -append")

I suspect I'm missing something fundamental and obvious, but I can't figure out what it is.

Thanks, Mark.

(2.1) By Larry Brasfield (larrybr) on 2021-03-15 19:21:38 edited from 2.0 in reply to 1 [source]

(Edited to add paragraph past "BTW, ...")

As far as you have described what you're doing, it should work. And I can do what you're doing, (as I interpret that), on my 64-bit Ubuntu 20.04 system with the latest release. So I suspect that either you are doing something that differs from what I think you've described, or something else is significantly different. For the latter, the possibilities are extremely limited since the appendvfs cares little as to what it is (unless it is a SQLite DB.)

If you are able to publish your executable somewhere, in its non-appended-upon form, and point me to it, I will see if there is something about it that interferes with appendvfs. Or, tell me what length it is and I'll see whether, for reasons unknown, that presents a problem for appendvfs.

BTW, here is another way to look at your appended DB that should work: attach 'file:some-elf?mode=rw&vfs=apndvfs' as X; .

One possible explanation for your difficulty is that you are using a version of the SQLite shell or appendvfs.c which predates the v3.35.0 release on March 12. That code has been recently modified and better tested to correct one or two subtle bugs involving creation scenarios and memory-mapped files. It is easy to think "the SQLite shell" when that may refer to either some system-installed, older version or a more recent version one has recently built or downloaded.

(3) By anonymous on 2021-03-15 20:46:24 in reply to 2.1 [link] [source]

Hi, Larry,

Thanks for the reply and the ideas.

I was using v3.35.0, built expressly for the purpose - not whatever Ubuntu thinks I should have. :) I also just grabbed v3.35.1 and have the same issue. (And the shell is the shell I just built - again, not the platform's default.)

The executable itself seems irrelevant: it's only 940K of minimal code to try to open itself and run a query, just to prove it's working. (Or not, as the case may be.)

The DB itself is 11MB - the total is under the 953MB APND_MAX_SIZE limit.

I copied /bin/cat (35K) to "./testcat". I opened testcat via the shell (with .open --append); created some tables; exited; and reopened it in the shell and all was well. (And testcat still runs properly.)

Then I fed testcat to my code that opens the database via apndvfs and it was able to query the tables.

My full blown database generation code seems to produce "malformed" database disk images. So, I wrote a minimal test:

int main (void)
{
    sqlite3_appendvfs_init(0, 0, 0);
    int flags = SQLITE_OPEN_READWRITE|SQLITE_OPEN_CREATE;
    sqlite3 *db;
    if (sqlite3_open_v2("testcat", &db, flags, "apndvfs") != SQLITE_OK) {
        printf("%s\n", sqlite3_errmsg(db));
    } else {
        char *sql = "CREATE TABLE foo (id INTEGER PRIMARY KEY, name TEXT);";
        char *error = 0;
        if (sqlite3_exec(db, sql, 0, 0, &error) != SQLITE_OK) {
            printf("%s", error);
            sqlite3_free(error);
        }
    }
    sqlite3_close(db);
    exit(0);
}

Which produces a good result. I'll have to keep adding bits until it breaks and post some results.

Thanks,

Mark

(4) By anonymous on 2021-03-15 22:04:10 in reply to 3 [link] [source]

So this is weird.

It kept working until I added a couple tables containing UNIQUE BLOB columns.

You can see breakage from the shell:

$ cp /bin/cat testcat
$ ./sqlite3 -append testcat
SQLite version 3.35.1 2021-03-15 16:53:57
Enter ".help" for usage hints.
sqlite> CREATE TABLE one (name TEXT PRIMARY KEY);
sqlite> CREATE TABLE two  (id INTEGER PRIMARY KEY,uuid BLOB UNIQUE, name TEXT, size INTEGER,mtime INTEGER, UNIQUE (uuid, name) );
sqlite> CREATE TABLE three (id INTEGER PRIMARY KEY,uuid BLOB UNIQUE, compressed BLOB, size INTEGER );
sqlite> .dump
PRAGMA foreign_keys=OFF;
BEGIN TRANSACTION;
CREATE TABLE one (name TEXT PRIMARY KEY);
CREATE TABLE two  (id INTEGER PRIMARY KEY,uuid BLOB UNIQUE, name TEXT, size INTEGER,mtime INTEGER, UNIQUE (uuid, name) );
CREATE TABLE three (id INTEGER PRIMARY KEY,uuid BLOB UNIQUE, compressed BLOB, size INTEGER );
COMMIT;
sqlite> .q
$ ./sqlite3 -append testcat
SQLite version 3.35.1 2021-03-15 16:53:57
Enter ".help" for usage hints.
sqlite> .schema
sqlite> .dump
PRAGMA foreign_keys=OFF;
BEGIN TRANSACTION;
COMMIT;
sqlite> .q

If you look at a hex dump of testcat you can see that it looks ok.And if you slice off the "non-database" prefix, the result is also ok:

$ tail -c -32 testcat | hd
00000000  00 00 00 00 00 00 00 53  74 61 72 74 2d 4f 66 2d  |.......Start-Of-|
00000010  53 51 4c 69 74 65 33 2d  00 00 00 00 00 00 90 00  |SQLite3-........|
00000020
$ dd if=testcat of=test.db bs=4096 skip=9
8+1 records in
8+1 records out
32793 bytes (33 kB, 32 KiB) copied, 0.000355294 s, 92.3 MB/s
$ ./sqlite3 test.db
SQLite version 3.35.1 2021-03-15 16:53:57
Enter ".help" for usage hints.
sqlite> .dump
PRAGMA foreign_keys=OFF;
BEGIN TRANSACTION;
CREATE TABLE one (name TEXT PRIMARY KEY);
CREATE TABLE two  (id INTEGER PRIMARY KEY,uuid BLOB UNIQUE, name TEXT, size INTEGER,mtime INTEGER, UNIQUE (uuid, name) );
CREATE TABLE three (id INTEGER PRIMARY KEY,uuid BLOB UNIQUE, compressed BLOB, size INTEGER );
COMMIT;
sqlite> .q

Adding an extra table pushes it over the edge:

$ cp /bin/cat testcat
$ ./sqlite3 -append testcat
SQLite version 3.35.1 2021-03-15 16:53:57
Enter ".help" for usage hints.
sqlite> CREATE TABLE one (name TEXT PRIMARY KEY);
sqlite> CREATE TABLE two  (id INTEGER PRIMARY KEY,uuid BLOB UNIQUE, name TEXT, size INTEGER,mtime INTEGER, UNIQUE (uuid, name) );
sqlite> CREATE TABLE three (id INTEGER PRIMARY KEY,uuid BLOB UNIQUE, compressed BLOB, size INTEGER );
sqlite> CREATE TABLE four (id1  INTEGER, id2 INTEGER, FOREIGN KEY (id1) REFERENCES two(id), FOREIGN KEY (id2) REFERENCES three(id), PRIMARY KEY (id1, id1));
sqlite> .dump
PRAGMA foreign_keys=OFF;
BEGIN TRANSACTION;
CREATE TABLE one (name TEXT PRIMARY KEY);
CREATE TABLE two  (id INTEGER PRIMARY KEY,uuid BLOB UNIQUE, name TEXT, size INTEGER,mtime INTEGER, UNIQUE (uuid, name) );
CREATE TABLE three (id INTEGER PRIMARY KEY,uuid BLOB UNIQUE, compressed BLOB, size INTEGER );
CREATE TABLE four (id1  INTEGER, id2 INTEGER, FOREIGN KEY (id1) REFERENCES two(id), FOREIGN KEY (id2) REFERENCES three(id), PRIMARY KEY (id1, id1));
COMMIT;
sqlite> .q
./sqlite3 -append testcat
SQLite version 3.35.1 2021-03-15 16:53:57
Enter ".help" for usage hints.
sqlite> .database
Error: database disk image is malformed
sqlite> 

(5) By Larry Brasfield (larrybr) on 2021-03-15 22:14:36 in reply to 4 [link] [source]

That's a nice repro case. I will spend some time with it and get back Tuesday morning. (Or earlier if it does not reproduce.)

(6) By Larry Brasfield (larrybr) on 2021-03-16 08:11:58 in reply to 4 [link] [source]

Mark, your problem case was very helpful in chasing down a bug which managed to escape tests designed to catch issues with growing DBs in appendvfs files. Branch appendvfs_fix cures the problem you reported. I don't know when it will get into a release, but it can obtained from the repo. The only substantive changes are in appendvfs.c, which is embedded in the shell.c which comes with the amalgamation.

Thanks for your report and clean repro steps.

(7) By anonymous on 2021-03-16 13:52:36 in reply to 6 [link] [source]

Hi, Larry,

Glad I could be of help - and thank you for the quick fix!

I grabbed the new appendvfs.c and rolled it into my build - and everything is now working.

Thanks again,

Mark.