SQLite Forum

Database disk image is malformed

Database disk image is malformed

(1) By ayagmur75 on 2020-10-20 06:48:07 [link]

I have been working on an embedded system. My database size is almost 3GB. I have encountered with "Database disk image is malformed" error. After this, I copied database to windows operating system and executed the same query without any problem. After that I execute analyze command with no error, but pragma integrity_check with following errors; 
On tree page 51181 cell 0: 232 of 244 pages missing from overflow list starting at 51183...
...similar lines...
Page 51195 is never used
Page 51196 is never used
Page 51197 is never used
...similar lines...

I want to ask some questions on this situation. 

1. Why do I get "Database disk image is malformed" error?
2. Why do I get "232 of 244 pages missing from overflow list starting at 51183" error?
3. Why do I get "Page 51195 is never used" error?

If you can give some insights, I try to understand what kind of an error that I have made cause such a situation.

Thanks for advance,
Best Regards,

(2) By Gunter Hick (gunter_hick) on 2020-10-20 08:35:36 in reply to 1

See https://sqlite.org/howtocorrupt.html for an overview of the most common causes of database file corruption.

(3) By ayagmur75 on 2020-10-20 14:06:16 in reply to 2 [link]

Thanks for the reply.
I have found that the problem is related with blob insertion.

What kind of character sequence could cause such an error? Can you give me some insights?

(4.3) By Simon Slavin (slavin) on 2020-10-20 14:44:53 edited from 4.2 in reply to 3 [link]

There should be no way to create that fault just through coming up with an unexpected sequence of characters.  In fact BLOBs don't hold characters at all.  They are sequences of octets.  The individual octets are not understood as characters, just binary patterns.

Run <code>pragma integrity_check</code> on the original database, not the one copied to Windows.  Make sure that the corruption is in the original.  If it is, then stop using that database file, even though it is not returning an error when you try to use it in your code.

Are you sure you are using the API calls correctly ?  Are you sure you are handling memory correctly ?

(5) By Gunter Hick (gunter_hick) on 2020-10-20 14:42:30 in reply to 3 [link]

The error has nothing to do with the data you are inserting. It has to do with what you are doing to the database.

You need to say a lot more about the "embedded system" you are running on, which SQLite release you are using, etc. before anyone can even try to help with your problem.

(7) By Jeff Hemphil (jhemphill) on 2021-09-03 16:01:31 in reply to 3 [link]

I encountered a similar problem in HHVM.

The root cause of the corruption was that unrelated code in HHVM was sometimes writing to a closed file descriptor. When you write to a closed file descriptor, you’re playing Russian Roulette with the files you currently have open, and there’s a chance you’ll write unspecified data to an unspecified place in your SQLite DB, instead of writing to the file you wanted to write to.

I was able to debug this issue by writing a C program that ran `PRAGMA integrity_check` on my DB. I then compiled it with all optimizations off, and stepped through it with GDB. GDB showed me that an int had a value of 115, which was impossible if only SQLite were writing to the DB file. Reading this corrupted area as a sequence of bytes, I saw that someone had written the C string `"stop"` to the DB, causing the corruption. I then searched for `"stop"` to find the code that caused the issue. [Here was my fix](https://github.com/facebook/hhvm/commit/46721ee0ad8d7cfc94f36e20781a0e4215684ed8#diff-e30509bce63de23ea7d45a2091be9c4039bf4defc03c70b1576e48114ccf8eae), which didn’t touch SQLite code at all and instead ensured that, in this code, writes to a file descriptor only happen on the thread that will close the file descriptor.

I haven’t done a public-facing writeup on this, but [this public-facing writeup shows how to solve a harder version of the same bug.](https://engineering.fb.com/2014/08/12/ios/debugging-file-corruption-on-ios/)

(8) By ddevienne on 2021-09-03 16:20:28 in reply to 7 [link]

Thanks for sharing. Very interesting.

(9) By J.M. Aranda (JMAranda) on 2021-09-03 19:00:01 in reply to 7 [link]

Cherchez la thread!

(6) By anonymous on 2021-09-02 14:52:08 in reply to 1 [link]

Hi, vacuum comand works for me,