"disk is full" error only once per connection?
(1) By Sven (sven.milker) on 2020-05-07 12:09:12 [link] [source]
Hello, after updating SQLite from 3.29.0 to 3.31.1 we noticed that our "disk is full" handling is not working reliantly any more. The error is only thrown once and then never again. Everything seems to work, but when looking at the database, nothing is persisted. Short version: After the upgrade the "disk is full" error is only thrown once per connection. That is, having an app whose connection is never closed the error occurs only the first time and after that SQLite throws no error even if the data could not be persisted. Long version: I tried to get proper reproduction steps, but it seems quite tricky. I used DB Browser for SQLite for that, but in our solution we are using sqlite-net which gets similar results. Using sqlite 3.29.0, the following behaviour occurs: If the disk is completey full, there is an error on the query immediately, but if there is ~8KB free space, the error occurs when trying to persist the changes. The error in this second case is 'database or disk full (RELEASE "RESTOREPOINT";)'. Trying to persist it again, the next error is 'no such savepoint: RESTOREPOINT (RELEASE "RESTOREPOINT";)'. The error type is not important, but there is at least an error. When simply replacing the sqlite3.dll in the installation folder of DB Browser for SQLIte to upgrade to version 3.31.1, the first message still occurs, but there is no second error message. So it seems that the data have been written which is not the case. The same behaviour occurs when using sqlite-net: On the first Release of the transaction there is an error, but doing so again no error is thrown by the sqlite3.dll, it returns Done. This occurs only per connection, meaning opening a new connection throws the error once again. So as a workaround reopening the connection is working. Is this the intended behaviour?
"disk is full" is not a recoverable error, because SQLite can no longer keep track of anything by writing to a journal file. You can't do anything using the SQLite API to recover from it so the SQLite API will no longer work in a dependable way. This has always been the case. Quitting once you get a "disk is full" error will lead to a database which can be recovered (i.e. not corrupt) once you have freed up some space and restarted the program.
If your application has to work under situations where a SQLite database volume may fill up, guard against it outside SQLite. For instance, you might run a routine once a minute (hour ?) which checks to see whether there is less than a megabyte free on the drive. If that happens, terminate the app or delete old files accordingly, then check again to see whether it's safe to continue. Do it to prevent SQLite getting a "disk is full" error.
(3) By Sven (sven.milker) on 2020-05-08 12:17:31 in reply to 2 [link] [source]
The problem is that I cannot guard against a full disk because it could change any second. Even guarding every request could not prevent it certainly.
And about the recoverability: It is totally fine for me if nothing ca be recovered as long the database won't be corrupt. If the disk is full I want to stop immediately and notify the user. This works fine, but only once! The second time the error is not thrown any more and this is what I don't understand.
I think I will just re-open the connection if the error occurs. This way the disk is full error is thrown again.
Thanks for your answer!
The API expects you to inform the user and then quit until the user has fixed the problem. Obviously, quitting closes all connections so to continue, the user would free up some space, then restart the app.
I'm puzzled about how your application can be useful if the disk is still full. If you do use the procedure you described above, and are not quitting the app after the warning, then when you get the "disk is full" error …
- close the connections
- Inform the user
- idle the app in a loop that checks free space until the user has freed up enough space for the app to continue
- reopen the connections
idle the app in a loop that checks free space until the user has freed up enough space for the app to continue
Noting that free space is not, on all OS/filesystem combinations, and indication that the user can actually use that space. If FS quotas are in effect, a typical space-free check will not indicate that. Similarly, some OS/FS combinations allow the superuser to reserve, at filesystem-formatting time, some percentage of the space for superuser-only use. Thus apps may start failing due to filesystem-is-full errors even though 10% of the filesystem is ostensibly free.