SQLite Forum

sqlite-wasm: does OPFS have an automatic data clearing mechanism or bug?
Login

sqlite-wasm: does OPFS have an automatic data clearing mechanism or bug?

(1) By Mayne (mayneyao) on 2023-10-28 18:02:08 [source]

Hello everyone,

This is my first time posting on the SQLite forum. I would like to express my gratitude for the work done on sqlite-wasm. It is an exciting feature. I am using sqlite-wasm to build a web app similar to Notion that runs entirely in the browser.

sqlite-wasm uses OPFS for data storage. In the past few months, I have encountered two instances where the data was cleared, but only the OPFS data was affected, and the data in LocalStorage remained intact. I have done some research but haven't found any useful information on this relatively new technology.

Unfortunately, I cannot provide a reproduction path as this issue seems to occur approximately once a month. I am wondering if anyone else has encountered a similar situation.

My browser, Edge version 118.0.2088.69

Thank you all in advance for any insights or assistance you can provide.

(2) By Roy Hashimoto (rhashimoto) on 2023-10-28 18:22:39 in reply to 1 [link] [source]

Are you requesting and did you receive user permission to use persistent storage?

(6) By Mayne (mayneyao) on 2023-10-28 23:37:10 in reply to 2 [link] [source]

OMG,Thank you very much! I think that should be the reason. I tried executing code in the console

if (navigator.storage && navigator.storage.persist) {
  navigator.storage.persist().then((persistent) => {
    if (persistent) {
      console.log("Storage will not be cleared except by explicit user action");
    } else {
      console.log("Storage may be cleared by the UA under storage pressure.");
    }
  });
}

and it returned "Storage may be cleared by the UA under storage pressure."

I looked back and checked the documentation, and found the relevant instructions. This is a point that is easy to overlook.

I will try adding the permission request and then observe for a period of time.

(7) By Mayne (mayneyao) on 2023-10-29 00:24:50 in reply to 2 [link] [source]

I tried it and it was very interesting. I installed my app as a PWA, but it didn't receive much attention from Edge. It automatically rejected the request for data persistence. Only when I granted notification permissions did Edge also grant persistence storage permissions. Therefore, currently, developers can only obtain data persistence permissions by requesting notification permissions.This is very strange.

(9) By Mayne (mayneyao) on 2023-11-09 11:36:19 in reply to 7 [link] [source]

I obtained persistent permission, but today my data is lost again. I don't understand.

(10) By Stephan Beal (stephan) on 2023-11-09 11:42:18 in reply to 9 [link] [source]

I obtained persistent permission, but today my data is lost again. I don't understand.

Unfortunately, neither do we. There is nothing in the library which will outright delete your databases. At worst, the library would leave a corrupt db in place and require you to delete it manually. Some as-yet-unidentified element in your environment, possibly third-party software or a browser quirk, is doing this.

(14) By Mayne (mayneyao) on 2023-11-29 16:01:37 in reply to 10 [link] [source]

I regularly use Windows and Edge browser, and it seems that there is a lack of consistency between Windows' storage cleanup mechanisms and Edge, leading to the removal of data from OPFS regardless of whether persistent permissions were obtained or not. I have reported this bug to the Edge team, but I have not received a response.

(16) By anonymous on 2023-12-20 14:03:00 in reply to 14 [link] [source]

Do you have a link or bug reference?

(17) By Simon Slavin (slavin) on 2023-12-21 12:19:42 in reply to 7 [link] [source]

While you may be developing specifically for Edge, JavaScript and its supporting objects can be huge and complicated. It is definitely not exhaustively documented. It's useful to test how a different browser development team thought things should work.

I recommend you try your code also in another browser that supports OPFS. It may or may not get identical errors/warnings, and this may inform you whether you definitely have buggy code, or whether they may be a problem with Edge itself.

(18) By Stephan Beal (stephan) on 2023-12-21 12:48:38 in reply to 17 [link] [source]

... and this may inform you whether you definitely have buggy code, or whether they may be a problem with Edge itself.

FWIW: in this particular case there's no bug a user could write with the sqlite JS API which would cause a database to disappear. sqlite does not expose any filesystem-level APIs (for any platforms, which is why we also don't for OPFS), so there's no way a client could accidentally delete a database using this API. They'd have to go out of their way to explicitly use the OPFS-specific APIs to do that, in which case they wouldn't (one hopes!) be surprised that it's happening.

My suspicion (with no way to prove it) is still a 3rd-party tool like a virus scanner or a "cleanup" app.

(3) By Randy (randyl) on 2023-10-28 21:14:49 in reply to 1 [link] [source]

If it helps, I've had some users report their data in OPFS disappeared, and it was caused by the CCleaner program. Unchecking the "Internet Cache" boxes within CCleaner prevents it from happening again.

This has led me to discover that CCleaner is just one of a whole class of programs that are designed to optimize PCs by doing such things as deleting temporary/cached files. Other examples are BleachBit, AVG PC Tune Up, and many others. I don't know if they all have the same problem.

(4) By Tim Streater (Clothears) on 2023-10-28 21:43:42 in reply to 3 [link] [source]

Do these programs just make such deletions and then not tell anyone (by which I mean other system components)? Or are there no mechanisms to make such notifications?

(8) By Randy (randyl) on 2023-10-29 00:55:09 in reply to 4 [link] [source]

I don't know for sure. I haven't used any of these programs myself.

(5) By Mayne (mayneyao) on 2023-10-28 23:04:32 in reply to 3 [link] [source]

It's probably not the reason, because I don't have CCleaner installed on my computer. Thank you for the reminder, I will remember this unexpected situation and take measures to avoid it in the future.

(11) By Simon Slavin (slavin) on 2023-11-09 16:43:53 in reply to 5 [link] [source]

Do you have any anti-virus or anti-malware program installed ? Many of them do the same thing: clear up files which are not currently in use. And many of them don't understand that a browser can close a file it created but still need it later.

(15) By Mayne (mayneyao) on 2023-11-29 16:12:30 in reply to 11 [link] [source]

I use Windows 11, which has a feature called Storage Sense that automatically cleans up space. I suspect it caused the issue, so now I've turned it off.

(12) By charris on 2023-11-11 12:30:20 in reply to 1 [link] [source]

subscribe (if someone is aware of a way to "subscribe" to a thread without actually posting, please share)

(13) By Stephan Beal (stephan) on 2023-11-11 14:27:30 in reply to 12 [link] [source]

subscribe (if someone is aware of a way to "subscribe" to a thread without actually posting, please share)

There's not one. The server side of the forum stores only the absolute minimum amount of per-subscriber state needed to do its job, which rules out nice-to-haves such as per-thread subscriptions and keeping track of which posts you've seen or not seen.

(19) By Randy (randyl) on 2024-02-25 19:45:36 in reply to 1 [link] [source]

FWIW, one user of my Chrome extension, who also uses MS Edge, reported last month they believe WinToys was the cause of their disappearing data. However, they did not specify which feature of WinToys might have caused the problem.

I tried installing WinToys and using it to clean up junk files for MS Edge, but could not recreate the disappearing OPFS data.