SQLite Forum

Is it possible to make -shm/-wal files permanent?
Login

Is it possible to make -shm/-wal files permanent?

(1) By anonymous on 2022-01-09 17:30:13 [link] [source]

I have a process that has only read only access to the database filesystem. It works fine and able to make select queries with a concurrent writer running on the same machine, but when the writer closes the database and delete those files a reader fails to continue. Correct me if I wrong, is this way of operation normal? If so, is it possible to make those files permanent so the reader will continue to read database even when writer is closed?

(2) By Richard Hipp (drh) on 2022-01-09 19:43:50 in reply to 1 [source]

SQLite version 3.22.0 (2018-01-22) and later should be able to read a read-only WAL-mode database that does not have -wal and -shm files present. What version of SQLite are you using that is giving trouble?

(3) By anonymous on 2022-01-09 20:00:48 in reply to 2 [link] [source]

I am using 3.34.1. It is a python binding, I am opening it in a read only mode "sqlite3.connect('file:...?mode=ro', check_same_thread=False, uri=True)" and it gives me "sqlite3.OperationalError: unable to open database file" when no such files present in the directory. This confirms with a manual or it was not updated yet

(5) By Richard Hipp (drh) on 2022-01-09 21:00:04 in reply to 3 [link] [source]

Does it work like you want if you leave off the "mode=ro" option?

(12) By anonymous on 2022-01-09 22:16:11 in reply to 5 [link] [source]

No it doesn't help

(8.1) By Keith Medcalf (kmedcalf) on 2022-01-09 23:36:37 edited from 8.0 in reply to 2 [link] [source]

WAL only, by the way. Does not apply to write-behind logging.

Are you thinking an immutable database (no one is updating the database while it is open, ever) and read-only (which does require tracking transaction state because the database file may be updated by another connection)?

That would be immutable=1, not mode=ro. I can see that a database should be able to be opened and read when immutable=1, but mode=ro would require the wal/shm to exist and would fail if they cannot be created and do not exist.

(4) By anonymous on 2022-01-09 20:51:29 in reply to 1 [link] [source]

It is possible to tell SQLite to not delete the WAL file, but to prevent other programs (such as the SQLite command shell) also doing so, you can set the sticky bit for the directory containing it. (That is what I did with sqlnetnews.)

(6) By Richard Hipp (drh) on 2022-01-09 21:01:13 in reply to 4 [link] [source]

The SQLITE_FCNTL_PERSIST_WAL option supports that. But it is not sticky and has to be done for each writer.

(7) By anonymous on 2022-01-09 21:03:57 in reply to 6 [link] [source]

The sticky bit that I mean is not a part of SQLite; it is the operating system (I am using Linux). If you set the owner to others than the users accessing it, then the file will not be deleted automatically, due to not having the permission to delete the file.

(14) By AlexJ (CompuRoot) on 2022-01-10 01:42:52 in reply to 7 [link] [source]

Why do you thing it should be done on SQLite level that can't predict upfront which underlying file system will be in use (even on Linux, it can be exFAT, FAT or even NTFS where no such things as sticky bit). IMO such protection should be offloaded out of SQLite responsibility

(10.1) By Keith Medcalf (kmedcalf) on 2022-01-09 23:37:20 edited from 10.0 in reply to 6 [link] [source]

and reader. Basically, every connection. Unless the database is immutable.

(9) By Keith Medcalf (kmedcalf) on 2022-01-09 21:13:53 in reply to 4 [link] [source]

While that might work, why would you not set the same filectrl in the CLI?

SQLite version 3.38.0 2022-01-08 23:33:02
Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.
sqlite> .filectrl
Available file-controls:
  .filectrl chunk_size SIZE
  .filectrl data_version
  .filectrl has_moved
  .filectrl lock_timeout MILLISEC
  .filectrl persist_wal [BOOLEAN]
  .filectrl psow [BOOLEAN]
  .filectrl reserve_bytes [N]
  .filectrl size_limit [LIMIT]
  .filectrl tempfilename

(11) By anonymous on 2022-01-09 21:33:20 in reply to 9 [link] [source]

Mainly to avoid having to do so every time.

(13) By anonymous on 2022-01-09 22:34:39 in reply to 4 [link] [source]

I thought to make directory not writable to the writer on a OS level but is it safe? Wouldn't it break things or slow down sqlite?