SQLite Forum

Status of wal2 branch
Login

Status of wal2 branch

(1) By anonymous on 2020-07-07 13:07:22 [link] [source]

Is wal2 considered stable enough to be used in production scenarios?

It looks like the branch is kept well up-to-date with the main branch. Are there any plans to make it into an official feature?

Motivation: our database has an uninterrupted stream of small write operations with relatively strict latency constraints. We want to run in WAL mode, but WAL checkpoints exceed these latency constraints. When WAL checkpoints are moved to a background thread, as the documentation suggests, we run into the ever-growing-wal-file problem.

The wal2 branch solves exactly this problem (we can confirm it works). Other solutions like moving all writes to a background thread or buffering the incoming writes mean we lose a lot of the database's nice features.

Thanks.

(2) By Dan Kennedy (dan) on 2020-07-07 19:14:18 in reply to 1 [link] [source]

Is wal2 considered stable enough to be used in production scenarios?

I don't think so. It doesn't cause any obvious failures in SQLite's test suite. But there are very few new tests - no real effort has been made to find problems. To my knowledge nobody has used it in an actual application, only for performance testing the basic concept of using two wal files.

It looks like the branch is kept well up-to-date with the main branch. Are there any plans to make it into an official feature?

No such plans at this point.

(3) By phpstatic on 2020-11-18 13:27:08 in reply to 2 [link] [source]

Hi Dan,

I like to report a bug find about wal2.

call sqlite3_database_file_object from xOpen get this:

* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x48)
    frame #0: 0x0000000100056459 tests_db_bench.exe`sqlite3_database_file_object(zName="/private/tmp/test.db-wal2") at sqlite3c.h:56776:18
   56773	    zName--;
   56774	  }
   56775	  pPager = *(Pager**)(zName - 4 - sizeof(Pager*));
-> 56776	  return pPager->fd;
   56777	}
   56778	
   56779	
Target 0: (tests_db_bench.exe) stopped.

(4) By Dan Kennedy (dan) on 2020-11-19 16:10:02 in reply to 3 [source]

Thanks. Should now be fixed here:

https://sqlite.org/src/info/c2426ae8a80d61e1

Dan.

(5) By phpstatic on 2020-11-20 02:13:32 in reply to 4 [link] [source]

You are most welcome.

A small warning:

warning: assigning to 'char *' from 'const char *' discards qualifiers
      [-Wincompatible-pointer-types-discards-qualifiers]
  pRet->zWalName2 = &zWalName[sqlite3Strlen30(zWalName)+1];

If I create a new database, the sqlite will try access journal first, then create journal file. After I set journal_mode = WAL2, sqlite delete + journal and create 2 WAL file.

Is there a way to let sqlite skip create journal file when I intend to use WAL ? (a flags pass to sqlite3_open_v2 to avoid call "pargama journal_mode = WAL" late)

The is VFS only SQLITE_OPEN_WAL, maybe reuse it to do the work(only for create new database).

(6) By Dan Kennedy (dan) on 2020-11-20 11:24:40 in reply to 5 [link] [source]

Thanks again. Now fixed here:

https://sqlite.org/src/info/4f5481bf291c39e2

Is there a way to let sqlite skip create journal file when I intend to use WAL ? (a flags pass to sqlite3_open_v2 to avoid call "pargama journal_mode = WAL" late)

No such flag unfortunately.

Creating and deleting the journal file would be necessary when initializing a new wal database anyhow. Otherwise you would either create race conditions with other processes trying to open the db simultaneously, or else risk leaving a corrupt file on the disk if the application or system failed at an inopportune moment while opening the file.

Dan.

(7) By Joshua Wise (wisej12) on 2021-05-17 20:52:09 in reply to 2 [link] [source]

I'd like to add my name to the list of people who are interested in this feature becoming official. It would solve many of the problems of SQLite I struggle to work around on a daily basis, when working with real-time systems.

(8) By Tyke (tyke007) on 2021-08-09 06:10:53 in reply to 2 [link] [source]

I like to know is there any plan to provide a roadmap for wal2 branch?

From the code I see, it is a runtime option to enable wal2.

Merge it into main branch and a compile time option to enable it(default disabled), will made more people use it, and more people to provide patch to improve it.