Hello, we recently discovered BEGIN CONCURRENT and started investigating its use for a new product (https://zerosync.dev).

To accomplish its work, our product needs to maintain materialized views of large, expensive queries. It's not tractable for us re-run the query each time it is needed. Also, what we really need is the differences in the view, not the view itself (because we will be sending those diffs to the client).

We have developed an approach to maintaining these views efficiently and getting their diffs on top of SQLite (loosely based on https://www.vldb.org/pvldb/vol16/p1601-budiu.pdf). Our approach requires a series of interactive write/read steps – for an incoming write transaction, we write a row to sqlite, do some queries, write another row, do some queries, etc. At the end of these series of steps we have the output that we send to the client and we commit.

Importantly, we cannot do all the writes up front - the correctness of the algorithm requires the write/read interleaving.

This all works, but we'd like to run our IVM on multiple threads to fully utilize server capacity. If we do so naively, we will have to synchronize each thread at the write points of the algorithm which would mitigate much of using multiple threads in the first place. And if we do the writes on each thread, they will take locks in SQLite and conflict.

BEGIN CONCURRENT give us an awesome out of this situation: we can open transactions in parallel, do our write/read pairs, then abort the transactions. Then we just have a single 'writer' transaction that actually flushes the persistent writes to the db at the end.

Thus, I'm very curious if there is any updated information on whether BEGIN CONCURRENT might become officially supported, or anything we can do to influence that.

Thank you!

(With my apologies for the delay.)

I'm very curious if there is any updated information on whether BEGIN CONCURRENT might become officially supported

It's not on any written-down roadmap, but almost nothing is in this project - features get added as they become relevant/interesting...

or anything we can do to influence that.

... or, in some cases, they're sponsored by a 3rd party.

I'll cheat and ask a different question.

For a user that ignores BEGIN CONCURRENT and wal2 how buggy do you expect the bedrock branch to be?

My point is, are these features gated a in a way that (for the most part) they can safely be ignored, or do they change the “core” in ways that are likely to introduce bugs for those that don't use them?

I want to provide a Wasm build a that includes both, and I'm wondering if I should have 2 binaries or just one (where users who value stability can safely ignore the features rather than bundle a different binary).

For a user that ignores BEGIN CONCURRENT and wal2 how buggy do you expect the bedrock branch to be?

i happen to know that there is at least one highly-active user of the bedrock branch. As for the wal2 and begin-concurrent branches... no clue.

... or do they change the “core” in ways that are likely to introduce bugs for those that don't use them?

i unfortunately cannot answer that beyond saying that i have no reason to believe that they introduce new bugs in features/subsystems which predate them. That unhelpfully vague, but anything more specific would be pure speculation on my part.

Thanks! I wasn't expecting any promises, as that's unrealistic. So this is really the best I could hope for.

Lets ask the question a little differently.

SQLite has very strong testing guarantees for its main branch.

Do these guarantees extend to other branches? These two branches?

Do these guarantees extend to other branches?

Not specifically, no - we'd be doing little more running tests if that were the case.

These two branches?

No clue - my involvement with those features is only as an idle observer.

I'm getting these warnings...

Those are resolved now in the bedrock and (where relevant) begin-concurrent branches. Thank you for the report.

Hi Stephan! Any updates on the "BEGIN CONCURRENT" feature?

Any updates on the "BEGIN CONCURRENT" feature?

My initial response still applies to that one:

It's not on any written-down roadmap.

That branch, like several others, continues to get actively updated from trunk, but there are currently no concrete plans to merge them to the trunk.

Just as a follow-up, if you're using Go, I make the bedrock branch easy to test.