SQLite User Forum

problem implementing multi-statement transactions
Login

problem implementing multi-statement transactions

(1) By Greg Wilson (gvwilson) on 2024-01-27 12:07:28 [source]

Cross-posted from https://stackoverflow.com/questions/77888248/how-to-implement-rollback-of-multi-statement-transaction-in-sqlite

Using sqlite3 3.43.2 on macOS, I feed the following into an otherwise-empty in-memory database:

.mode markdown
.headers on

create table job(
    name text not null,
    billable real not null,
    check(billable > 0.0) on conflict abort
);

begin transaction;
insert into job values ("calibrate", 1.5);
insert into job values ("reset", -0.5);
insert into job values ("clean", 0.5);
commit;

select * from job;

Since the three insert statements are inside a single transaction, I expect to get nothing from the final select because there is a check violation in the second insert, which I initially expected would cause the entire transaction to tail. Instead, the other two insert statements have executed:

Runtime error near line 12: CHECK constraint failed: billable > 0.0 (19)
|   name    | billable |
|-----------|----------|
| calibrate | 1.5      |
| clean     | 0.5      |
  1. Removing on conflict abort from the table definition does not change this behavior.
  2. Replacing on conflict abort with on conflict rollback in the table definition does not change this behavior.

After a bit of reading, I believe the behavior may be caused by auto-commit-per-statement, which is not the behavior I want. (In my full example, I want to insert records into two different tables, and be left with either both inserts successful or neither.)

Let's move the error handling to the insert statements. In a fresh in-memory database:

create table job(
    name text not null,
    billable real not null,
    check(billable > 0.0)
);

begin transaction;
insert or rollback into job values ("calibrate", 1.5);
insert or rollback into job values ("reset", -0.5);
insert or rollback into job values ("clean", 0.5);
commit;

select * from job;

The output is:

Runtime error near line 12: CHECK constraint failed: billable > 0.0 (19)
Runtime error near line 14: cannot commit - no transaction is active
| name  | billable |
|-------|----------|
| clean | 0.5      |

which shows that SQLite carried on and executed the third statement inside the transaction (which inserted clean) after the error in the second statement in that transaction. Again, I expected that when the error occurred in the second statement, the entire transaction would abort, i.e., SQLite wouldn't even try the third insert. (My mental model is exceptions being raised and caught in programming languages.)

Finally, in yet another fresh database:

insert or rollback into job values ("calibrate", 1.5);
insert or rollback into job values ("reset", -0.5);
insert or rollback into job values ("clean", 0.5);

produces:

Runtime error near line 11: CHECK constraint failed: billable > 0.0 (19)
|   name    | billable |
|-----------|----------|
| calibrate | 1.5      |
| clean     | 0.5      |

which is what I now expect, but not what I want.

Is there a way to create a true multi-statement transaction in SQLite? If so, what is the syntax? I want something like:

-- this is not valid SQLite syntax
begin transaction;
insert into job values ("calibrate", 1.5);
insert into job values ("reset", -0.5);
insert into job values ("clean", 0.5);
commit or rollback;

but as noted, that doesn't parse. The desired effect is that if any statement inside the transaction fails, the entire transaction is rolled back.

Alternatively: is there a detailed tutorial online somewhere about transactions, commit, and rollback in SQLite? Everything I have found over the last two days either (a) doesn't actually work when tested, (b) explains the very basics of rolling back a single commit without going further, or (c) both of the above. Thanks in advance.

(2) By Adrian Ho (lexfiend) on 2024-01-27 12:40:24 in reply to 1 [link] [source]

The desired effect is that if any statement inside the transaction fails, the entire transaction is rolled back.

Then you should say that in your CREATE TABLE (I've also corrected your quote usage, as using double-quotes for literal strings will get you an error in more recent SQLite versions):

create table job(
    name text not null,
    billable real not null,
    check(billable > 0.0) on conflict rollback
);

begin transaction;
insert into job values ('calibrate', 1.5);
insert into job values ('reset', -0.5);
insert into job values ('clean', 0.5);
commit;

select * from job;

Do read the ON CONFLICT page for details on the difference between ROLLBACK and ABORT. The behavior you see is both correct and SQL-standard.

(3) By Greg Wilson (gvwilson) on 2024-01-27 12:49:17 in reply to 2 [link] [source]

Thanks for the note about quote usage - old habits die hard :-). I tried on conflict rollback (apologies for not including that in the original post), but it doesn't seem to handle the multi-table case I mentioned. My full application wants to do something like this (using on conflict rollback to illustrate, but on conflict abort produces the same result):

create table job(
    name text not null,
    billable real not null,
    check(billable > 0.0) on conflict rollback
);
create table person(
   name text not null,
   job text not null
);

begin transaction;
insert into job values ('calibrate', 1.5);
insert into person values ('adrian', 'calibrate');
insert into job values ('reset', -0.5); -- this fails
insert into person values ('greg', 'reset');
commit;

If I run this and then select * from each table in turn:

|   name    | billable |
|-----------|----------|
| calibrate | 1.5      |

|  name  |    job    |
|--------|-----------|
| adrian | calibrate |
| greg   | reset     |

So the insert into job with an illegal value was rolled back, but what I want is to roll back the entire transaction: both tables should be empty. Table-level on conflict doesn't seem to do this (which I guess is reasonable); can I get the multi-table behavior I want, and if so, how?

(4) By jchd (jchd18) on 2024-01-27 13:07:38 in reply to 2 [link] [source]

In https://www.sqlite.org/lang_createtable.html click Show under table-constraint: to see that conflict_clause is not available to Check constraints, only Primary Key and Unique.

(5) By Greg Wilson (gvwilson) on 2024-01-27 13:17:02 in reply to 4 [link] [source]

Thank you, but returning to my original problem: how then do I implement a multi-statement transaction that only-but-always rolls back if an error occurs in any of the enclosed statements? (See my second post on this thread showing inserts into two different tables, one of which fails deliberately.)

(6) By jchd (jchd18) on 2024-01-27 13:28:48 in reply to 4 [link] [source]

AFAIK the only way to automatically achieve the wanted behavior is:

create table job(
    name text not null,
    billable real not null,
    check(billable > 0.0)
);

insert or rollback into job values ('calibrate', 1.5), ('reset', -0.5), ('clean', 0.5);

select * from job;

Or interrupt the flow of supplied statements on insert error. Remember: YOU supply the statements to SQLite. The library has no way to read ahead in the sequence of future statements you intend to have processed.

(8) By Adrian Ho (lexfiend) on 2024-01-27 14:01:21 in reply to 4 [link] [source]

Which seems to conflict (pun intended) with this statement in the ON CONFLICT page:

The ON CONFLICT clause applies to UNIQUE, NOT NULL, CHECK, and PRIMARY KEY constraints.

(10) By jchd (jchd18) on 2024-01-27 14:16:08 in reply to 8 [link] [source]

Even more "conflicts"!

In https://sqlite.org/lang_createtable.html#ckconst and below § 4.1 "Response to constraint violations" one can read:

The conflict resolution algorithm for CHECK constraints is always ABORT. (For historical compatibility only, table CHECK constraints are allowed to have a conflict resolution clause, but that has no effect.)

(11) By Adrian Ho (lexfiend) on 2024-01-27 14:22:50 in reply to 10 [link] [source]

Good catch, that'll learn me to read docs in the dead of night. :)

(7) By David Raymond (dvdraymond) on 2024-01-27 13:35:58 in reply to 1 [link] [source]

As tempting as it is to paste multiple lines into the CLI, it treats it as a buffer of commands to run 1 at a time, not 1 single large instruction.

If you really need for it to be all or nothing I'd suggest pasting the commands into a file. Then either run, or put at the top of the file the command:

.bail on

Then use the CLI's .read FILE  command to read commands from that file.

The ".bail on" command will tell it to stop at the first error it hits while reading from the file rather than continuing. You'll end up still in the transaction with the error, but at least the rest of the lines won't have been attempted.


...
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.
sqlite> .bail on
sqlite> .read myCommands.sql
Runtime error near line 8: CHECK constraint failed: billable > 0.0 (19)
sqlite> select * from job;
calibrate|1.5
sqlite> rollback;
sqlite> select * from job;
sqlite>

(9) By Greg Wilson (gvwilson) on 2024-01-27 14:09:40 in reply to 7 [link] [source]

Thanks for your feedback:

  1. I am running these commands from a file rather than interactively in order to save typing: sqlite3 example.db < example.sql.

  2. I don't want SQLite to stop at the first error: I want to roll back the entirety of a particular multi-table transaction without affecting (in particular, without aborting or rolling back) anything outside that transaction in the same SQL file. Is that possible?

(12) By Richard Hipp (drh) on 2024-01-27 15:24:34 in reply to 9 [link] [source]

SQLite does not work that way.

What you are asking for is some like this: If any command anywhere in a transaction failed, then when doing the COMMIT for the transaction, do a ROLLBACK instead.

With SQLite, each command succeeds or fails on its own. The transaction does not try to remember whether or not any commands have failed during that transaction. The COMMIT causes all commands that succeeded during the transaction to be committed.

(13.2) By punkish on 2024-01-27 16:04:06 edited from 13.1 in reply to 12 [link] [source]

this from someone who doesn't fully understand the nitty gritty of transactions (but wants to), if I understand you correctly, the following would be the SQLite way?

BEGIN TRANSACTION;
WITHDRAW 10000 FROM savings; ←-- succeeds
DEPOSIT 10000 INTO checking; ←-- fails
COMMIT;

edited to add a more db-like example

BEGIN TRANSACTION;
UPDATE savings SET balance = balance - 10000 WHERE rowid = 5;
-- failure happens here ← 
UPDATE checking SET balance = balance + 10000 where rowid = 88;
COMMIT;

What would happen differently if the above was not wrapped in that transaction?

It seems either way (with or without transactions) I am out 10000? I (perhaps mistakenly) thought the whole purpose of wrapping commands in a transaction was to get "all or nothing".

I realize the OP is working with an in-memory db, but for more normal, on disk dbs, I thought the reason transactions were so fast was because SQLite would save only when the transaction would successfully complete. Any failure in the transaction would mean nothing would get written to the db. Seems like I have misunderstood transactions all this time.

(14) By Isokaze on 2024-01-27 15:51:12 in reply to 13.0 [link] [source]

Your application sees that deposit fails and does not issue commit.

You get nothing.

(15) By punkish on 2024-01-27 15:59:24 in reply to 14 [link] [source]

which is exactly the behavior I was expecting from SQLite since it is the one providing the transaction mechanism. As I said, seems I have been completely misunderstanding this so far…

(16) By SeverKetor on 2024-01-27 16:27:26 in reply to 13.2 [link] [source]

What transactions get you is the ability to rollback, and for other connections to not see anything until you commit. If one of your queries fail and you plow on regardless, executing queries anyway, that's on you. Whatever program you have needs to account for queries failing and handle it appropriately. SQLite's job is to try to execute queries; if you give it a query you do not want it to execute, that's your fault.