SQLite Forum

What is "in-process" in "SQLite is an in-process library"?
Login

What is "in-process" in "SQLite is an in-process library"?

(1) By anonymous on 2022-03-10 08:00:02 [link] [source]

Sorry for a noob question, but is meant by "in-process" in "SQLite is an in-process library"?

(2) By Stephan Beal (stephan) on 2022-03-10 08:22:55 in reply to 1 [link] [source]

"in-process" in "SQLite is an in-process library"?

It means that sqlite's features are running in your application, not an external process which your application connects to. This is the main feature which differentiates sqlite from other major database engines (Postgres, Oracle, MySQL, etc.).

(3) By ddevienne on 2022-03-10 08:42:57 in reply to 1 [source]

To completement Stephan's answer, with a traditional database engine,
the DB code in your application is not processing the SQL. It's sending
it to a remote server for processing, and reading the response from a
network socket.

While with SQLite, there is no client nor server. The SQL engine is in
the library, thus in your application directly. Thus no network involved.
This makes SQLite very fast, limited only by your local disk speed typically.

The drawback is that SQLite MUST use local disks, because it relies on
the filesystem for locking, and all network file-systems are either buggy or liars,
which leads to corruption in concurrent accesses. Remote file-system are also
lying about fsync, leading to loss of durability.

(4) By Warren Young (wyoung) on 2022-03-10 09:44:07 in reply to 3 [link] [source]

a remote server

It could instead be on localhost, but even then, a TCP localhost or Unix domain socket conn requires a bounce through kernel space for each transaction, adding latency to each.

Consequently, "Many Small Queries Are Efficient In SQLite."

(5.1) By cj (sqlitening) on 2022-03-11 19:02:51 edited from 5.0 in reply to 3 [link] [source]

Do you use "begin exclusive" on a network?
I run SQLitening server and also locally peer to peer with no problems.

What Windows bugs on a network? Please post some failing test code.

(7) By Larry Brasfield (larrybr) on 2022-03-11 21:44:18 in reply to 5.1 [link] [source]

What Windows bugs on a network?

Mr. Devienne stated an alternative of which only one involves "bugs". I do not see that he claimed any "Windows bugs".

Please post some failing test code.

The reason that use of SQLite with a network separating its running code from its data store is disrecommended is because it is known to be problematic, not that it fails consistently. Many people get away with cross-network usage scenarios for a while, then come here with "How can I uncorrupt my database?" when, eventually, they do not get away with ignoring the recommendation.

Absence of a reliable failure case does not equate to "reliable".

(8) By Donal Fellows (dkfellows) on 2022-03-14 12:44:13 in reply to 5.1 [link] [source]

What Windows bugs on a network?

There is literally no way to get locking to work correctly on a network, irrespective of OS. The fundamental problem is that networking hardware necessarily has failure modes that local disks do not (or at least not with a frequency that anyone cares about), such as packets getting lost — even selectively lost — due to the activity of other, unrelated systems, or cables getting unplugged; these are real things that happen. That makes taking any kind of decision on locking in a networked filesystem either deeply unsafe or susceptible to ending up deadlocked. For most applications this doesn't matter, but for a database this is absolutely critical; if you set up a classic database server then I'll bet the instructions say to use local disks too. (There are remote consensus protocols, but they can take a very long time to reach a decision at all.)

Of course, if you're just doing read-only access to a remote SQLite DB then the chance of corruption is very small, but write access needs locks to be safe, and locks require being local.

(6) By Keith Medcalf (kmedcalf) on 2022-03-11 21:29:59 in reply to 3 [link] [source]

The drawback is that SQLite MUST use local disks, because it relies on the filesystem for locking, and all network file-systems are either buggy or liars, which leads to corruption in concurrent accesses. Remote file-system are also lying about fsync, leading to loss of durability.

This same caveat applies to every other DBMS system as well (except that there can be some exceptions because a separate server process does not have to "share" database access between multiple processes) and those database files must be local to the DBMS server (however, that may be remote from the client application).

SQLite has no separate "server process" (the "client" accesses the database file directly) hence the database must be local to the client.

Same tea, different cups ...