SQLite Forum

Alternative implementation
Login

Alternative implementation

(1) By anonymous on 2024-07-14 20:41:01 [source]

Hi,

Are there any alternative implementation of SQLite format? There is a document describing the format, but my searches shows no other implementation beside the official code.
Most popular formats like ZIP or JPEG have multiple implementations and I am curios to find if others exists for SQLite too, and if not, why?

(2) By Tim Streater (Clothears) on 2024-07-14 21:29:10 in reply to 1 [link] [source]

Well - feel free to write one.

(3) By Harald Hanche-Olsen (hanche) on 2024-07-15 07:45:21 in reply to 1 [link] [source]

if not, why?

Perhaps because the existing implementation is of such high quality (and public domain to boot) that nobody sees a need to invest the effort?

That said, there is a pure go implementation underway. I think it's still in alpha, though.

(11) By Nuno Cruces (ncruces) on 2024-07-17 04:17:10 in reply to 3 [link] [source]

That's not a reimplementation, though. It's a machine translation of preprocessed C code. Each of the 18 architecture specific files is almost 10 MB, and has tens of thousands of LoC.

(4) By JayKreibich (jkreibich) on 2024-07-15 19:06:42 in reply to 1 [link] [source]

I feel like that's missing the point. ZIP and JPEG are file formats. The standards are about putting bits in a row in a file. Sure, there are reference implementations, but the whole point is that they are data storage formats completely and utterly independent of language, processing architecture, capture/display devices, or anything else.

SQLite is a database engine. It happens to have a file format associated with it, no different from many applications, but it's main purpose is not to be a file format. The file format can and has changed when new features are added, without the end-users (or developers) knowing or caring. And there are, in fact, alternative file formats, thanks to virtual tables, or branches like Oracle's Berkeley DB SQL API product.

I mean, I'm not aware of any "alternative" implementations of the Oracle DB or PostgreSQL file format. Does anyone care? Does it limit how you'd choose to use those databases?

(5) By Richard Hipp (drh) on 2024-07-15 23:00:54 in reply to 4 [link] [source]

SQLite has become a file format. I didn't think of it like that when I first designed it, but it quickly turned into a file format.

SQLite is a file format in the following senses:

  1. It is simple - at least as simple as you can get for a relational database. The average programmer can read the file format specification and understand it without a great deal of study.

  2. It is a single file (in the steady state - ignoring the transient rollback journal or WAL file) not a collection of files.

  3. It is well-defined and well-documented. A complete description of the file format takes about 20 pages, depending on what font size you use to print it.

  4. It is backwards compatible for 20 years, and our goal is to maintain this compatibility through the year 2050.

  5. It is cross-platform. The same file format works on 32-bit and 64-bit systems, and on big-endian and little-endian systems, and on Windows or unix.

None of the above can be said about the on-disk storage of any other RDBMS, as far as I am aware.

Developers have been using SQLite as a file transfer format for years, apparently. In https://odin.cse.buffalo.edu/papers/2015/TPCTC-sqlite-final.pdf, Kennedy et al found that a large number of SQLite database on Android phones are only read by SQLite and never written. They theorize that many applications use SQLite as a container file format to transfer large amounts of structured data down to the phone from the cloud.

To answer the OP's question, there have been a number of readers of the SQLite file format developed for forensic analysis tools. I know this because they wrote to me about ambiguities in the file format spec (which I subsequently corrected). However, I did not keep a record of who is doing that sort of thing.

I am not aware of anybody who has written an alternative writer for the SQLite file format.

(6) By Richard Hipp (drh) on 2024-07-15 23:08:52 in reply to 5 [link] [source]

Thinking more about this...

SQLite really is more of a file format than a database implementation. If you compare the C code for SQLite to the C code for version 3.0.0, there is some commonality, but the differences are striking. The byte-code has been completely redesigned. The query planning algorithms have been completely redesigned and reimplemented. The SQLite database engine is a very different creature than it was 20 years ago.

But the file format is the same, or is at least compatible (new features have been added).

Every release of SQLite contains significant changes to the C-code implementation. But the file format is unchanged.

I would argue that SQLite is, in fact, a file format and that the C-code we deliver is just a particular implementation of that file format.

(7) By ingo on 2024-07-16 09:06:57 in reply to 6 [link] [source]

Just out of curiosity and lack of knowledge on these matters, if I'd write a tool that writes a file conforming to the spec, can SQLite always read it?

(8) By Stephan Beal (stephan) on 2024-07-16 09:39:59 in reply to 7 [link] [source]

... if I'd write a tool that writes a file conforming to the spec, can SQLite always read it?

If it (SQLite) didn't, it would indicate an incompatibility vis a vis the spec. However, since the spec was written together with that specific initial/baseline implementation, and any changes made to what that implementation actually writes would likely result in compatibility problems, any fixes for such incompatibilities would probably happen in the spec rather than in the baseline implementation.

In any case, a separate implementation would definitely help validate the spec.

(9) By Harald Hanche-Olsen (hanche) on 2024-07-16 13:17:40 in reply to 8 [link] [source]

In any case, a separate implementation would definitely help validate the spec.

And not only that: Given a separate implementation, one could imagine submitting it to the IETF and have it become a standard data exchange format, with its own RFC and all.

(10) By David Jones (vman59) on 2024-07-16 14:29:51 in reply to 9 [link] [source]

There have been plenty of internet protocols which have independent implementations that are broken because they aren't bug for bug compatible with the reference code.

Then there was the phenomenon of Microsoft's 'embrace and extend' approach to standards.

(13) By Kees Nuyt (knu) on 2024-07-18 10:25:43 in reply to 9 [link] [source]

USA: The Library of Congress Recommended Formats Statement (RFS) includes SQLite as a preferred format for datasets.

(12) By Bill Wade (billwade) on 2024-07-17 13:03:04 in reply to 7 [link] [source]

At the level described in https://www.sqlite.org/fileformat2.html, I believe the answer is as close to yes as you'll ever find with human written software with this level of complexity.

At that level "always read it" works very reliably.

There are higher levels though, including the C API, and the SQL language as understood and implemented by sqlite.

Some SQL statements are ambiguous, time-dependent, or intentionally use randomness. They might also use undocumented "features".

If you search the documentation for "future", you'll see many cases where the developers say that today's behavior might change.

(14) By ralf (ralfbertling) on 2024-07-18 12:22:29 in reply to 5 [link] [source]

Not to be nitpicking, there is an interessting partial Implementation at https://www.arduino.cc/reference/en/libraries/sqlite-micro-logger/ that kind of strenghtens the point.

It would be interessting imho to write an Implementation in embedded swift with protection against race conditions to avoid the "evilness" of threads in traditional imperative languages (see https://sqlite.org/faq.html#q6)

Also, is there a roadmap beyond 2050? When that your was put into the documentation, that seemed much more like distant future then it does today.

(15) By Richard Hipp (drh) on 2024-07-18 12:36:39 in reply to 14 [link] [source]

is there a roadmap beyond 2050?

You think human civilization will survive past 2050? I appreciate your optimism and pray that you are correct. Even so, I have enough to do in just trying to get to 2050 without worrying about what comes after.

(16) By J. King (jking) on 2024-07-18 12:41:58 in reply to 14 [link] [source]

Also, is there a roadmap beyond 2050? When that your was put into the documentation, that seemed much more like distant future then it does today.

Hopefully Dr. Hipp will be happily retired and someone else will be adroitly setting the roadmap, by then.

(17) By ddevienne on 2024-07-18 13:32:04 in reply to 14 [link] [source]

interesting partial Implementation at https://github.com/siara-cc/sqlite_micro_logger_arduino/tree/master

Thanks for sharing. I've often considered something like this, nice to discover someone actually doing it (because of embedded constraints)

(19) By Donald Griggs (dfgriggs) on 2024-07-19 15:08:59 in reply to 14 [link] [source]

That micro-logger looks like a great idea. Maybe it should have a nickname of SqUltraLite.

(18) By Max (Maxulite) on 2024-07-19 08:37:19 in reply to 5 [link] [source]

To answer the OP's question, there have been a number of readers of the SQLite file format developed for forensic analysis tools.

It's interesting. Initially I thought why the native reader is not enough for those guys. Then an obvious application came to mind when an sqlite database has a bunch of non-zeroed free pages and one needs to restore as much as possible from them. Having so many mobile apps when most of them store the data in an SQLite base, the task is interesting for forensic analysis. One has to know the format well enough to extract information even from an orphan database page. I wonder whether such tools are good for general users when it's impossible to return the base to some correct previous state, but maybe to show some deleted columns/rows from a table.

A side note. While doubting that freepages might give much, found the article that reveals that not only much, but not only from free pages. This is obviously a commercial product-supported article, nevertheless worth mentioning

(20) By Stephan Beal (stephan) on 2024-07-23 12:19:09 in reply to 1 [link] [source]

Are there any alternative implementation of SQLite format?

This sqlite file reader, implemented in Rust, just showed up via HackerNews:

https://blog.sylver.dev/build-your-own-sqlite-part-1-listing-tables