SQLite Forum

Can Richard do some YouTube videos explaining transaction implementation?
Login

Can Richard do some YouTube videos explaining transaction implementation?

(1) By example-user on 2020-09-04 19:54:30 [link] [source]

On Federico Razzoli's YouTube channel Richard mentions how he does not have time to explain transactions (at around 30min):

https://youtu.be/2eaQzahCeh4?t=1800

I know the code is commented and there is text based documentation, but I think it would be interesting and useful to have Richard walk through at a high level how transactions work.

Richard, is this a possibility? Would any one else be interested in this also?

(2) By Stephan Beal (stephan) on 2020-09-04 20:34:31 in reply to 1 [link] [source]

I know the code is commented and there is text based documentation, but I think it would be interesting and useful to have Richard walk through at a high level how transactions work.

Does googling for "how do database transactions work?" not tell you what you're wanting to know? At a high level, the process is the same for most transactional systems.

If you're looking for the specifics of sqlite3's implementation, those details are not part of the public interface, and it seems a shame to waste time documenting, in a "permanent" video, any internal implementation details which may change from one day to the next.

(3.1) By example-user on 2020-09-04 20:54:21 edited from 3.0 in reply to 2 [link] [source]

Does googling for "how do database transactions work?" not tell you what you're wanting to know?

No because SQLite is embedded so the implementation details are different from a typical server db (that implements isolation by controlling all the clients in a central process).

I'm more interested in how SQLite implements transactions specifically.

I think Richard could also provide a lot of historical context as to why it was implemented the way it was.

and it seems a shame to waste time documenting, in a "permanent" video, any internal implementation details

Well, commenting and writing code isn't "wasting time", so I would consider the video as code comments encoded in video and audio. Also the time spent is work towards the goal of supporting SQLite until 2050 - any programmers working on the code base could watch the video.

(4) By Keith Medcalf (kmedcalf) on 2020-09-04 22:29:52 in reply to 3.1 [link] [source]

Low Level file locking is well described and documented. Perhaps start here:

https://sqlite.org/lockingv3.html

(5) By Keith Medcalf (kmedcalf) on 2020-09-04 22:53:04 in reply to 3.1 [link] [source]

Note that this is pretty much how all "Server RDBMS" handle local storage consistency. The only difference between a "Server RDBMS" and SQLite3 is the "Server" part -- a "Server RDBMS" is nothing more than SQLite3 wrapped up with remote procedure call interface. The "database file" is still "local" to the "RDBMS" and the process for maintaining consistency between the "local database file on disk" and the "RDBMS Server process" is practically indistingushable.

Generally however the "Server RDBMS" is a single process so there is no capability or provision (or need) for the "Server RDBMS" to provide consistency between multiple instances of itself.

(6) By Ryan Smith (cuz) on 2020-09-05 02:00:54 in reply to 3.1 [link] [source]

Well, commenting and writing code isn't "wasting time",

That's completely not what Stephen said - he hints at the idea that making extraneous effort to hard-document (Like books, video, etc.) soft code (stuff that may change daily) is a great waste, which it is.
Of course he didn't say commenting code is a waste, as the strawman argument implies, and I can confirm that the Transactional code in SQlite is very-well commented (much like the rest of it) and what is more, is open to the public, so you are very welcome to read it there and perhaps make a video of it - but be warned, it could change tomorrow, and if it does it will still be well-commented, but your video will have become obsolete - which embodies the "waste" that Stephen pointed out.

so I would consider the video as code comments encoded in video and audio.

Really? You would consider it so? Well then, how many videos have you made and posted that serves as code-comments to your code?

Also the time spent is work towards the goal of supporting SQLite until 2050 - any programmers working on the code base could watch the video.

There are only 3 programmers working on the code-base, and they all understand it intimately already. For those who wish to fork it, there are the actual code comments.

Honestly though, I can agree with your sentiment - as a programmer, it would be fun to watch Richard wax lyrical about the trials and tribulations of designing the SQLite Transaction code. Any such effort has to however be on his time and dime (unless you are offering a bounty for it?) and will be a luxury at best.

Offering unfounded "reasons" why it should do be done in service of the community/code/greater good/support/etc. is just a tad, well, let's say "naughty".

(7.1) By example-user on 2020-09-05 11:55:13 edited from 7.0 in reply to 6 [link] [source]

That's completely not what Stephen said - he hints at the idea that making extraneous effort to hard-document (Like books, video, etc.) soft code (stuff that may change daily) is a great waste

  • My perspective is that it is what he said.

  • Code comments and video both aim to do achieve the same goal: "put my understanding into your brain".

  • So in general both comments and video are "time spent by one human to communicate to another".

  • Because they are both in the same set, the same logic applies.

    • Comments are also a waste if the underlying code changes, the same as video based communication.

      • This is where my point comes from - Im saying if its worth commenting line by line, perhaps its also worth making a few hours of video based content to explain an overview of a feature.

Really? You would consider it so? Well then, how many videos have you made and posted that serves as code-comments to your code?

  • Comments are only used because they are text based and are the most efficient way of communicating.

  • If editors had a standard way of attaching video, audio and images inline with code, I think they would be quite popular.

  • I do not do "video as code comments" because it is more expensive with my current tools, but if I were to explain a code base to a newcomer, a face to face interaction explaining the system at a high level is often much more useful than starting with the lower level comments.

Offering unfounded "reasons" why it should do be done in service of the community/code/greater good/support/etc. is just a tad, well, let's say "naughty".

  • As above, I think the reasons are valid.

(8) By Stephan Beal (stephan) on 2020-09-05 11:56:49 in reply to 7.0 [link] [source]

My perspective is that it is what he said.

His own perspective is that Ryan's interpretation is precisely what was initially intended.

(9) By example-user on 2020-09-05 12:12:27 in reply to 8 [link] [source]

To be clear:

  • Your point is:

    • Video = wasted time because code changes.
    • Comments = Not wasted time.
  • My point:

    • Video and comments = both time spent communicating about code that changes.
    • Comments are worth doing therefore so is video.
      • Inverse: If video is a waste because code changes, comments are also a waste.

(10) By Richard Damon (RichardDamon) on 2020-09-05 12:54:07 in reply to 9 [link] [source]

One big difference, the comments are tied to the code, so if you change the code, it is fairly easy to change the comments, so the reflect the new code.

Video, or any other 'external' documentation isn't so clearly attached. First, which version of the code does this documentation apply to, this matters a lot if you are 'documenting' internal details, and not public API. It also may be less clear that the video/other documentation needs to be updated, unless you add a comment like this section described in YouTube video wxyz.

Also, editing a comment to reflect a small change is a small task. To 'edit' a video (to change technical content) tends to mean totally redoing it.

(11) By Warren Young (wyoung) on 2020-09-05 13:17:07 in reply to 9 [link] [source]

As someone who has produced a few dozen technical videos, there's no comparison at all between commenting code and producing high-quality video.

The very best case is that you've already got the tech set up to produce good-enough video, and that you can just rip through the content stream-of-consciousness style and upload the video file at the end, as-is.

Most people don't have a good studio set up. A laptop with a potato camera pointing up your nostrils with the blue-purple LCD glow making the subject look like a necromancer's failed experiment is not what I call "good video." For content like this, audio is more than half the value, so you also need a studio mic on a boom, a pop filter, and audio post-processing. (Dynamic range compression, noise removal, etc.)

For highly technical topics like this, you want to write a script first. Even if you go off-script when making the video, the very act of writing the script improves the result by ensuring you touch on all of the important topics. That takes more much more time to write than these code comments you're talking about. Most of the comments are already there, so they only need maintenance, and writing for presentation is harder than just jotting down notes-to-self in the code.

If you need/want to edit the resulting video, it'll cost you at least 3x the finished running time. Over 1x to record it, another 1x to watch it again, and another 1x worth of time in stops and starts as you do the actual editing. And that's a quick edit, mainly just cutting out ums and ahs. A complex edit with titles and segment rearrangement and multiple cuts might take 5x the video's finished time or more.

Having spent all that effort and time, now you have an artifact that's pinned in place, nearly unchangeable, because you can't just insert a new sentence into the middle of a video, like you can into a code comment. You have the 3x rule again, the video studio's likely been changed around since you set it up last, so you end up with visual jumps in the video and audio quality by just inserting new snippets.

Most of the time, it's simplest to just re-record the whole thing, so you pay another 3-5x real time cost to redo the video.

(12) By anonymous on 2020-09-05 15:10:52 in reply to 11 [link] [source]

I have a solution...What if drh made a video as perfectly as he could and never changed the code. That way the video and code are in sync with one another.

(13) By Keith Medcalf (kmedcalf) on 2020-09-05 19:17:32 in reply to 12 [link] [source]

Why not just read the written documentation on the sqlite.org website?

(14) By example-user on 2020-09-05 19:31:12 in reply to 13 [link] [source]

It was just a suggestion that maybe a video or two would be awesome.

I didn’t think I would get such strong opposition

(15) By Warren Young (wyoung) on 2020-09-06 13:30:19 in reply to 14 [link] [source]

a video or two would be awesome.

Certainly it would. That's not in question. What I think people are trying to show you is that drh is already terribly busy, and you're now wanting him to put in something like 4-8 hours for a single video, simply so you can avoid studying the source code he's already generously provided.

I mean, think about that for a second: do you realize how lucky you are to live in a time when you can simply expect that a product like SQLite is FOSS? When I was getting started, a sales guy hawking the closest alternative to SQLite quoted me US $30000 per developer seat in decades-ago dollars, plus royalties on the distributed copies!

In my analysis of the effort needed to produce high quality technical videos above, I left out the cost of producing visual aids. My 3-5x number is for videos where the visuals are "free," things like screencasts and talking-head presentations. As soon as you add infographics and 3D animations and such, the costs balloon tremendously. Budget something on the order of 10000 times real-time: a visual that stays on screen 3 seconds will likely cost 30000 seconds to produce, or about 3.5 hours.

Want two visuals? 7 hours. It adds up.

One of the best ways to learn how something works is to try to teach it to someone else, so consider this: learn enough about how SQLite transactions work that you can put something on YouTube. When the comments roll in, dig back into the code to address all of the objections, then put out a second video. Ten videos in, I'll bet you understand SQLite transactions better than 99.9% of the people on this forum.

(16) By example-user on 2020-09-06 14:27:31 in reply to 15 [source]

Ok