It can be determined by the means of static source code analysis that several return values from function calls are not used for desirable condition checks.

Would you like to add more error handling at places like the following?

• fclose ⇒ main
• fseek ⇒ Parse
• ftell ⇒ main
• fwrite ⇒ putsVarint
• malloc ⇒ asciiToUtf16le
• printf ⇒ usage
• pthread_create ⇒ main
• pthread_mutex_init ⇒ pthreadMutexAlloc
• putc ⇒ sqlite3VdbeReset
• strdup ⇒ list_insert

Would you like to add more error handling at places like the following?

No.

All of the examples you provide (except pthread_mutex_init) occur in debugging code or in auxiliary programs that are used during the SQLite build process or for SQLite troubleshooting and analysis, not in the core SQLite library. There is no point is checking for obscure and exceedingly rare errors in what amounts to scaffolding.

The pthread_mutex_init() call is in the core SQLite library. And the API documents do say that it is possible for pthread_mutex_init() to fail. But, unless I'm badly mistaken, the actual implementation of pthread_mutex_init() never fails, at least not on any systems that I have access to. So adding a test for a non-zero return from pthread_mutex_init() would be pointless. It would also be impossible to write a test case for that branch, as it would never be taken.

Thank you for the report. But this is a "won't fix".

obscure and exceedingly rare errors

It's not even that: some of these are "cannot happen" cases.

For instance, two are places where the program's just opened a file successfully, then it immediately seeks to the end to get the file size. How does it fail here, except by something catastrophic like the filesystem being force-unmounted at that exact instant? And then, this having occurred, it'll get a bogus size value on the very next line, right?

What's the bad thing that can then happen? The file's gone now! What's the tool going to do to the file to damage it?

This is a classic case of using a tool as a substitute for thinking instead of as an adjunct to it.

The only criticism of SQLite I came up with while studying these is the use of bare constants in the first fseek() call the OP points to instead of SEEK_END. I doubt I'm the only one not to have memorized that the constant is defined somewhere as "2".

And the API documents do say that it is possible for pthread_mutex_init() to fail.

I suggest to avoid ignorance of return values a bit more.

• Would you like to detect every error situation as early as possible?
• It is just safer to handle also “exceptional” situations, isn't it?

I stopped studying these at the fourth one, having come to drh's conclusion with the first and having it confirmed by the next two. If you want to report static analyzer output, you'd do far better to filter out those that are outright ridiculous, to help people focus on those that matter.

Should you come back with reasons why we should look more deeply into these reports, I suggest that you point to the actual SQLite source code repository rather than to the GitHub mirror. Not only is it more polite to point at the Fossil instance — being the tool drh wrote to manage SQLite — we'd have a better chance of helping you that way.

I say this because your fourth link doesn't point to an fwrite() call at all. If that were a Fossil link, I could then make an informed guess about how you came to produce that URL and help you out with it, since the Fossil URL scheme is generally sensible, and in the rare cases where it's confusing, we have the source code to dig into to find out why it's doing what it is. As it is, I'm left with guesswork as to why it isn't pointing at what you think it ought to. The GitHub URL parser is closed source code, and the docs I've been able to find on their URL format are thin and opaque.

…, I suggest that you point to the actual SQLite source code repository rather than to the GitHub mirror.

The corresponding probability can grow when the source code search capabilities of the information system “Fossil” will be improved.

I say this because your fourth link doesn't point to an fwrite() call at all.

Your view is appropriate for the link target.

I imagine that the implementation of a function like “putsVarint” can trigger further software development considerations.

While it's true that Fossil's web search function doesn't grep the source code for you, presumably this unnamed static analyzer of yours gave you a file name and a line number for each of the complaints it emitted. Therefore, the procedure is:

• Click "Files" at the top of the site I linked you to
• Navigate to the file in question
• Optional: select a particular commit of that file so the result of this procedure gives a stable link. (This, I believe, is what happened to your GH links, but I can only guess from the opaque /blob URLs you gave. With a Fossil URL, I'd know for sure what you did and why it isn't working.)
• Enable the "line numbers" option
• Click the line number your tool gave you
• Click the "copy link" button you get from that

Compare these URLs:

Fossil, default: https://sqlite.org/src/file?name=tool/enlargedb.c&ln=58

Fossil, version-pinned: https://sqlite.org/src/info?name=3e8b2612b985cfa7&ln=58

GitHub: https://github.com/sqlite/sqlite/blob/37d4ec86bfa78c31732132b7729b8ce0e47da891/tool/enlargedb.c#L58

Do you not agree that the Fossil URLs are not only shorter, but more readable?

Do you not agree that the Fossil URLs are not only shorter, but more readable?

I hope that Fossil's web navigation interface can become better and more convenient (also for discussed implementation details).

We really should take this up on the Fossil forum, but just to respond to this point, I don't see that GitHub is materially different here.

GitHub shows what Fossil calls the "Files" view by default, whereas Fossil lets you choose what you see by default. In this case, drh chose to present an information page instead, but it's trivial to configure Fossil to give the /dir view instead. I do this on some of my repos, where a list of files is indeed what you're most likely to want, rather than information about the project. (Example)

The procedure above otherwise applies almost step-for-step to GitHub. Did you get your URLs by another path?

Should you chose to reply — on the Fossil Forum, please — I'd like to know what specifically you think is so bad about the Fossil way of doing this and what ought to change, rather than "become better and more convenient".

In this case, drh chose to present an information page instead, but it's trivial to configure Fossil to give the /dir view instead. I do this on some of my repos, where a list of files is indeed what you're most likely to want, rather than information about the project.

This sounds promising.

Would another direct link into the development repository of SQLite become helpful accordingly?

I'm not sure what you're asking for. This is the /dir link I referenced above for that repo, which is what you get by clicking the Files link at the top of the default page. What more "direct link" did you want?

By the way, I belatedly noticed one minor difference between my procedure above and what happens on GitHub: they show line numbers by default, where Fossil does not. It's something of a controversial topic due to lack of an obviously-correct answer, only opinions.

If you count a single-click difference as significant here, then it nets out to zero, since GH makes you click the "..." button before you get the "copy link" submenu, whereas Fossil shows that by default.

… we'd have a better chance of helping you that way.

Will the development attention ever grow also according to more complete error detection and corresponding exception handling?

Examples:

• Parse
  
• pthreadMutexAlloc
• putsVarint

Thank you for updating the URLs. They're well-formed permalinks now, and short besides.

Taking those in order…

• Lemon generates SQLite's SQL parser, though I'm not certain if the line of code you reference is part of any normal SQLite code path, since that would imply getting SQL from a file.

  Regardless, I covered this issue above: how do you open a file successfully, then have a seek to EOF fail on the very next line, other than a very wild race condition? And, if this does happen, what bad thing then occurs?

• The only documented failure modes for pthread_mutex_init() on my platform are "invalid parameter", which we can be reasonably sure isn't going to happen here, and "out of memory" when allocating the mutex. The latter seems sensible to cover, though wildly unlikely on modern systems.

• If putsVariant() fails on the fwrite() call, what should it do? It currently returns void, meaning no calling code is checking for errors. Unless you change all that, I'd say (void)fwrite(... is the appropriate response here. That is, document that it's suppressing the warning and move on.

Lemon generates SQLite's SQL parser, …

Can any more function calls occasionally fail during the parser generation?

And, if this does happen, what bad thing then occurs?

Is the determination of the file size questionable accordingly?

The latter seems sensible to cover, though wildly unlikely on modern systems.

Mentioned function calls can eventually fail for more reasons than you expect here.

It currently returns void, meaning no calling code is checking for errors.

If a programming language like C++ could be used, a selected exception could be thrown accordingly.

That is, document that it's suppressing the warning and move on.

I hope that a better error response can be achieved.

Is the determination of the file size questionable accordingly?

Sure, but my question above stands: if fseek() fails, doesn't that imply that the file isn't even there any more? What does it matter if you get a bogus value back from ftell() afterwards? What will then happen? You can't write to the file; it's gone. Likely, the storage device the file was on is gone, too.

If you come up with a plausible failure mode, you'll find a more receptive audience.

Mentioned function calls can eventually fail for more reasons than you expect here.

Again, come up with a real failure mode, not output from an ultra-picky static analyzer.

Realize that a modern C compiler is also a type of static analyzer, and I'm not aware of any that warn on this case, even in -Wall mode.

It feels like you're just trying to be right here, rather than pointing out actual flaws that need to be fixed. Slavishly obeying dumb tools is that substitute for thinking I referenced above.

If a programming language like C++…

Ain't gonna happen. Part of being "lite" is using C. It's why it's as portable as it is, and why it's used on platforms as small as it is.

Don't argue; this isn't the place for it.

an exception could be thrown

How's that going to work in a project like this where a huge number of users don't write to SQLite in the same language it's written in? None of the API wrapper mechanisms will translate a thrown exception into the caller's language, that I'm aware of.

I hope that a better error response can be achieved.

To what end? Name a case where that fwrite() call fails and it matters.

…, rather than pointing out actual flaws that need to be fixed.

I am trying to reduce remaining programming mistakes for several software components also by increasing the awareness for incomplete error detection.

Name a case where that fwrite() call fails and it matters.

How do you think about further software tests with a file like “/dev/full”?

Realize that a modern C compiler is also a type of static analyzer, and I'm not aware of any that warn on this case, even in -Wall mode.

Would you like to take another look at development tools like the following?

• Secure Programming Lint
• Semantic patch language (Coccinelle software)
• Cppcheck

How do you think about to extend the application of function attributes?

• warn_unused_result
• nodiscard

I will do "complete error detection and exception handling" in cases where it makes, or might make, a real-world difference. But in the three cases you cite, I think the chance of encountering problems is only theoretical.

1. The "Parse" issue is in the Lemon parser generator that runs as part the build process. If you are using the "sqlite3.c" amalgamation that we make available (most people do) then this is not a factor at all. Even if not, when was the last time you had an fseek() failure, really? Suppose fseek() did fail? What is the worst outcome that could happen.

2. We've already discussed the fact that pthread_mutex_init() never fails in practice, even if the documentation says it can fail in theory.

3. The fwrite() failure in the "sqldiff" utility will only come up if you "sqldiff" command-line utility with the --changeset option and you subsequently run out of disk space to hold the result. Has this ever happened in the real world? Even once?

Why are you stressing over this? What are you hoping to accomplish?

Adding "complete error detection" is not free. There are costs involved. There are run-time costs to check for the error conditions. There are long-term maintenance costs associated with the error checking. There is the developer time needed to write new test cases and to validate the error checking. On the one hand, it is important to detect and deal with error. But on the other hand, there is no point in devoting resources to dealing with errors that never happen in the real world.

I'm claiming that none of these errors are ever likely to happen in the real world. And even if they do, the problem will be detected by other means shortly thereafter. So the maximum benefit to adding a bunch of new code is a slightly better error message.

I don't think addressing any of your concerns will move the project forward nor be of any service to humanity. You are welcomed to dispute that if you want, but until I have a cogent explanation of why these issues are important my answer remains: WONT_FIX

Adding "complete error detection" is not free.

How much would you like to care for secure coding guidelines (and hints from the Common Weakness Enumeration)?

There are costs involved.

This is a known fact.

But on the other hand, there is no point in devoting resources to dealing with errors that never happen in the real world.

I suggest to reconsider such a view together with improvements for advanced development tools.

I'm claiming that none of these errors are ever likely to happen in the real world.

Development methodologies are available which can cope better with exception handling.

And even if they do, the problem will be detected by other means shortly thereafter.

I propose to reconsider also this assumption in more detail.

So the maximum benefit to adding a bunch of new code is a slightly better error message.

The error information transfer can be improved at several places, can't it?

I am not persuaded by your arguments. WONT_FIX

I am not persuaded by your arguments.

I hope that other contributors can influence the presented change reluctance in more desirable directions.

Do you know how many databases are out there using SQLite, and in how many different OSes and flavors of OSes? Don't you think that some other folks would have complained if any of your "supposed scenarios" had occurred?

That's because he hasn't presented any cogent arguments or sensible suggestions.

I would appreciate if such a view can be reconsidered.

How much do you care for advices from linked information sources?

Would you like to care more for implementation details which are connected with undefined behaviour?

See also:

• EXP34-C: Do not dereference null pointers
• CWE-690: Unchecked Return Value to NULL Pointer Dereference

Would you like to care more for implementation details which are connected with undefined behaviour?

An actual, demonstrable "connection" of that sort, (rather than a speculative one drawn from generalizations such as you cite), would of course be of interest.

Can you point to any SQLite code where a return from some call that might return a NULL pointer is not checked prior to its dereference? If so, I urge you to partially repair your credibility here by pointing to that code. If not, you will have further diminished such credibility as might still remain.

See also: ...

All very good, and utterly obvious, advice. Advice which has not been shown to be needed or applicable, yet.

I only looked at a few examples above before giving up. I noticed that none of those cases are in deliverable code. It all seems to be code used only for testing or in disused, historical code artifacts.

Here's an idea: Why don't you run your static analysis on the deliverable code for SQLite - the two files "sqlite3.c" and "sqlite3.h". The "amalgamation". That is the code that actually matters. That is the code that runs in your cell phone and in your web browser and in many (most) of the application programs that you use every day.

The amalgamation is not in the source tree. It is a build product. You can download a pre-built amalgamation from the SQLite website. Or you can build it yourself using:

./configure && make sqlite3.c

Or

nmake /f Makefile.msc sqlite3.c

In this forum thread, you have so far cried wolf multiple times, to the point that people are beginning to ignore everything that you say. You may yet redeem your reputation by using your static analyzer to find actual problems in the amalgamation.

I noticed that none of those cases are in deliverable code.

I suggest to reconsider such information once more according to affected components.

• Loadable database extensions
• Software build
• Test cases

…, to the point that people are beginning to ignore everything that you say.

I became curious if further OOM checks will become relevant (in more development branches).

Adjustment examples from 2023-01-14:

• 19:27:40
• 19:36:47

I'm going to explain what is perhaps obvious.

Richard has suggested that your efforts with static analysis would be better directed at shipping code, and he specifically mentioned the amalgamation (known as "sqlite3.c" with "sqlite3.h".)¹ The same considerations apply to another build product, "shell.c", which is normally downloaded alongside the amalgamation. This, too, is shipping code.

Because of my work on the CLI, implemented in "shell.c", I recognized a filename in the middle of one of your lists as one which is incorporated into "shell.c" rather than being disused or part of test scaffolding. For that reason, and because I was curious as to why an allocation result might go unchecked, I went to see what was going on. This quickly showed an unchecked malloc return which was then used in a function which, per its documentation, may dereference the pointer. So I fixed it.

The exact same change, (the last of your "examples"), was taken from the branch I put the fix on because it represents a theoretic defect² in shipped code.

I became curious if further OOM checks will become relevant ...

That's hard to say in advance. If you were to report such unchecked returns found in "sqlite3.c" or "shell.c", I would investigate and either correct the code or inject comment(s) to indicate why no check is needed there. However, I have to say: When alleged improvement opportunities are embedded in a flurry of irrelevant finds from a tool which is ignorant of the code's larger context, it is more difficult to see them as worth the time needed for investigation. This sort of difficulty is only increased by the presence of formulaic advice which does not account for the project's objectives and methodologies.

1. ^{^} I have seen no evidence that this suggestion took root in your thinking or action, so my further suggestion here seems a bit futile.
2. ^{^} The CLI is used in contexts very different from the memory-constrained systems that the SQLite library is designed to support, so OOM issues in the CLI are handled with abrupt termination in most cases, a result little different from accessing memory at address 0.

So I fixed it.

Would you like to point any corresponding software updates out?

When alleged improvement opportunities are embedded in a flurry of irrelevant finds from a tool which is ignorant of the code's larger context, it is more difficult to see them as worth the time needed for investigation.

I imagine that there are possibilities available for further development considerations.

Can content management systems help a bit more to adjust remaining concerns?

This sort of difficulty is only increased by the presence of formulaic advice which does not account for the project's objectives and methodologies.

How will communication challenges evolve further?

Would you like to point any corresponding software updates out?

It's right there in the fossil timeline, including a reference to this forum post: https://sqlite.org/src/info/eda84dcffee6016f. I barely know how to use fossil and this took 5 seconds to find (via manual inspection of the timeline for recent commits from Larry).

Would you like to demonstrate some basic effort instead of asking a thousand questions which prompt others to do all the work?

Would you like to demonstrate some basic effort …?

Did you notice that I added links according to software development for OOM checks?

Hi Markus.

The requests from the SQLite development is this: Provide a sample code or script that can cause everything you are talking about, and it will be addressed. Something like this:

/*
  some vars
*/
int main(char[][] args)
{
  int result = 0;
  //some code that causes the problem you state
  return result;
}

and create a new forum request with a meaningful title or subject, copy and paste that program or script in the body and provide any steps to replicate the problem and also think of errors that one can search on for others to find that same problem, preview it, fix any thing that you see wrong, preview it again, and when you think that it's all set, then hit submit. That is what they need.

… Provide a sample code …

There are no extra examples needed.

The source code for components of software like SQLite (or Fossil) contains several open issues which can easily be found.

Some development tools can help further.

Why are you stressing over this? What are you hoping to accomplish?

You've not answered this- you seem quite keen and focused on this 'issue' (using the word issue loosely), and if the main developer confirms it's not an issue, why are you still digging, trying to find an issue?

You've not answered this

I got the impression that I answered enquiries in several ways.

…, and if the main developer confirms it's not an issue,

Some programmers would like to invest their development resources in other ways.

(Questionable “stories” are kept in a backlog, aren't they?)

why are you still digging, …

I hope that available information will get more attention so that further collateral evolution will happen.

I guess that risks can be considerably reduced according to undesirable software behaviour.

I suggest to take also another look at the software “SQLite Expert Extension”.

The analysis tool “Cppcheck 2.9.3-1.2” provided the information “Redundant initialization for 'rc'. The initialized value is overwritten before it is read.” (See also: CWE-563) for the implementation of the function “idxCreateVtabSchema”.

This detail indicates that a return value was not appropriately checked from a call of the function “idxRegisterVtab” (and therefore from a sqlite3_create_module() call), doesn't it?

I suggest to take also another look at an utility program for the determination how indexes are used in a database schema.

The analysis tool “Cppcheck 2.9.3-1.2” provided the information “Variable 'rc' is reassigned a value before the old one has been used.” (See also: CWE-563) for an implementation of the function “main”.

This detail indicates that a return value was not appropriately checked from a call of the function “sqlite3_exec”), doesn't it?

By the way: Can it be clarified if a null pointer may be passed in the parameter “sql” of the function “sqlite3_exec”?

Would you become interested to take another look at further improvement possibilities also by the means of ​the semantic patch language (Coccinelle software)?

Will anybody become interested for updates according to source code analyses also by the means of the tool “Coverity Scan”?

Will anybody become interested for updates according to source code analyses also by the means of the tool “Coverity Scan”?

As a rule, if a "bug" discovered by static analysis tools is not proven by code which demonstrates how the bug can be triggered, it won't get much attention here.

The simple fact is that static code analysis is wrong at least as often as it is right and the core developers of this project (a team of two people) do not have the bandwidth to chase down "false alarms." If, however, a bug or vulnerability is demonstrated with working code, they invariably correct it quickly.

As a rule, if a "bug" discovered by static analysis tools is not proven by code which demonstrates how the bug can be triggered, it won't get much attention here.

Is such a development view in conflict with secure programming guidelines?

Is such a development view in conflict with secure programming guidelines?

It's in conflict with practical programming practices and real-world constraints. The core sqlite team is only two people and most "bugs" reported via static analysis tools are not bugs, so it's a waste of their time to spend time pursuing every "bug" report made via static analysis tools. If real code which demonstrates a bug is shown, they invariably immediately work to resolve it.

If real code which demonstrates a bug is shown, they invariably immediately work to resolve it.

Would any more contributors dare to admit that several discarded/unused/ignored return values are worth for further development considerations and corresponding improvements?

Sure - if you can prove that fixing the unused/ignored return values improves the product by fixing one or more bugs ..

Would spending a month fixing all ignored return values actually improve the finished binary? This would be like going through the source code manually changing all the tab spaces from four to five spaces. Would it actually improve the binary? Would it reduce the issue count from (hypothetical) 50 issues to 30 issues?

At the moment, spending time fixing code is a cost, and has to be balanced by making the product 'better' - quantified by the reduction in the number of issues.

SQLite is open source - you can download the source code. Fix all the issues and release your own fork. FixedSQLite. See how many people download your fixed version compared to the official version. Maybe then you'd have a stronger argument for the official code to then implement your changes. You can also quantify that by saying, 'it only took me ten years to fix the code. With two developers you have, it should only take you five years'. Win win.

Sure - if you can prove that fixing the unused/ignored return values improves the product by fixing one or more bugs ..

Secure programming guidelines provide incentives in such directions, don't they?

SQLite is open source - …

Some desirable source code transformations will need advanced development tools.

(Some of them are freely available already.)

How challenging will it become to increase support for corresponding software dependencies?

Is such a development view in conflict with secure programming guidelines?

Not according to the consortium members . Note that not all members are listed there.

We can assume SQLite is used in civil and military aerospace products¹, which are industries that maintain high standards, and I am sure Microsoft, Apple, and Alphabet, to name just a few, only accept it in their products after thorough evaluation.

SQLite relies on a test harness that covers 100% of the production runtime code paths, and on fuzz tests, rather than on static source code analysis.

1. ^{^} they usually will not talk about it in public

Is such a development view in conflict with secure programming guidelines?

Not according to the consortium members.

I propose to reconsider this view in more detail.

SQLite relies on a test harness …, rather than on static source code analysis.

It seems that there are some target conflicts involved.

I propose to reconsider this view in more detail.

You seem to be missing point A. Unsure if by accident or on purpose. I'm guessing the latter.

If bug #78 occurs and affects X number of users (with X being a number > 1) then bug #78 would get fixed.

If you're proposing changing a block of code which neither fixes bug #78 nor affects X number of users (with X being a number >1) then the proposed change of the aforementioned block of code is unnecessary. It may be /desired/, but code changes are weighted in terms of necessity.

You can propose all you like. I propose all code is written in Windows Notepad while standing on one leg and drinking out of a straw. I doubt the end result would be a binary that would be 'better' than one where that didn't occur.

I'm guessing the latter.

• How many users can you imagine who would be affected by return value ignorance in questionable ways?
• Do you still expect to get defined software behaviour even after overlooked function call failures?

Amazing. This reminds me of a conversation with GPT3.

The Turing test, originally called the imitation game by Alan Turing in 1950, is a test of a machine's ability to exhibit intelligent behaviour equivalent to, or indistinguishable from, that of a human.

Turing proposed that a human evaluator would judge natural language conversations between a human and a machine designed to generate human-like responses. The evaluator would be aware that one of the two partners in conversation was a machine, and all participants would be separated from one another. The conversation would be limited to a text-only channel, such as this forum. The evaluator would then guess which of the participants is the machine, and if the evaluator cannot reliably pinpoint the machine (i.e close to 50% accuracy), then such a machine would be deemed "intelligent" or having natural intelligence. A higher accuracy would indicate the machine is not reliably fooling the evaluator, and not to be deemed as intelligent.

I propose a reverse Turing test - One with more or less the same set-up, but with the difference that if the evaluator judges consistently wrong, i.e. it always mistakes the human for a machine (close to 0% accuracy) due to less human-like conversational skills or intelligence, then such a human can be deemed "unintelligent" or not exhibiting natural intelligence.

Disclaimer: In case anyone thinks I'm suggesting such of the OP, not at all: The conversation merely prompted me to imagine such a test. I'm pretty sure the OP could easily force a 50% outcome.

Badum tish 🤣

Given the virtually identical structures of each post and their quantity of open-ended questions I was also wondering if these threads were created by AI.

also wondering if these threads were created by AI.

What a time to be alive!