SQLite Forum

Strange CLI .exit issue
Login

Strange CLI .exit issue

(1) By anonymous on 2021-03-19 17:39:08 [link] [source]

(This is under Win7. It may or may not affect other platforms.)

When reading a script from a pipe and the script contains .exit followed by many [1] lines, an error is issued like:

c:\temp>sample.sql
The process tried to write to a nonexistent pipe.

The setup (from administrator console) is:

ftype sqlite3=cmd /c "type %1 | c:\bins\sqlite3.exe -batch %*"
assoc .sql=sqlite3

I'm almost certain this worked forever but I can't tell when it stopped working as I only keep the latest (3.35.2) plus the previous release, and it's present in both.

Has something changed lately with respect to stdin processing?

I'm not sure it's SQLite3's fault but given that Win7 is no longer updated I doubt there is a change Win7.

[1] How many is 'many'? I see the problem with about 120 lines following .exit while fewer lines make it go away (possibly this is related to the file buffer used by ether Windows or SQLite).

(2) By anonymous on 2021-03-22 21:06:17 in reply to 1 [link] [source]

(OP here) I seem to have solved it this way:

--- src/shell.c.in
+++ src/shell.c.in
@@ -7845,10 +7845,11 @@
       rc = 1;
     }
   }else

   if( c=='e' && strncmp(azArg[0], "exit", n)==0 ){
+    while (fgetc(p->in) != EOF);
     if( nArg>1 && (rc = (int)integerValue(azArg[1]))!=0 ) exit(rc);
     rc = 2;
   }else

The line numbers correspond to my modified version so they may not match official source. But it's in that vicinity.

As I have no knowledge of SQLite3 internals, can someone verify if this seems correct solution?

(3) By Larry Brasfield (larrybr) on 2021-03-22 23:06:41 in reply to 2 [link] [source]

Earlier, you wrote:

ftype sqlite3=cmd /c "type %1 c:\bins\sqlite3.exe -batch %*"

If you entered that command, literally, the %1 and %* will expand to nothing as seen by ftype. I'm going to pretend below that you escaped the % characters.

When the system shell expands this for the extension you have associated with "sqlite" files, for the command line "sample.sql" the result will be: cmd /c type sample.sql | c:\bins\sqlite3.exe -batch sample.sql , which you must admit is strange. As your sqlite3 invocation is written, it will try to open sample.sql as a database, and as the whole pipeline is written/read, the type command will be trying to copy its content to stdout which is piped to sqlite3's stdin. I'm not sure what should happen in this case, but I am far from declaring that what you have reported as happening should not happen and represents a fixable bug in the SQLite3 shell.

Given that I do not see a bug, I see the correct solution as "do nothing". And because your solution is not "do nothing", I suspect it is incorrect.

If you could show a shell usage scenario for which useful behavior should be expected from the ordinary, understandable rules of cmd.exe [a] but cannot happen due to sqlite3.exe's behavior, we may have a bona fide bug. But I'm not there yet.


a. That's not entirely a joke. There are some commonly understood rules, and the interaction of a file being both opened by "type" and opened by another process for exclusive access is not among that rule set.

(4) By Keith Medcalf (kmedcalf) on 2021-03-22 23:32:18 in reply to 3 [link] [source]

Try typing the following command:

help ftype

at a command prompt near you. You will note that %0 and %1 are the object-name (the command filename) and %* are all the parameters to the command-file.

(7.1) By Keith Medcalf (kmedcalf) on 2021-03-23 00:28:02 edited from 7.0 in reply to 4 [link] [source]

Note that for austerity %L is also the object-name (command filename) because some people cannot tell the difference between %1 and %l (they use silly fonts or need a new spectacle prescription).

(10) By Larry Brasfield (larrybr) on 2021-03-23 00:40:05 in reply to 7.1 [link] [source]

You may consider me now somewhat more educated than before. Thanks.

There are still people around who used typewriters that have no '1' key left of the '2' key because the 'l' (pronounced "el") key was routinely used to produce the smallest non-zero numeral, and nobody saw the folly of it!

(11) By Rowan Worth (sqweek) on 2021-03-23 03:33:23 in reply to 10 [link] [source]

ITYM and nobody saw the fo11y of it!

(5.1) By Keith Medcalf (kmedcalf) on 2021-03-22 23:55:53 edited from 5.0 in reply to 2 [link] [source]

Do you get the same result if you do not use the pipe but merely attach the file to stdin?

ftype sqlite3=cmd.exe /c ""C:\Apps\NTUtils\sqlite.exe" -batch < "%1" %*"

Change the specifier for the executable to use your executable. Also, parsing the redirectors is a shell function, so you must have the shell parse the command line -- you cannot start the executable directly.

(6) By Keith Medcalf (kmedcalf) on 2021-03-22 23:41:31 in reply to 5.0 [link] [source]

Note that if I use your pipe then I get an error too on Windows 10 (and probably every other version of DOS / OS/2 / OS/2 New Technology (aka NT) to have ever existed.

(8) By Keith Medcalf (kmedcalf) on 2021-03-22 23:55:03 in reply to 6 [link] [source]

Modify that because DOS (and other single tasking) Operating Systems process the pipe operator (|) as running the command before the pipe to completion and redirecting stdout to a temp file, then run the command after the pipe with stdin attached to the temp file.

Only Operating Systems which process pipes (|) by running the two processes in parallel with stdout of the first attached to stdin of the latter will experience the first process thowing an error if the consumer closes the pipe before the producer is done producing. Some common examples that this applies to are some versions of Concurrrent DOS, OS/2, OS/2 New Technology (aka Windows NT and its derivatives up to and including Windows 10), Linux, and the Unix and unix-like systems, including Mac OS X (but not MacOS), and QNX.

(9) By anonymous on 2021-03-23 00:20:31 in reply to 2 [link] [source]

The problem I reported is both accurate and real. The setup using FTYPE is also 100% correct as is. It expands to a new CMD running the command:

type sample.sql | c:\bins\sqlite3 --batch

I won't dignify bad-tempered/rude, and above all WRONG answers with a reply. (This is, unfortunately, a recurrent theme in this forum by various members.)

The proposed change I posted I have already tried and it appears to have solved the problem in my limited testing. I'm just not sure if it's correct in the sense it does not have ill side-effects as I don't know SQLite3 source.

What it does (or rather, should do, if there are no side bugs), is simply consume all chars until end-of-file upon .exit command.

I did not ask for an opinion on the correctness of my setup, only of the 'fix'.

If one can't see the problem in the original - that TYPE feeds SQLite3 with data after SQLite3 has stopped running - I can't help. It's not SQLite3's problem per se (so technically not a bug) as Windows could stop feeding a dead process data, but if it can be solved from within SQLite3 very simply, why not?

(12) By Larry Brasfield (larrybr) on 2021-03-23 08:11:01 in reply to 9 [link] [source]

The problem I reported is both accurate and real. The setup using FTYPE is also 100% correct as is. It expands to a new CMD running the command: type sample.sql | c:binssqlite3 --batch

I can see, from running a cmd.exe session (which I take to be your "administrator console"), that it runs ftype commands differently from most others, in that it does not expand %-preceded words. That's not surprising because cmd.exe is chock-full of quirky exceptions like that. I suppose I should have known, and so I will cop to an error on my absence of quoting comment. I do not accept that that comment was rude, however.

I also accept Keith's corrective point and explanation that argument expansion by the Windows GUI shell is done in a certain way, different from cmd.exe's interpretation of the same argument expressions. I will cop to the additional error of supposing that those expansions occur the same way. However, I do not accept that my erroneous statement of supposed expansion was rude.

... it appears to have solved the problem ... it ... consume all chars until end-of-file upon .exit command.

I did not ask for an opinion on the correctness of my setup, only of the 'fix'.

I stand by my conclusion that there is nothing to fix. The broken pipe message is not something to be avoided by changing potential consumers to read everything in the stream. The message normally indicates either that there has been a mismatch between what a stream producer produces and what a stream consumer expects, or that the consumer terminated prematurely without consuming what was expected to be consumed. The detection of a "broken pipe" and what to say about it are the responsibility of the invoking shell that setup the "pipe" connections. It would be overly presumptuous, IMO, for a consumer to preclude such reports by consuming all input regardless of its content. While some users may dislike seeing broken pipe reports, others value them as indicators that something unexpected is happening.

In this case, the SQLite shell consumes its standard input up to and including the .quit or .exit meta-command and its terminating newline. It should do no more and no less for valid input. It would be a disfavor to those who (arrange to) pipe input into the SQLite shell to deprive them of the useful diagnostic that there is extra garbage in the pipeline. To those who disagree, I would ask what should be the result of doing this on a *Nix system: cat myValidSQLiteShellInput /dev/zero | sqlite3 -batch ?

(13) By anonymous on 2021-03-23 08:59:44 in reply to 12 [link] [source]

It would be overly presumptuous, IMO, for a consumer to preclude such reports by consuming all input regardless of its content.

Except that upon .exit there is a guarantee no more input is needed.

While some users may dislike seeing broken pipe reports, others value them as indicators that something unexpected is happening.

The problem also appears from within editors, and it's more serious than just an annoying (possibly useful to some) message. The difference in that case is the editor (PNotepad to be specific for my case) just hangs. And this is very annoying.

In any case, I don't propose SQLite3 to 'fix' this in the official source.

I just care to fix my own copy that gives me trouble.

And to that end, I still don't have an opinion on whether the code I showed seems to be correct in that it does not cause other problems.

(14) By anonymous on 2021-03-23 09:10:23 in reply to 13 [link] [source]

If a program writes to a pipe, it has to watch out for a broken pipe signal (i.e. if the next program in the pipe terminates) and stop writing in this case.

It is the "type" program and the editors that show buggy behaviour and need to be fixed.

(15) By Larry Brasfield (larrybr) on 2021-03-23 09:16:17 in reply to 13 [link] [source]

I see only two improvable aspects to your "fix".

One is that for never-ending stream producers, which keep their output open and keep writing to it sometimes instead of closing it, your fixed shell will never terminate. Whether that is correct depends on expectation, which is a bit vague now.

The other is that the same treatment would logically apply to the .quit meta-command. Unless, of course, that is to differ for some reason.

I cannot see any other problems your "fix" creates, beyond the issue with defeating broken pipe detection already mentioned.

(16) By Keith Medcalf (kmedcalf) on 2021-03-23 09:21:40 in reply to 13 [source]

In the particular case you have described it will solve the issue and not create other problems for that particular case.

How it will behave when some other thing opens uses popen (for example) to sqlite3.exe depends on that other thing and its expectations.

(19) By anonymous on 2021-03-23 11:10:01 in reply to 16 [link] [source]

Thank you all

(17) By TripeHound on 2021-03-23 11:07:49 in reply to 13 [link] [source]

A possibly pertinent question is why you sometimes have 120 lines or more in your script file after the .exit command?

On the face of it, if you have a script to some program, and that program stops consuming the script, then "something has gone wrong", and it doesn't seem unreasonable to indicate an error. In your case, it seems you have "a script + something else" (the stuff after the .exit)... my first instinct (without knowing more, hence the question) is to say the error is in the script for having (seemingly) superfluous lines in it.

(20) By anonymous on 2021-03-23 11:12:29 in reply to 17 [link] [source]

... why you sometimes have 120 lines or more in your script file after the .exit command?

Because I use .exit for debugging.

During debugging I may want to exit after the part of interest (e.g. some SELECT) without running additional code that possibly also produces unwanted tons of output.

(18) By anonymous on 2021-03-23 11:09:50 in reply to 12 [link] [source]

While some users may dislike seeing broken pipe reports, others value them as indicators that something unexpected is happening.

And that is exactly why this should be fixed. If people get used to broken pipe errors caused by something like this, they will eventually ignore the real errors.

(23) By anonymous on 2021-03-24 13:13:23 in reply to 2 [link] [source]

(OP here) Sadly, the previous 'fix' caused a crash when .exit is given interactively instead of being read from a file (either redirected or by .read command).

An improved possibility is probably this:

--- src/shell.c.in
+++ src/shell.c.in
@@ -7845,11 +7845,12 @@
       rc = 1;
     }
   }else

   if( c=='e' && strncmp(azArg[0], "exit", n)==0 ){
+    if( !stdin_is_interactive && p->in==stdin )
+      while (fgetc(p->in) != EOF);
     if( nArg>1 && (rc = (int)integerValue(azArg[1]))!=0 ) exit(rc);
     rc = 2;
   }else

... always pending review from eager SQLite3 insiders ...

(24) By Larry Brasfield (larrybr) on 2021-03-24 14:28:04 in reply to 23 [link] [source]

It was not a "crash". It was a "read stdin to end-of-stream." That condition can be effected by typing control-Z at the beginning of a line on Windows.

No comment on checking for stdin_is_interactive, except this: I dislike programs that act differently depending on whether input or output is redirected. It complicates the mental model, and compromises modularity.

(25) By anonymous on 2021-03-24 15:01:14 in reply to 24 [link] [source]

Actually, it appeared like a "crash"; a Windows popup saying the program has stopped working, and you can no longer interact with the program.

As for using stdin_is_interactive I found it inside sqlite3 source so I suppose if it's already used for something else why not here too.

(26) By Larry Brasfield (larrybr) on 2021-03-24 17:44:18 in reply to 25 [link] [source]

The code you showed earlier, which read whatever stream was known as stdin until seeing EOF from it, had the behavior I stated and should not have crashed. You claimed that it solved your problem, implying that it read stdin up to getting EOF rather than "crashing". I'm just pointing out that such input sucking is going to go on for a long time when stdin is the console input, presumably coming from a typing person, until they indicate end-of-stream as I indicated.

Of course, I cannot speak to "crashing" for code you have not shown.

My preference for programs acting the same when fed by redirected input as when typed at comes from seeing the complications that ensue and increased trouble-shooting inconvenience. You should suit yourself on that, of course. Once you have many parts of a system acting differently according to how they are connected to each other, and find it difficult to debug and reason about, you may appreciate the point.

(21.1) By Richard Hipp (drh) on 2021-03-23 12:30:29 edited from 21.0 in reply to 1 [link] [source]

This is as it should be.

Can you give an example of any other utility or program that continues reading its input after instructed to exit? I can't find one. The most obvious case of a program that does this is the the "head" utility. It also simply stops reading its input and exits once it acquires the requested number of lines. Upstream programs that are concerned about broken pipes will complain. I cannot come up with any other program that behaves differently. Why should SQLite be any different?

What if I have a 1MB script in which ".exit" occurs on the 5th line. You would have SQLite read the entire 1MB just in case the script was piped in? I don't think so.

The ".exit" command means "stop reading input and exit". You are asking to redefine ".exit" to mean "read and ignore all subsequent input and then quit". That is a very different thing, and something I do not want to support.

If you really need to avoid broken pipe errors, I suggest you write a new utility program that does that for you. Here is a TCL script called "shunt.tcl" that transfers content from input to output, but if the output pipe closes, it continues reading the input to avoid complaints from the upstream producers:

while {![eof stdin]} {
  if {[catch {puts [gets stdin]; flush stdout}]} {
    while {![eof stdin]} {gets stdin}
  }
}

With this program you could do:

ftype sqlite3=cmd /c "type %1 | tclsh shunt.tcl | c:\bins\sqlite3.exe -batch %*"

The shunt.tcl script will nicely catch and suppress the broken pipe error for you. That would be the proper solution to this problem.

The sqlite3.exe program is currently working as it ought with respect to the .exit command and reading standard input.

(22) By anonymous on 2021-03-23 13:16:13 in reply to 21.1 [link] [source]

The sqlite3.exe program is currently working as it ought with respect to the .exit command and reading standard input.

Let me make clear once again, I never asked for the change to be part of the official SQLite3.

Initially, I was curious how come I never encountered it before (same workflow for years) so maybe something had changed in the CLI, and then if my found solution for my own deviant copy seems correct given I have no clue about SQLite3 source internals.

So, I simply asked if someone could see a problem with the way I did it. That's all.

Can you give an example of any other utility or program that continues reading its input after instructed to exit? I can't find one.

I already mentioned that my editor (PNotepad using just sqlite3.exe --batch) hangs. Again, this could be a problem with that editor.

What if I have a 1MB script in which ".exit" occurs on the 5th line. You would have SQLite read the entire 1MB just in case the script was piped in? I don't think so.

Sure. Only if one could know when the stdin is redirected, then it could consume only for that case. Again, that's only for own use, so pay no more attention to it.

On the other hand, you could think of all content that follows as wrapped inside a false #ifdef ... #endif block. I believe a compiler would read all the way to find the matching #endif. So, it's a matter of perspective.