SQLite Forum

Debugging opcodes in vdbe.c
Login
Note you can also use the `stdbuf` tool in the shell to control stdio buffering within pipelines. The default behaviour depends on the tool, but for those which use glibc's stdio (`FILE*` and friends) it goes like this:

(a) if stdout is a tty, then output is buffered on a line-by-line basis
(b) otherwise, output is buffered in fixed size chunks (IIRC the default is 1MB)

Which is why things can change when you tack `| tee` on the end of a command vs. having it output straight to the console. In this case you can prepend `stdbuf -oL` to whichever command is buffering to force stdout back to line buffered.

For a quick demo, start with a simple loop emitting one message per second and log timestamps when they are received:

```
$ for i in $(seq 1 3); do echo $i; sleep 1; done | awk '{print strftime("%F %T"), $1}'
2020-09-25 16:48:13 1
2020-09-25 16:48:14 2
2020-09-25 16:48:15 3
```

Now add a filter for odd numbers: `sed -n '/[13579]$/p'`:

```
$ for i in $(seq 1 3); do echo $i; sleep 1; done | sed -n '/[13579]$/p' | awk '{print strftime("%F %T"), $1}'
2020-09-25 16:49:18 1
2020-09-25 16:49:18 3
```

Note nothing is printed for three seconds, and then both lines have the same timestamp. sed is buffering the output -- rude. stdbuf to the rescue:

```
$ for i in $(seq 1 3); do echo $i; sleep 1; done | stdbuf -oL sed -n '/[13579]$/p' | awk '{print strftime("%F %T"), $1}'
2020-09-25 16:50:10 1
2020-09-25 16:50:12 3
```


It's done this way because block buffering enables better pipeline performance compared to line buffering, but it's such a subtle thing to have tools changing behaviour like this that I feel GNU userspace would be simplified so much if block buffering was strictly opt-in...