Null value returned by TOTAL
(1) By Jinsheng Ba (bajinsheng) on 2022-05-15 06:06:02 [link] [source]
With the +Inf and -Inf at the same column, the TOTAL aggregation function returns Null value.
CREATE TABLE t0(c0);
INSERT INTO t0 VALUES(1e900);
INSERT INTO t0 VALUES(-1e900);
INSERT INTO t0 VALUES(10);
SELECT TOTAL(c0) from t0;
-- Null
SELECT TOTAL(aggr) from (SELECT total(c0) as aggr from t0 where rowid<=2 UNION ALL SELECT total(c0) as aggr from t0 where rowid >=3);
-- 10.0
The first SELECT returns Null because of the +Inf and -Inf.
Based on the document:
The result of total() is always a floating point value.
I think the result is unexpected.
And also, the Null affect the result of the second SELECT, which should be equivalent to the first SELECT. The results are inconsistent.
Environment:
Version: 3.39.0
Commit ID: 2277f9ba
OS: Ubuntu 20.04
Configuration Options: ./configure --enable-all
Compiler: gcc-9
Client: CLI
(2) By anonymous on 2022-05-15 07:53:59 in reply to 1 [source]
SQLite always converts floating-point NaNs to SQL NULL values. This is clear from the source but might not be explicitly stated in the documentation.
(3) By Adrian Ho (lexfiend) on 2022-05-15 09:39:46 in reply to 1 [link] [source]
That's because you overflowed the Binary-64 range representation:
So-called "REAL" or floating point values are stored in the IEEE 754 Binary-64 format. This gives a range of positive values between approximately 1.7976931348623157e+308 and 4.9406564584124654e-324 with an equivalent range of negative values. A binary64 can also be 0.0 (and -0.0), positive and negative infinity and "NaN" or "Not-a-Number".
Hence, what you actually stored is:
sqlite> SELECT c0 FROM t0;
Inf
-Inf
10
Now, if I remember correctly, Inf + (-Inf)
is undefined under IEEE 754, and since the document you linked to also states:
The sum() and total() aggregate functions return the sum of all non-NULL values in the group.
SQLite's NULL return is arguably correct, since all three inputs are not NULL (nor, as another responder mentioned, NaN-degenerated-to-NULL), but a meaningful total cannot be computed. Returning 10
or 10.0
would be absolutely wrong.
If you don't overflow Binary-64, things work as expected:
sqlite> CREATE TABLE t1(c1);
sqlite> INSERT INTO t1 VALUES(1e308);
sqlite> INSERT INTO t1 VALUES(-1e308);
sqlite> INSERT INTO t1 VALUES(10);
sqlite> SELECT c1 FROM t1;
1.0e+308
-1.0e+308
10
sqlite> SELECT TOTAL(c1) FROM t1;
10.0
(5) By Jinsheng Ba (bajinsheng) on 2022-05-16 02:34:30 in reply to 3 [link] [source]
Thanks for your clarify!
I understand that a meaningful TOTAL cannot be computed, but is it possible to return a Inf as well? Otherwise, the second SELECT returns 10, which is also incorrect.
(7) By Adrian Ho (lexfiend) on 2022-05-17 14:13:02 in reply to 5 [link] [source]
is it possible to return a Inf as well?
That's up to Dr. Hipp, but why would Inf + (-Inf) + 10 = Inf
make any more sense than -Inf
, or NULL
, or 42.0
?
Otherwise, the second SELECT returns 10, which is also incorrect.
Every part of your second query performs exactly as documented, so I'm not sure what you mean by "incorrect":
sqlite> .nullvalue '!!!NULL!!!'
sqlite> SELECT c0 FROM t0 WHERE rowid <= 2;
Inf
-Inf
sqlite> SELECT total(c0) FROM t0 WHERE rowid <= 2;
!!!NULL!!!
sqlite> SELECT c0 FROM t0 WHERE rowid >= 3;
10.0
sqlite> SELECT total(c0) FROM t0 WHERE rowid >= 3;
10.0
sqlite> SELECT total(c0) AS aggr FROM t0 WHERE rowid <= 2 UNION ALL
SELECT total(c0) AS aggr FROM t0 WHERE rowid >= 3;
!!!NULL!!!
10.0
sqlite> SELECT total(aggr) FROM (
SELECT total(c0) AS aggr FROM t0 WHERE rowid <= 2 UNION ALL
SELECT total(c0) AS aggr FROM t0 WHERE rowid >= 3
);
10.0
(4) By Adrian Ho (lexfiend) on 2022-05-15 09:53:53 in reply to 1 [link] [source]
Also, the sentence you quoted needs to be understood in its context:
The result of total() is always a floating point value. The result of sum() is an integer value if all non-NULL inputs are integers. If any input to sum() is neither an integer nor a NULL, then sum() returns a floating point value which is an approximation of the mathematical sum.
In other words, "total() always returns a floating point" is meant to contrast with "sum() sometimes returns a float, and sometimes an integer".
(6) By Jinsheng Ba (bajinsheng) on 2022-05-16 02:39:30 in reply to 4 [link] [source]
Thanks!
It is indeed a bit confusing for me. I got no document on how to calculate Inf in the TOTAL/SUM.