SQLite User Forum

Is there any way to get last non-null value of a column ?
Login

Is there any way to get last non-null value of a column ?

(1) By anonymous on 2022-02-07 07:21:35 [link] [source]

I have a column that containing null values. for example,
value
------
1
null
null
2
3
null
null

Now, I'd like to display this column and fill these null values with its previous non-null value. for example
value
------
1
1
2
3
3
3

I tried to use last_value() function, but it does not support ignore nulls keyword. How to use any existing build-in functions to accomplish this ?

(2.1) By Mark Lawrence (mark) on 2022-02-07 11:59:05 edited from 2.0 in reply to 1 [link] [source]

Here is my naive attempt:

create table t(
    a integer
);

insert into t values
    (1),
    (null),
    (null),
    (2),
    (3),
    (null),
    (null),
    (4),
    (null);

select
    t.rowid,
    t.a AS have,
    coalesce(t.a,max(t2.a)) AS want
from t
left join t t2
on
    t.a is null and
    t2.rowid < t.rowid and
    t2.a is not null
group by t.rowid
order by t.rowid;

Which outputs this:

rowid  have  wanted
-----  ----  ------
1      1     1
2      NULL  1
3      NULL  1
4      2     2
5      3     3
6      NULL  3
7      NULL  3
8      4     4
9      NULL  4

I'm sure there will be better ways of doing this. I also haven't put any thought into edge cases or assumptions.

(3) By anonymous on 2022-02-07 12:27:18 in reply to 2.1 [link] [source]

For each row, search backwards for a non-null value.

select rowid, a as have,
    (select t2.a from t t2
         where t2.rowid<=t1.rowid and t2.a is not null
         order by rowid desc) as wanted
    from t t1;

(4) By anonymous on 2022-02-07 14:40:08 in reply to 2.1 [link] [source]

Thank you for the solution.
It looks a litte bit complicated. What if the new row is inserted with the value of its previous row ? So I will not need to create a view to fill those null values.
Is it possible to specify the column's defaut value to be the last value of this column?
Or is there any other fast way to copy  the value of last row while inserting a new row except the primary unique key?

(5) By Gunter Hick (gunter_hick) on 2022-02-07 15:46:18 in reply to 4 [link] [source]

Note that the offered solution implies that ascending record numbers accurately reflect your notion of "previous". This may not hold for two reasons:
 a) the philosophical reason being that a table is a set without intrinsic order
 b) the practical reason being that implicit row numbers are neither immutable nor immune to being reused.

Of course your application may keep the values of fields it intends to be used in the above fashion (i.e. keep or increment the value) in its memory for re-use in the next insert. But this may not reflect the value last inserted into the table if more than one thread is doing the inserting.

The sqlite2_last_insert_rowid() function is equally (if not more) unpredictable if threads share a connection, since it is oblivious to the table the rowid was used on.

There is no DEFAULT LAST ROW clause. You would have to retrieve the value inside a trigger program. Or maybe INSERT ... IFNULL( ? , SELECT MAX( <field> ) FROM <table>.

(6) By Simon Slavin (slavin) on 2022-02-07 16:14:13 in reply to 1 [link] [source]

Can you fix this when the data is put in the table, instead of trying to fix it when the data is retrieved from the table ?

My reasons are the same ones that Gunter covered: table rows don't have an order. Rows in the table are a 'set' or a 'collection', not a 'sequence'. Referring to a 'previous' or 'last' row doesn't mean anything. According to the specifications for SQL, you can run the same SELECT twice and get two different orders.

(7) By Harald Hanche-Olsen (hanche) on 2022-02-07 18:19:53 in reply to 1 [link] [source]

You can solve it with a recursive CTE. Below is some sample code.

Here s is a sort key (you can use rowid if you must). Since the recursive CTE needs successive numbers, that is done in the first CTE, adding the numbers n for this purpose.

CREATE TABLE ta( s INTEGER, x INTEGER );

INSERT INTO ta VALUES
    (10,1), (12,null), (15,NULL),
    (20,2),
    (30,3), (33,null), (36,null), (38,null),
    (42,4), (44,null);

SELECT oid, * FROM ta;

WITH RECURSIVE
  tan (oid, n, x) AS (
    SELECT oid, row_number() OVER (ORDER BY s), x
    FROM ta),
  tbb (oid, n, o, x, xx) AS (
    VALUES(NULL,NULL,1,NULL,NULL)
    UNION ALL
    SELECT tan.oid, tan.n, tbb.o+1, tan.x,
      COALESCE(tan.x,tbb.xx)
    FROM tbb
    JOIN tan on(tan.n=tbb.o))
UPDATE ta SET x = xx FROM tbb WHERE ta.oid = tbb.oid;

SELECT oid, * FROM ta;

When this is run in sqlite, the output is as follows:

rowid  s   x
-----  --  -
1      10  1
2      12  ⟂
3      15  ⟂
4      20  2
5      30  3
6      33  ⟂
7      36  ⟂
8      38  ⟂
9      42  4
10     44  ⟂
rowid  s   x
-----  --  -
1      10  1
2      12  1
3      15  1
4      20  2
5      30  3
6      33  3
7      36  3
8      38  3
9      42  4
10     44  4

(8) By Marco Bubke (marcob) on 2022-02-07 19:05:44 in reply to 1 [link] [source]

Do you tried window function: https://www.sqlite.org/windowfunctions.html? Maybe something like first_value() is working if you find the right window.

(9) By anonymous on 2022-02-08 01:53:28 in reply to 6 [source]

Yes, I'm trying to find another way to fix null value when inserting rows to the table. 
Can I use _rowid_ to retrieve the latest added row ? e.g.

CREATE VIEW lastOne AS SELECT _rowid_,* FROM t WHERE _rowid_ = (SELECT MAX(_rowid_) - 1 FROM t) LIMIT 1

Then 

CREATE TRIGGER onInsert AFTER INSERT ON t
BEGIN
UPDATE t
SET value=lastOne.value
FROM lastOne
WHERE _rowid_=lastOne._rowid_+1;
END

For my application, the table is worked as a ring table, the records are inserted to the table one by one in a single thread.  After the row count reaches the maximum setting value, it will remove the oldest one and then insert a new one. 
So, I think if I do not execute VACCUM command, the _rowid_ will always keep its sequence.

(10) By Keith Medcalf (kmedcalf) on 2022-02-08 02:50:59 in reply to 9 [link] [source]

The following trigger will work more better. There is no point updating any row other than the one that was inserted, and then only if a NULL was provided. Also, it will not work if the last added record is deleted before the next is inserted. This will fix those problems.

CREATE TRIGGER onInsert AFTER INSERT ON t WHEN new.value IS NULL
BEGIN
   UPDATE t
      SET value = (
                     select value
                       from t
                      where _rowid_ < new._rowid_
                   order by _rowid_ desc
                      limit 1
                  )
    WHERE _rowid_ = new._rowid_;
   SELECT RAISE(ABORT, 'value cannot be null')
     FROM t
    WHERE _rowid_ == new._rowid_
      AND value IS NULL;
END;

(11) By Keith Medcalf (kmedcalf) on 2022-02-08 03:37:20 in reply to 10 [link] [source]

You could even put in a trigger like this (added the capability to look backwards until it finds a not null value) and after one cycle of your ring buffer all the data will be fixed.

CREATE TRIGGER onInsert AFTER INSERT ON t WHEN new.value IS NULL
BEGIN
   UPDATE t
      SET value = (
                     select value
                       from t
                      where _rowid_ < new._rowid_
                        and value IS NOT NULL
                   order by _rowid_ desc
                      limit 1
                  )
    WHERE _rowid_ = new._rowid_;
   SELECT RAISE(ABORT, 'value cannot be null')
     FROM t
    WHERE _rowid_ == new._rowid_
      AND value IS NULL;
END;