SQLite Forum

Timeline
Login

12 forum posts by user pooryorick

2021-09-14
14:58 Reply: segmentation fault when closing a database from within a transaction (artifact: be303c3704 user: pooryorick)

The transaction context be detected and finalized in the process of closing the database.

13:43 Post: segmentation fault when closing a database from within a transaction (artifact: 1a0cce8048 user: pooryorick)

The following Tcl script script triggers a segmentation fault when run with a debugging build of sqlite and a debugging build of Tcl:

package require sqlite3
sqlite3 db :memory:
db transaction {db close}

-- Poor Yorick

2021-08-03
18:01 Reply: null character sorts greater than 0x7f and less than 0x80 (artifact: 1db321e0b4 user: pooryorick)

Yes, I did.

10:27 Reply: null character sorts greater than 0x7f and less than 0x80 (artifact: ae1ad899bb user: pooryorick)

I don't insist on anything, and the Tcl script in question is not doing anything other than using the interface distributed along with SQLite to access a database and store values using variable substitutions performed by SQLite. I'm working in a language where every value is a Unicode string, and am storing these strings the database. I'd like these stored values to be useful regardless of which SQLite interface is used on which platform to access the database.

Neither the the SQLite documentation for the Tcl interface nor the SQlite documentation itself provides any clue that some Tcl internal non-standard utf representation is what ends up stored as "text", and there's a reason for that: It's a bug. The issue here is that the SQLite Tcl interface is not doing its job. It is the component that is in the position to translate the Tcl internal representation of a value into standard utf-8. The Tcl C API provides routines to do that.

If text strings in SQLite may not include the null character, that should be clearly documented, and the definition of "text" that is not true should be replaced with one that is.

08:40 Reply: null character sorts greater than 0x7f and less than 0x80 (artifact: 052eedcebe user: pooryorick)

Where is this policy of accepting an arbitrary sequence of bytes for the "text" storage class stated? Doesn't it contradict SQLite's own documentation, which provides the following definition of the "text" storage class?

Text. The value is a text string, stored using the database encoding (UTF-8, UTF-16BE or UTF-16LE)

Isn't the whole point of the "blob" storage class to have a place to hold values that aren't valid for the "text" class? Wouldn't the appropriate interpretation of a value that is not valid in the encoding of the database then be "blob"? Why even have a "text" storage class if that's not what it is? Alternatively, why have a "blob" storage class if "text" accepts arbitrary values?

So SQLite doesn't allow the null character in a text value? If so, that means SQLite is not capable of storing all possible Unicode strings. I had supposed this was one of the big advantages of SQLite over, i.e. PostgreSQL.

2021-08-02
12:16 Edit reply: null character sorts greater than 0x7f and less than 0x80 (artifact: 0fd8c9e424 user: pooryorick)

Until this issue is fixed, data stored in an SQLite database whose encoding is ostensibly utf-8 is not necessarily actually valid utf-8. Rather, a value stored in the data base may be in Tcl's own internal representation. This will work mostly fine (except for the sorting issue) as long as the database is only accessed through the Tcl interface, but other systems accessing the database without the Tcl interface will get munged results.

11:55 Reply: null character sorts greater than 0x7f and less than 0x80 (artifact: 9375abf500 user: pooryorick)

Until this issue is fixed, data stored in an SQLIte database whose encoding is ostensibly utf-8 is not necessarily actually valid utf-8. Rather, a value stored in the data base may be in Tcl's own internal representation. This will work mostly fine (except for the sorting issue) as long as the database is only accessed through the Tcl interface, but other systems accessing the database without the Tcl interface will get munged results.

2021-06-04
12:33 Post: conversion of literal values, or lack thereof (artifact: 0ff4ede3e1 user: pooryorick)

In the following example, the literal value '1' is not converted to a numeric value in the first case, but in the second and third cases, it is:

create table t1(c1 integer)
; insert into t1(c1) values (1)

;select 1 = '1'
, cast(1 as numeric) = '1'
, (select c1 from t1 where c1 = 1) = '1'

Output:

0 1 1

Why not make that more consistent by converting in the first case as well?

2020-08-02
20:06 Reply: null character sorts greater than 0x7f and less than 0x80 (artifact: ebe94073a1 user: pooryorick)

In a nutshell, tclsqlite.c should use Tcl_UtfToExternal on incoming values to convert the value from the Tcl internal utf-8 form to standard utf-8.

2020-08-01
08:19 Post: null character sorts greater than 0x7f and less than 0x80 (artifact: 720bcd2b43 user: pooryorick)

The output of the following script is 127 0 128 rather than the expected 0 127 128

#! /usr/bin/env tclsh
package require sqlite3
sqlite db :memory:

db eval {
    create table t (
        v text
    )
}

foreach val {
    \x80
    \x7f
    \x00
} {
    string range $val 0 end
    #puts [::tcl::unsupported::representation $val]
    db eval {
        insert into t values ($val)
    }
}

set sorted {}
db eval {
    select v from t order by v collate binary asc 
} {
    lappend list $v
    scan $v %c ord
    lappend sorted $ord
}

puts $sorted

The correct output would be 0 127 128, woulddn't it?

-- Yorick

2020-04-09
21:43 Reply: Tcl coroutine, db eval ... script, yield, segmentation fault (artifact: 4c9ba4f8a0 user: pooryorick)

This is on trunk. There has been some work in the last few years on NRE-enabling the Tcl binding.

There is also a related invalid memory issue when deleting a coroutine that is in the middle of a sqlite transaction. Any fix for this issue is likely to fix that as well.

21:15 Post: Tcl coroutine, db eval ... script, yield, segmentation fault (artifact: 4638e41470 user: pooryorick)

The following script produces a segmentation fault:

#! /usr/bin/env tclsh

package require sqlite3
proc p {} {
    set statement {select * from t}
    db eval $statement {
        yield
    }
}
sqlite3 db :memory:
db eval {
    create table t (v)
    ; insert into t(v) values (1) ,(2)
}

coroutine p1 p 
p1
rename db {}
p1

With a Tcl debugging build the segmentation fault happens at a more descriptive spot. The SqliteDb structure is released then the database routine is deleted, but then later dbReleaseStmt() attempts to access that structure.