SQLite User Forum

sqlite3_open_v2 not failing for non-SQLite file.
Login

sqlite3_open_v2 not failing for non-SQLite file.

(1.2) By ddevienne on 2022-02-08 18:48:29 edited from 1.1 [source]

The code below returns true for a non-SQLite file...

I've double-checked the file does not even start with the SQLite magic cookie (using od -c on Linux).
Calling sqlite3_open_v2() is not enough to ascertain the SQLite-ness of a file?
Do I need to issue some pragmas too? This one caught me off guard!

I can of course read the 100-bytes header, and do my own checks,
but what's the official (and fastest!) way to check whether a file is an SQLite DB, using the SQLite API?
And what does sqlite3_open_v2 really do? Just an fopen() equivalent?

Thanks, --DD

PS: SQLite 3.36

bool isSQLiteDB(const fs::path& path) {
    assert(fs::is_regular_file(path));
    auto filename = path.string();

    sqlite3* db = nullptr;
    int rc = sqlite3_open_v2(filename.c_str(), &db, SQLITE_OPEN_READONLY, nullptr);
    if (rc != SQLITE_OK) {
        sqlite3_close(db); // must close even on error
        return false;
    }
    sqlite3_close(db);
    return true;
}

Here's the file in question:
(I've replaced the real ASCII magic cookie with F I L E to anonymize the file)

[ddevienne@marsu files]$ ll some.file
-rw-r--r-- 1 ddevienne users 196,608 May 17  2015 some.file

[ddevienne@marsu wells]$ od -c some.file | more
0000000   #       F   I   L   E       4       :      \n  \0  \0  \0  \0
0000020  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0
*
0000120  \0  \0  \0 001  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \v   |
0000140  \0  \0  \0  \0  \0 002 300  \0  \0  \0   @  \0
0000160
*
0000240                                                  \0  \0  \0  \0
...

(2) By ddevienne on 2022-02-08 19:04:37 in reply to 1.2 [link] [source]

If I change the last two lines of code to read:

    rc = sqlite3_exec(db, "select rowid from sqlite_master limit 1", nullptr, nullptr, nullptr);
    sqlite3_close(db);
    return rc == SQLITE_OK;

Then it is seems more reliable, except my runtime just doubled!
I'm testing about 5'500 files for SQLite-ness (among other things).

I shouldn't have to run a query to reliably ascertain a file is an SQLite DB, no?
That's the point of magic cookies after all. Can I avoid the query engine and rely solely on the 100-bytes header somehow?

(3.1) By Larry Brasfield (larrybr) on 2022-02-08 19:35:35 edited from 3.0 in reply to 2 [link] [source]

The 16 byte magic at the file's beginning is very stable for SQLite 3.

Before you become too dissatisfied with your doubled runtime, I advise timing your "select ... sqlite_master" with a simple prepare. If you try that with a recent version of the CLI, you will get an error such as: Error: near line 1: in prepare, file is not a database (26) , which shows that no execution is necessary (or possible.)

Then, before retaining your dissatisfaction, compare that new runtime with what is needed to actually open the file and read in the first 16 bytes. I'm pretty sure that the comparison will be nowhere near as shocking as "doubled".

Edited to answer the middle question.

I shouldn't have to run a query to reliably ascertain a file is an SQLite DB, no?

No. But you need to prepare to run one (or find a pragma which requires consulting the DB header.) The library defers actual reading of the DB until it is needed. This saves some time for normal usage patterns.

(4) By ddevienne on 2022-02-08 20:13:44 in reply to 3.1 [link] [source]

Well, there's plenty of SQLITE_OPEN_... flags already.
I really wish there was one that forced checking the 100-bytes header for correctness.
E.g. SQLITE_OPEN_READ_AND_CHECK_HEADER.

That way, it remains opt-in and backwards compatible.
And establishes a clear best practice for SQLite-ness of a file going forward.

That way no need for some work-around to indirectly reading that header. My $0.02.

(5) By Aloys (aloys) on 2022-02-08 20:47:09 in reply to 4 [link] [source]

I really wish there was one that forced checking the 100-bytes header for correctness. E.g. SQLITE_OPEN_READ_AND_CHECK_HEADER.

That would only work for plain, unencrypted SQLite databases. However, there are several SQLite encryption extensions out there like SEE (SQLite Encryption Extension) provided by the SQLite developers or SQLCipher provided by the Zetetic company. For an encrypted SQLite database file it is usually impossible to identify the file as a SQLite database file by just looking at the database header. Therefore such an Open flag wouldn't make much sense in the general case, since even if the test fails the file could still be a valid SQLite database file.

(10) By anonymous on 2022-02-13 19:12:46 in reply to 5 [link] [source]

I would think that the SQLITE_OPEN_READ_AND_CHECK_HEADER shouldn't be expected to work for encrypted databases anyways, unless the VFS decrypts it at the time that it is being opened. Since, if you are using this flag then it would be that you could not use such a database file anyways.

(6) By Bill Wade (billwade) on 2022-02-09 13:27:29 in reply to 2 [link] [source]

I'm pretty sure you know this, but for the benefit of others:

This select statement will likely succeed if the file is empty.

In that sense, the file is a valid sqlite file, even without the header.

Of course it is also a valid (if not very exciting) script for many scripting languages including sh, python, and sqlite3.

So depending on what you want out of your test, you may want to add special handling for empty files.

(7) By Larry Brasfield (larrybr) on 2022-02-09 13:53:09 in reply to 6 [link] [source]

This select statement will likely succeed if the file is empty.

It will not succeed in the sense of surviving a sqlite3_prepare().

In that sense, the file is a valid sqlite file, even without the header.

Impossible. The header is written along with all other per-database info. Until the first page is written, there is no way to call the file a valid SQLite3 database.

So depending on what you want out of your test, you may want to add special handling for empty files.

It suffices to either verify presence of the 16-byte header or call sqlite3_prepare_v2() with "SELECT * FROM sqlite_schema" and see if that succeeds. Of course, a stat() call could be used for empty files, by why special-case such rarities?

(8) By ddevienne on 2022-02-09 14:02:47 in reply to 7 [link] [source]

It will not succeed in the sense of surviving a sqlite3_prepare().

Actually, even sqlite3_prepare() for a simple "pragma user_version" succeeds...
One must actually sqlite3_step() the statement to get an error.

FWIW, here's the code I ended up using, which is faster than the previous one using sqlite_master.
I still think it's a shame to have to parse SQL and involve the SQL and perhaps VDBE engine just
to check the header of a file to ascertain whether it is an SQLite DB or not.
There should be an easier and official way to do this IMHO...

PS: The point about encrypted DBs is moot and beside the point IMHO.

bool isSQLiteDB(const fs::path& path, int* p_user_version = nullptr) {
    assert(fs::is_regular_file(path));
    auto filename = path.string();

    sqlite3* db = nullptr;
    int errc = sqlite3_open_v2(filename.c_str(), &db, SQLITE_OPEN_READONLY, nullptr);
    if (errc != SQLITE_OK) {
        sqlite3_close(db); // must close even on error
        return false;
    }
    sqlite3_stmt* stmt = nullptr;
    const char* sql_end = nullptr;
    constexpr char sql[] = "pragma user_version";
    errc = sqlite3_prepare_v2(db, sql, sizeof(sql) - 1, &stmt, &sql_end);
    if ((errc == SQLITE_OK) && stmt) {
        assert((sql_end - sql) == sizeof(sql) - 1);
        if (SQLITE_ROW != sqlite3_step(stmt)) {
            errc = SQLITE_NOTADB;
        } else if (p_user_version) {
            *p_user_version = sqlite3_column_int(stmt, 0);
        }
        sqlite3_finalize(stmt);
    } else {
        errc = SQLITE_NOTADB;
    }
    sqlite3_close(db);
    return errc == SQLITE_OK;
}

(9) By Bill Wade (billwade) on 2022-02-10 15:38:26 in reply to 7 [link] [source]

It succeeds in the sense that the sqlite3 command line doesn't report an error.

$ sqlite3 nosuchfile.db
SQLite version 3.35.2 2021-03-17 19:07:21
Enter ".help" for usage hints.
sqlite> select rowid from sqlite_master limit 1;
sqlite> select rowid from no_such_table limit 1;
Error: no such table: no_such_table
sqlite> .exit

No error was reported for the first select statement.

$ ls -s nosuchfile.db
0 nosuchfile.db

The file is empty, but at least the command line tool is willing to run queries against sqlite_master.