sqlite doesnot save until close db
(1) By marcrew on 2022-11-08 02:16:29 [link] [source]
Hello I am beginner to sqlite. my development enviroment is : RTEMS5 + sqlite3.31.1. And my compile option is -DSQLITE_MAX_MMAP_SIZE=0 -DSQLITE_DEFAULT_LOCKING_MODE=1 -DSQLITE_ENABLE_MEMSYS5. and startup config is journal_mode = OFF; I use Unix-dotfile as VFS to operate sqlite. I can open db, create table and insert record to db. but the changes will not be made until I close the db. If I reset the whole system before close the db , everything will be lost, the db size is 0 bytes.
(2) By Simon Slavin (slavin) on 2022-11-08 05:58:19 in reply to 1 [source]
What makes you think that the changes are not saved ? Are you just looking at the database filesize ? Do you see a journal file being made in the same directory ? Does that file also have zero size ?
How are you using SQLite ? Are you making calls to the C API ? Or are you calling a SQLite library for a different programming language ?
Are you checking the values returned by your calls to CREATE TABLE and INSERT ? Are the equal to 0 (= SQLITE_OK) ?
(4) By marcrew on 2022-11-08 07:24:57 in reply to 2 [link] [source]
thanks Simon Slavin for reply. What makes you think that the changes are not saved ? - I found my test.db file size is 0 bytes. so I think the changes is not saved. but when I use Linux to test, the changes are affected immediatgely.(before I close the db), Are you just looking at the database filesize ? Do you see a journal file being made in the same directory ? Does that file also have zero size ? - Yes, there is journal file(0 bytes) in the diectory(if I switch PRAGMA journal_mode = OFF, it wouldn't be appear.) How are you using SQLite ? Are you making calls to the C API ? Or are you calling a SQLite library for a different programming language ? - I am using the C API sqlite3_exec Are you checking the values returned by your calls to CREATE TABLE and INSERT ? Are the equal to 0 (= SQLITE_OK) ? - I have checked the return value, it is SQLITE_OK.
(6) By Simon Slavin (slavin) on 2022-11-08 08:55:45 in reply to 4 [link] [source]
Your replies to my questions are all good replies. I see nothing wrong with what you are doing.
You have tested correctly in Linux. And in Linux this problem does not occur.
I think that RTEMS5 is causing this problem. I do not know anything about it. I hope someone else can help you. Good luck.
(8) By marcrew on 2022-11-17 01:47:45 in reply to 6 [link] [source]
hello Simon you are right, I found it is the reason of RTEMS. there is a cache between filesystem and harddisk. I have to use fsync to force data writing to disk after write method. My question is: is there any kind of flush or sync method in Sqlite I could use ?
(10) By Gunter Hick (gunter_hick) on 2022-11-17 07:01:20 in reply to 8 [link] [source]
See https://sqlite.org/forum/forumpost/b24023c75c84c596 Journal mode OFF disables COMMIT; COMMIT calls fsync() after writing the data. You are bringing this upon yourself.
(11) By marcrew on 2022-11-17 08:58:26 in reply to 10 [link] [source]
hello Gunter , thanks for reply I have tried again, it is still the same. could you check my code? I might think it is something related to VFS, but I dont know how to fix it. sprintf(sql, "INSERT INTO TICKET VALUES (NULL, \'%s\', NULL, NULL, \'%s\',\'%s\', \'%s\',\'%s\');" , uid, date, time, status[ticket->status], type[ticket->type]); rc = sqlite3_exec(db, "BEGIN;", NULL, 0, &zErrMsg); rc = sqlite3_exec(db, sql, NULL, 0, &zErrMsg); if(rc != SQLITE_OK) { printf("SQL %s INSERT FAILED: %s\n", sql, zErrMsg); sqlite3_free(zErrMsg); } else { #if DEBUG printf("insert ok\n"); #endif } rc = sqlite3_exec(db, "COMMIT;", NULL, 0, &zErrMsg);
(12) By marcrew on 2022-11-17 09:04:10 in reply to 11 [link] [source]
hello Gunter , thanks for reply I have tried again, it is still the same. could you check my code? I might think it is something related to VFS, but I dont know how to fix it. ------------- below is for init sqlite3_vfs* pDefault = sqlite3_vfs_find("unix-dotfile"); if(!pDefault) { printf("cannot find vfs\n"); } sqlite3_vfs_register(pDefault, 1); char *zErrMsg = 0; char* name = cJSON_GetObjectItem(products->handle, "item")->valuestring; int rc = sqlite3_open("./test.db", &db); if( rc ){ printf( "Can't open database: %s\n", sqlite3_errmsg(db)); return FALSE; }else{ printf( "Opened database successfully\n"); } char* sql = "CREATE TABLE if not exists TICKET(" \ "ID INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL," \ "UIDTID VARCHAR(24) ," \ "EPC VARCHAR(124) ," \ "SN VARCHAR(12) , " \ "DATE VARCHAR(8) NOT NULL," \ "TIME VARCHAR(8) NOT NULL," \ "STATUS VARCHAR(8) NOT NULL," \ "TYPE VARCHAR(8) NOT NULL" \ ");"; rc = sqlite3_exec(db, sql, NULL, 0, &zErrMsg); if(rc != SQLITE_OK) { printf("DB: %s\n", zErrMsg); sqlite3_free(zErrMsg); return FALSE; } else { printf("table created\n"); } -------------below is for insert sprintf(sql, "INSERT INTO TICKET VALUES (NULL, \'%s\', NULL, NULL, \'%s\',\'%s\', \'%s\',\'%s\');" , uid, date, time, status[ticket->status], type[ticket->type]); rc = sqlite3_exec(db, "BEGIN;", NULL, 0, &zErrMsg); rc = sqlite3_exec(db, sql, NULL, 0, &zErrMsg); if(rc != SQLITE_OK) { printf("SQL %s INSERT FAILED: %s\n", sql, zErrMsg); sqlite3_free(zErrMsg); } else { #if DEBUG printf("insert ok\n"); #endif } rc = sqlite3_exec(db, "COMMIT;", NULL, 0, &zErrMsg);
(13) By Gunter Hick (gunter_hick) on 2022-11-18 07:04:53 in reply to 12 [link] [source]
Check if your VFS filesystem implementation xSync function gets called. If yes, then the problem is in the VFS. If no, then your settings are preventing SQLite from calling it.
(15) By Simon Slavin (slavin) on 2022-11-19 15:17:31 in reply to 8 [link] [source]
The remedy is not to turn journalling off. By not writing a journal you are telling SQLite that you don't care about data being lost due to software or hardware failure. Since SQLite knows you don't care, it doesn't put the data anywhere on your storage medium, so no kind of synchronisation will cause that data to be saved.
There are two remedies:
- Leave journalling on. You can use a different journal mode but don't turn it off.
- When you have made enough changes to be worth saving, close the connection you opened. This causes all data to be saved in the database file.
The first of those is the one almost everyone uses. To learn what various journaling modes do, read this:
https://sqlite.org/pragma.html#pragma_journal_mode
[Above explanation simplified for clarity.]
(3) By Gunter Hick (gunter_hick) on 2022-11-08 06:12:28 in reply to 1 [link] [source]
Maybe you are not committing your transaction. Since everybody else is successfuly using SQLite, you must be doing something wrong. Which we cannot discern until you show us your code, or at least the statements you are executing.
(5) By marcrew on 2022-11-08 07:38:02 in reply to 3 [link] [source]
thansk Gunter Hick for reply
if I doesn't close the db, everything will not save to disk. here is my code, all return value is success :
//only unix-none and unix-dotfile are available for rtems
sqlite3_vfs* pDefault = sqlite3_vfs_find("unix-none");
if(!pDefault)
{
printf("cannot find vfs\n");
exit(0);
}
sqlite3_vfs_register(pDefault, 1);
sqlite3 *db ;
int rc = sqlite3_open("./test.db", &db);
if( rc )
{
fprintf(stderr, "Can't open database: %s\n", sqlite3_errmsg(db));
exit(0);
}
else
{
fprintf(stderr, "Opened database successfully\n");
}
char *zErrMsg = 0;
rc = sqlite3_exec(db, "PRAGMA journal_mode = OFF;", NULL, 0, &zErrMsg);
if( rc != SQLITE_OK )
{
fprintf(stderr, "SQL error: %s\n", zErrMsg);
sqlite3_free(zErrMsg);
}
else
{
fprintf(stdout, "close journal successfully\n");
}
rc = sqlite3_exec(db, "PRAGMA locking_mode = NORMAL;", NULL, 0, &zErrMsg);
if( rc != SQLITE_OK )
{
fprintf(stderr, "SQL error: %s\n", zErrMsg);
sqlite3_free(zErrMsg);
}
else
{
fprintf(stdout, "locking_mode successfully\n");
}
rc = sqlite3_exec(db, sql_create, NULL, 0, &zErrMsg);
if( rc != SQLITE_OK )
{
fprintf(stderr, "SQL error: %s\n", zErrMsg);
sqlite3_free(zErrMsg);
}else
{
fprintf(stdout, "Table created successfully\n");
}
rc = sqlite3_exec(db, sql_insert, NULL, 0, &zErrMsg);
if( rc != SQLITE_OK )
{
fprintf(stderr, "SQL error: %s\n", zErrMsg);
sqlite3_free(zErrMsg);
}
else
{
fprintf(stdout, "data insert successfully\n");
}
// if I doesn't close the db, everything will not save to disk.
//sqlite3_close(db);
(7) By Gunter Hick (gunter_hick) on 2022-11-08 09:32:18 in reply to 5 [link] [source]
Did you read up on the journal mode? https://sqlite.org/pragma.html#pragma_journal_mode "The OFF journaling mode disables the atomic commit and rollback capabilities of SQLite. ... If the application crashes in the middle of a transaction when the OFF journaling mode is set, then the database file will very likely go corrupt." I would say exiting the program without closing the database connection has a similar effect as crashing the application. Maybe your rtems implementation of the unix-none VFS is keeping data from hitting the storage device or setting the yournal mode is avoiding calls to flush or whatever. If you have an ltrace or strace utility to capture system calls you should be able to see the difference.
(14) By Simon Slavin (slavin) on 2022-11-18 14:51:08 in reply to 5 [link] [source]
Until you have neatly finished your SQLite work, your changes are saved in the journal file, not the database file. You have turned journalling OFF. This means you don't want a journal. Therefore your changes are lost.
If you need data to be preserved through hardware and software problems, you need some form of journaling. So don't turn journalling OFF.
(9) By cj (sqlitening) on 2022-11-17 07:00:51 in reply to 1 [link] [source]
Might check for success after writes using one of these: sqlite3_changes(hDB) sqlite3_total_changes(hDB)