Many hyperlinks are disabled.
Use anonymous login
to enable hyperlinks.
Overview
Comment: | Fix a problem in the test scripts for the asynchronous backend. (CVS 4400) |
---|---|
Downloads: | Tarball | ZIP archive |
Timelines: | family | ancestors | descendants | both | trunk |
Files: | files | file ages | folders |
SHA1: |
630fc71f3df5ab6129ddff9d8184893e |
User & Date: | danielk1977 2007-09-05 11:34:54.000 |
Context
2007-09-05
| ||
13:56 | Remove the unixFile.isOpen variable (no longer in use). (CVS 4401) (check-in: 1786e9c881 user: danielk1977 tags: trunk) | |
11:34 | Fix a problem in the test scripts for the asynchronous backend. (CVS 4400) (check-in: 630fc71f3d user: danielk1977 tags: trunk) | |
2007-09-04
| ||
22:31 | Do not use the TryEnterCriticalSection API on windows since it is unavailable on some platforms. (CVS 4399) (check-in: bf3d67d1bd user: drh tags: trunk) | |
Changes
Changes to src/test_async.c.
︙ | ︙ | |||
20 21 22 23 24 25 26 | ** a database write does not have to wait for (sometimes slow) disk I/O ** to occur. The write seems to happen very quickly, though in reality ** it is happening at its usual slow pace in the background. ** ** Asynchronous I/O appears to give better responsiveness, but at a price. ** You lose the Durable property. With the default I/O backend of SQLite, ** once a write completes, you know that the information you wrote is | | | | < < < < | < < | > | < | < | < < < < < | > > > | 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 | ** a database write does not have to wait for (sometimes slow) disk I/O ** to occur. The write seems to happen very quickly, though in reality ** it is happening at its usual slow pace in the background. ** ** Asynchronous I/O appears to give better responsiveness, but at a price. ** You lose the Durable property. With the default I/O backend of SQLite, ** once a write completes, you know that the information you wrote is ** safely on disk. With the asynchronous I/O, this is not the case. If ** your program crashes or if a power lose occurs after the database ** write but before the asynchronous write thread has completed, then the ** database change might never make it to disk and the next user of the ** database might not see your change. ** ** You lose Durability with asynchronous I/O, but you still retain the ** other parts of ACID: Atomic, Consistent, and Isolated. Many ** appliations get along fine without the Durablity. ** ** HOW IT WORKS ** ** Asynchronous I/O works by creating a special SQLite "vfs" structure ** and registering it with sqlite3_vfs_register(). When files opened via ** this vfs are written to (using sqlite3OsWrite()), the data is not ** written directly to disk, but is placed in the "write-queue" to be ** handled by the background thread. ** ** The special vfs is registered (and unregistered) by calls to ** function asyncEnable() (see below). ** ** LIMITATIONS ** ** This demonstration code is deliberately kept simple in order to keep ** the main ideas clear and easy to understand. Real applications that ** want to do asynchronous I/O might want to add additional capabilities. ** For example, in this demonstration if writes are happening at a steady ** stream that exceeds the I/O capability of the background writer thread, ** the queue of pending write operations will grow without bound until we ** run out of memory. Users of this technique may want to keep track of ** the quantity of pending writes and stop accepting new write requests ** when the buffer gets to be too big. */ /* ** If this symbol is defined, then file-system locks are obtained as ** required. This slows things down, but allows multiple processes ** to access the database concurrently. If this symbol is not defined, ** then connections from within a single process will respect each ** others database locks, but external connections will not - leading ** to database corruption. */ #define ENABLE_FILE_LOCKING #include "sqliteInt.h" #include <tcl.h> /* |
︙ | ︙ | |||
121 122 123 124 125 126 127 | /* ** THREAD SAFETY NOTES ** ** Basic rules: ** ** * Both read and write access to the global write-op queue must be | | > > > > | 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 | /* ** THREAD SAFETY NOTES ** ** Basic rules: ** ** * Both read and write access to the global write-op queue must be ** protected by the async.queueMutex. As are the async.ioError and ** async.nFile variables. ** ** * The async.aLock hash-table and all AsyncLock and AsyncFileLock ** structures must be protected by teh async.lockMutex mutex. ** ** * The file handles from the underlying system are assumed not to ** be thread safe. ** ** * See the last two paragraphs under "The Writer Thread" for ** an assumption to do with file-handle synchronization by the Os. ** |
︙ | ︙ | |||
304 305 306 307 308 309 310 311 312 313 314 | int op; /* One of ASYNC_xxx etc. */ i64 iOffset; /* See above */ int nByte; /* See above */ char *zBuf; /* Data to write to file (or NULL if op!=ASYNC_WRITE) */ AsyncWrite *pNext; /* Next write operation (to any file) */ }; /* ** An instance of the following structure is allocated along with each ** AsyncFileData structure (see AsyncFileData.lock), but is only used if the ** file was opened with the SQLITE_OPEN_MAIN_DB. | > > > > > > > > > > > > > > > > > > > > > > > > > < < < < < < < < < < < < < | 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 | int op; /* One of ASYNC_xxx etc. */ i64 iOffset; /* See above */ int nByte; /* See above */ char *zBuf; /* Data to write to file (or NULL if op!=ASYNC_WRITE) */ AsyncWrite *pNext; /* Next write operation (to any file) */ }; /* ** An instance of this structure is created for each distinct open file ** (i.e. if two handles are opened on the one file, only one of these ** structures is allocated) and stored in the async.aLock hash table. The ** keys for async.aLock are the full pathnames of the opened files. ** ** AsyncLock.pList points to the head of a linked list of AsyncFileLock ** structures, one for each handle currently open on the file. ** ** If the opened file is not a main-database (the SQLITE_OPEN_MAIN_DB is ** not passed to the sqlite3OsOpen() call), or if ENABLE_FILE_LOCKING is ** not defined at compile time, variables AsyncLock.pFile and ** AsyncLock.eLock are never used. Otherwise, pFile is a file handle ** opened on the file in question and used to obtain the file-system ** locks required by database connections within this process. ** ** See comments above the asyncLock() function for more details on ** the implementation of database locking used by this backend. */ struct AsyncLock { sqlite3_file *pFile; int eLock; AsyncFileLock *pList; }; /* ** An instance of the following structure is allocated along with each ** AsyncFileData structure (see AsyncFileData.lock), but is only used if the ** file was opened with the SQLITE_OPEN_MAIN_DB. */ struct AsyncFileLock { int eLock; /* Internally visible lock state (sqlite pov) */ int eAsyncLock; /* Lock-state with write-queue unlock */ AsyncFileLock *pNext; }; /* ** The AsyncFile structure is a subclass of sqlite3_file used for ** asynchronous IO. ** ** All of the actual data for the structure is stored in the structure ** pointed to by AsyncFile.pData, which is allocated as part of the ** sqlite3OsOpen() using sqlite3_malloc(). The reason for this is that the |
︙ | ︙ | |||
607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 | int eRequired = 0; if( pLock->pFile ){ for(pIter=pLock->pList; pIter; pIter=pIter->pNext){ assert(pIter->eAsyncLock>=pIter->eLock); if( pIter->eAsyncLock>eRequired ){ eRequired = pIter->eAsyncLock; } } if( eRequired>pLock->eLock ){ rc = sqlite3OsLock(pLock->pFile, eRequired); }else if(eRequired<pLock->eLock){ rc = sqlite3OsUnlock(pLock->pFile, eRequired); } if( rc==SQLITE_OK ){ pLock->eLock = eRequired; } } return rc; } /* | > | | < < < | | 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 | int eRequired = 0; if( pLock->pFile ){ for(pIter=pLock->pList; pIter; pIter=pIter->pNext){ assert(pIter->eAsyncLock>=pIter->eLock); if( pIter->eAsyncLock>eRequired ){ eRequired = pIter->eAsyncLock; assert(eRequired>=0 && eRequired<=SQLITE_LOCK_EXCLUSIVE); } } if( eRequired>pLock->eLock ){ rc = sqlite3OsLock(pLock->pFile, eRequired); }else if(eRequired<pLock->eLock){ rc = sqlite3OsUnlock(pLock->pFile, eRequired); } if( rc==SQLITE_OK ){ pLock->eLock = eRequired; } } return rc; } /* ** The following two methods - asyncLock() and asyncUnlock() - are used ** to obtain and release locks on database files opened with the ** asynchronous backend. */ static int asyncLock(sqlite3_file *pFile, int eLock){ int rc = SQLITE_OK; AsyncFileData *p = ((AsyncFile *)pFile)->pData; pthread_mutex_lock(&async.lockMutex); if( p->lock.eLock<eLock ){ |
︙ | ︙ | |||
652 653 654 655 656 657 658 | (eLock==SQLITE_LOCK_SHARED && pIter->eLock>=SQLITE_LOCK_PENDING) )){ rc = SQLITE_BUSY; } } if( rc==SQLITE_OK ){ p->lock.eLock = eLock; | < | < < | < | 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 | (eLock==SQLITE_LOCK_SHARED && pIter->eLock>=SQLITE_LOCK_PENDING) )){ rc = SQLITE_BUSY; } } if( rc==SQLITE_OK ){ p->lock.eLock = eLock; p->lock.eAsyncLock = MAX(p->lock.eAsyncLock, eLock); } assert(p->lock.eAsyncLock>=p->lock.eLock); if( rc==SQLITE_OK ){ rc = getFileLock(pLock); } } pthread_mutex_unlock(&async.lockMutex); ASYNC_TRACE(("LOCK %d (%s) rc=%d\n", eLock, p->zName, rc)); return rc; } static int asyncUnlock(sqlite3_file *pFile, int eLock){ AsyncFileData *p = ((AsyncFile *)pFile)->pData; AsyncFileLock *pLock = &p->lock; pthread_mutex_lock(&async.lockMutex); pLock->eLock = MIN(pLock->eLock, eLock); pthread_mutex_unlock(&async.lockMutex); return addNewAsyncWrite(p, ASYNC_UNLOCK, 0, eLock, 0); } /* ** This function is called when the pager layer first opens a database file ** and is checking for a hot-journal. |
︙ | ︙ | |||
1006 1007 1008 1009 1010 1011 1012 1013 1014 1015 | ** ** This routine is not even remotely threadsafe. Do not call ** this routine while any SQLite database connections are open. */ static void asyncEnable(int enable){ if( enable ){ if( !async_vfs.pAppData ){ async_vfs.pAppData = (void *)sqlite3_vfs_find(0); async_vfs.mxPathname = ((sqlite3_vfs *)async_vfs.pAppData)->mxPathname; sqlite3_vfs_register(&async_vfs, 1); | > > | > > < | 1007 1008 1009 1010 1011 1012 1013 1014 1015 1016 1017 1018 1019 1020 1021 1022 1023 1024 1025 1026 1027 1028 1029 1030 1031 1032 1033 | ** ** This routine is not even remotely threadsafe. Do not call ** this routine while any SQLite database connections are open. */ static void asyncEnable(int enable){ if( enable ){ if( !async_vfs.pAppData ){ static int hashTableInit = 0; async_vfs.pAppData = (void *)sqlite3_vfs_find(0); async_vfs.mxPathname = ((sqlite3_vfs *)async_vfs.pAppData)->mxPathname; sqlite3_vfs_register(&async_vfs, 1); if( !hashTableInit ){ sqlite3HashInit(&async.aLock, SQLITE_HASH_BINARY, 1); hashTableInit = 1; } } }else{ if( async_vfs.pAppData ){ sqlite3_vfs_unregister(&async_vfs); async_vfs.pAppData = 0; } } } /* ** This procedure runs in a separate thread, reading messages off of the ** write queue and processing them one by one. |
︙ | ︙ | |||
1145 1146 1147 1148 1149 1150 1151 | ** structures for this file. Obtain the async.lockMutex mutex ** before doing so. */ pthread_mutex_lock(&async.lockMutex); pLock = sqlite3HashFind(&async.aLock, pData->zName, pData->nName); for(ppIter=&pLock->pList; *ppIter; ppIter=&((*ppIter)->pNext)){ if( (*ppIter)==&pData->lock ){ | | | > > > > > < < | < | < < > | 1149 1150 1151 1152 1153 1154 1155 1156 1157 1158 1159 1160 1161 1162 1163 1164 1165 1166 1167 1168 1169 1170 1171 1172 1173 1174 1175 1176 1177 1178 1179 1180 1181 1182 1183 1184 1185 1186 1187 1188 1189 1190 1191 1192 | ** structures for this file. Obtain the async.lockMutex mutex ** before doing so. */ pthread_mutex_lock(&async.lockMutex); pLock = sqlite3HashFind(&async.aLock, pData->zName, pData->nName); for(ppIter=&pLock->pList; *ppIter; ppIter=&((*ppIter)->pNext)){ if( (*ppIter)==&pData->lock ){ *ppIter = pData->lock.pNext; break; } } if( !pLock->pList ){ if( pLock->pFile ){ sqlite3OsClose(pLock->pFile); } sqlite3_free(pLock); sqlite3HashInsert(&async.aLock, pData->zName, pData->nName, 0); if( !sqliteHashFirst(&async.aLock) ){ sqlite3HashClear(&async.aLock); } }else{ rc = getFileLock(pLock); } pthread_mutex_unlock(&async.lockMutex); sqlite3_free(pData); break; } case ASYNC_UNLOCK: { AsyncLock *pLock; AsyncFileData *pData = p->pFileData; int eLock = p->nByte; pthread_mutex_lock(&async.lockMutex); pData->lock.eAsyncLock = MIN( pData->lock.eAsyncLock, MAX(pData->lock.eLock, eLock) ); assert(pData->lock.eAsyncLock>=pData->lock.eLock); pLock = sqlite3HashFind(&async.aLock, pData->zName, pData->nName); rc = getFileLock(pLock); pthread_mutex_unlock(&async.lockMutex); break; } |
︙ | ︙ |
Changes to test/async.test.
1 2 3 4 5 6 7 8 | # # May you do good and not evil. # May you find forgiveness for yourself and forgive others. # May you share freely, never taking more than you give. # #*********************************************************************** # This file runs all tests. # | | | > > > > < | > > > > > > > > | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 | # # May you do good and not evil. # May you find forgiveness for yourself and forgive others. # May you share freely, never taking more than you give. # #*********************************************************************** # This file runs all tests. # # $Id: async.test,v 1.10 2007/09/05 11:34:54 danielk1977 Exp $ if {[catch {sqlite3async_enable}]} { # The async logic is not built into this system return } set testdir [file dirname $argv0] source $testdir/tester.tcl rename finish_test really_finish_test proc finish_test {} { catch {db close} catch {db2 close} catch {db3 close} } set ISQUICK 1 set INCLUDE { select1.test select2.test select3.test select4.test insert.test insert2.test insert3.test trans.test lock.test lock3.test lock2.test } # Enable asynchronous IO. sqlite3async_enable 1 rename do_test really_do_test proc do_test {name args} { uplevel really_do_test async_io-$name $args sqlite3async_halt idle sqlite3async_start sqlite3async_wait } foreach testfile [lsort -dictionary [glob $testdir/*.test]] { set tail [file tail $testfile] if {[lsearch -exact $INCLUDE $tail]<0} continue source $testfile # Make sure everything is flushed through. This is because [source]ing # the next test file will delete the database file on disk (using # [file delete]). If the asynchronous backend still has the file # open, it will become confused. # sqlite3async_halt idle sqlite3async_start sqlite3async_wait } # Flush the write-queue and disable asynchronous IO. This should ensure # all allocated memory is cleaned up. set sqlite3async_trace 1 sqlite3async_halt idle sqlite3async_start |
︙ | ︙ |
Changes to test/async2.test.
1 2 3 4 5 6 7 | # # May you do good and not evil. # May you find forgiveness for yourself and forgive others. # May you share freely, never taking more than you give. # #*********************************************************************** # | | | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 | # # May you do good and not evil. # May you find forgiveness for yourself and forgive others. # May you share freely, never taking more than you give. # #*********************************************************************** # # $Id: async2.test,v 1.7 2007/09/05 11:34:54 danielk1977 Exp $ set testdir [file dirname $argv0] source $testdir/tester.tcl if { [info commands sqlite3async_enable]=="" || |
︙ | ︙ | |||
66 67 68 69 70 71 72 | ioerr { set ::sqlite_io_error_pending $n } malloc-persistent { sqlite3_memdebug_fail $n -repeat 1 } malloc-transient { sqlite3_memdebug_fail $n -repeat 0 } } sqlite3async_halt idle sqlite3async_start sqlite3async_wait | > | | 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 | ioerr { set ::sqlite_io_error_pending $n } malloc-persistent { sqlite3_memdebug_fail $n -repeat 1 } malloc-transient { sqlite3_memdebug_fail $n -repeat 0 } } sqlite3async_halt idle sqlite3async_start sqlite3async_wait sqlite3async_enable 0 set ::sqlite_io_error_pending 0 sqlite3_memdebug_fail -1 sqlite3 db test.db set c [db eval {SELECT c FROM counter LIMIT 1}] switch -- $c { 1 { |
︙ | ︙ | |||
109 110 111 112 113 114 115 | } {klmnopqrst and seven} } FIN { set ::go 0 } } | | | 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 | } {klmnopqrst and seven} } FIN { set ::go 0 } } db close } } catch {db close} sqlite3async_halt idle sqlite3async_start sqlite3async_wait finish_test |