Many hyperlinks are disabled.
Use anonymous login
to enable hyperlinks.
Overview
| Comment: | Update the Apple OS-X branch to include all of the latest changes in trunk. |
|---|---|
| Downloads: | Tarball | ZIP archive | SQL archive |
| Timelines: | family | ancestors | descendants | both | apple-osx |
| Files: | files | file ages | folders |
| SHA1: |
96499b1dd69a0954e30e4b5b4f2272ab |
| User & Date: | drh 2010-01-20 01:26:07 |
Context
|
2010-01-20
| ||
| 13:20 | Align the os_unix.c source file with the version found on trunk. (check-in: fa0f6c14 user: drh tags: apple-osx) | |
| 01:26 | Update the Apple OS-X branch to include all of the latest changes in trunk. (check-in: 96499b1d user: drh tags: apple-osx) | |
| 01:20 | Manually copy over the rebustness fixes from the apple-osx branch. (check-in: 095c74ea user: drh tags: trunk) | |
|
2010-01-19
| ||
| 23:50 | robustness fixes for preventing a finalized statement from being reused (check-in: a7a0c8d6 user: adam tags: apple-osx) | |
Changes
Changes to ext/fts3/fts3.c.
19 19 ** * The FTS3 module is being built as an extension 20 20 ** (in which case SQLITE_CORE is not defined), or 21 21 ** 22 22 ** * The FTS3 module is being built into the core of 23 23 ** SQLite (in which case SQLITE_ENABLE_FTS3 is defined). 24 24 */ 25 25 26 -/* TODO(shess) Consider exporting this comment to an HTML file or the 27 -** wiki. 28 -*/ 29 26 /* The full-text index is stored in a series of b+tree (-like) 30 27 ** structures called segments which map terms to doclists. The 31 28 ** structures are like b+trees in layout, but are constructed from the 32 29 ** bottom up in optimal fashion and are not updatable. Since trees 33 30 ** are built from the bottom up, things will be described from the 34 31 ** bottom up. 35 32 ** ................................................................................ 44 41 ** B = 1xxxxxxx 7 bits of data and one flag bit 45 42 ** 46 43 ** 7 bits - A 47 44 ** 14 bits - BA 48 45 ** 21 bits - BBA 49 46 ** and so on. 50 47 ** 51 -** This is identical to how sqlite encodes varints (see util.c). 48 +** This is similar in concept to how sqlite encodes "varints" but 49 +** the encoding is not the same. SQLite varints are big-endian 50 +** are are limited to 9 bytes in length whereas FTS3 varints are 51 +** little-endian and can be upt to 10 bytes in length (in theory). 52 +** 53 +** Example encodings: 54 +** 55 +** 1: 0x01 56 +** 127: 0x7f 57 +** 128: 0x81 0x00 52 58 ** 53 59 ** 54 60 **** Document lists **** 55 61 ** A doclist (document list) holds a docid-sorted list of hits for a 56 62 ** given term. Doclists hold docids, and can optionally associate 57 -** token positions and offsets with docids. 63 +** token positions and offsets with docids. A position is the index 64 +** of a word within the document. The first word of the document has 65 +** a position of 0. 66 +** 67 +** FTS3 used to optionally store character offsets using a compile-time 68 +** option. But that functionality is no longer supported. 58 69 ** 59 70 ** A DL_POSITIONS_OFFSETS doclist is stored like this: 60 71 ** 61 72 ** array { 62 73 ** varint docid; 63 74 ** array { (position list for column 0) 64 75 ** varint position; (delta from previous position plus POS_BASE) 65 -** varint startOffset; (delta from previous startOffset) 66 -** varint endOffset; (delta from startOffset) 67 76 ** } 68 77 ** array { 69 78 ** varint POS_COLUMN; (marks start of position list for new column) 70 79 ** varint column; (index of new column) 71 80 ** array { 72 81 ** varint position; (delta from previous position plus POS_BASE) 73 -** varint startOffset;(delta from previous startOffset) 74 -** varint endOffset; (delta from startOffset) 75 82 ** } 76 83 ** } 77 84 ** varint POS_END; (marks end of positions for this document. 78 85 ** } 79 86 ** 80 87 ** Here, array { X } means zero or more occurrences of X, adjacent in 81 88 ** memory. A "position" is an index of a token in the token stream 82 -** generated by the tokenizer, while an "offset" is a byte offset, 83 -** both based at 0. Note that POS_END and POS_COLUMN occur in the 84 -** same logical place as the position element, and act as sentinals 85 -** ending a position list array. 89 +** generated by the tokenizer. Note that POS_END and POS_COLUMN occur 90 +** in the same logical place as the position element, and act as sentinals 91 +** ending a position list array. POS_END is 0. POS_COLUMN is 1. 92 +** The positions numbers are not stored literally but rather as two more 93 +** the difference from the prior position, or the just the position plus 94 +** 2 for the first position. Example: 95 +** 96 +** label: A B C D E F G H I J K 97 +** value: 123 5 9 1 1 14 35 0 234 72 0 98 +** 99 +** The 123 value is the first docid. For column zero in this document 100 +** there are two matches at positions 3 and 10 (5-2 and 9-2+3). The 1 101 +** at D signals the start of a new column; the 1 at E indicates that the 102 +** new column is column number 1. There are two positions at 12 and 45 103 +** (14-2 and 35-2+12). The 0 at H indicate the end-of-document. The 104 +** 234 at I is the next docid. It has one position 72 (72-2) and then 105 +** terminates with the 0 at K. 86 106 ** 87 107 ** A DL_POSITIONS doclist omits the startOffset and endOffset 88 108 ** information. A DL_DOCIDS doclist omits both the position and 89 109 ** offset information, becoming an array of varint-encoded docids. 90 110 ** 91 111 ** On-disk data is stored as type DL_DEFAULT, so we don't serialize 92 112 ** the type. Due to how deletion is implemented in the segmentation ................................................................................ 384 404 z[iOut++] = z[iIn++]; 385 405 } 386 406 } 387 407 z[iOut] = '\0'; 388 408 } 389 409 } 390 410 411 +/* 412 +** Read a single varint from the doclist at *pp and advance *pp to point 413 +** to the next element of the varlist. Add the value of the varint 414 +** to *pVal. 415 +*/ 391 416 static void fts3GetDeltaVarint(char **pp, sqlite3_int64 *pVal){ 392 417 sqlite3_int64 iVal; 393 418 *pp += sqlite3Fts3GetVarint(*pp, &iVal); 394 419 *pVal += iVal; 395 420 } 396 421 422 +/* 423 +** As long as *pp has not reached its end (pEnd), then do the same 424 +** as fts3GetDeltaVarint(): read a single varint and add it to *pVal. 425 +** But if we have reached the end of the varint, just set *pp=0 and 426 +** leave *pVal unchanged. 427 +*/ 397 428 static void fts3GetDeltaVarint2(char **pp, char *pEnd, sqlite3_int64 *pVal){ 398 429 if( *pp>=pEnd ){ 399 430 *pp = 0; 400 431 }else{ 401 432 fts3GetDeltaVarint(pp, pVal); 402 433 } 403 434 } ................................................................................ 777 808 *ppCsr = pCsr = (sqlite3_vtab_cursor *)sqlite3_malloc(sizeof(Fts3Cursor)); 778 809 if( !pCsr ){ 779 810 return SQLITE_NOMEM; 780 811 } 781 812 memset(pCsr, 0, sizeof(Fts3Cursor)); 782 813 return SQLITE_OK; 783 814 } 784 - 785 -/****************************************************************/ 786 -/****************************************************************/ 787 -/****************************************************************/ 788 -/****************************************************************/ 789 - 790 815 791 816 /* 792 817 ** Close the cursor. For additional information see the documentation 793 818 ** on the xClose method of the virtual table interface. 794 819 */ 795 820 static int fulltextClose(sqlite3_vtab_cursor *pCursor){ 796 821 Fts3Cursor *pCsr = (Fts3Cursor *)pCursor; ................................................................................ 1704 1729 *paOut = pOut; 1705 1730 *pnOut = nOut; 1706 1731 }else{ 1707 1732 sqlite3_free(pOut); 1708 1733 } 1709 1734 return rc; 1710 1735 } 1736 + 1737 +static int fts3NearMerge( 1738 + int mergetype, /* MERGE_POS_NEAR or MERGE_NEAR */ 1739 + int nNear, /* Parameter to NEAR operator */ 1740 + int nTokenLeft, /* Number of tokens in LHS phrase arg */ 1741 + char *aLeft, /* Doclist for LHS (incl. positions) */ 1742 + int nLeft, /* Size of LHS doclist in bytes */ 1743 + int nTokenRight, /* As nTokenLeft */ 1744 + char *aRight, /* As aLeft */ 1745 + int nRight, /* As nRight */ 1746 + char **paOut, /* OUT: Results of merge (malloced) */ 1747 + int *pnOut /* OUT: Sized of output buffer */ 1748 +){ 1749 + char *aOut; 1750 + int rc; 1751 + 1752 + assert( mergetype==MERGE_POS_NEAR || MERGE_NEAR ); 1753 + 1754 + aOut = sqlite3_malloc(nLeft+nRight+1); 1755 + if( aOut==0 ){ 1756 + rc = SQLITE_NOMEM; 1757 + }else{ 1758 + rc = fts3DoclistMerge(mergetype, nNear+nTokenRight, nNear+nTokenLeft, 1759 + aOut, pnOut, aLeft, nLeft, aRight, nRight 1760 + ); 1761 + if( rc!=SQLITE_OK ){ 1762 + sqlite3_free(aOut); 1763 + aOut = 0; 1764 + } 1765 + } 1766 + 1767 + *paOut = aOut; 1768 + return rc; 1769 +} 1770 + 1771 +int sqlite3Fts3ExprNearTrim(Fts3Expr *pLeft, Fts3Expr *pRight, int nNear){ 1772 + int rc; 1773 + if( pLeft->aDoclist==0 || pRight->aDoclist==0 ){ 1774 + sqlite3_free(pLeft->aDoclist); 1775 + sqlite3_free(pRight->aDoclist); 1776 + pRight->aDoclist = 0; 1777 + pLeft->aDoclist = 0; 1778 + rc = SQLITE_OK; 1779 + }else{ 1780 + char *aOut; 1781 + int nOut; 1782 + 1783 + rc = fts3NearMerge(MERGE_POS_NEAR, nNear, 1784 + pLeft->pPhrase->nToken, pLeft->aDoclist, pLeft->nDoclist, 1785 + pRight->pPhrase->nToken, pRight->aDoclist, pRight->nDoclist, 1786 + &aOut, &nOut 1787 + ); 1788 + if( rc!=SQLITE_OK ) return rc; 1789 + sqlite3_free(pRight->aDoclist); 1790 + pRight->aDoclist = aOut; 1791 + pRight->nDoclist = nOut; 1792 + 1793 + rc = fts3NearMerge(MERGE_POS_NEAR, nNear, 1794 + pRight->pPhrase->nToken, pRight->aDoclist, pRight->nDoclist, 1795 + pLeft->pPhrase->nToken, pLeft->aDoclist, pLeft->nDoclist, 1796 + &aOut, &nOut 1797 + ); 1798 + sqlite3_free(pLeft->aDoclist); 1799 + pLeft->aDoclist = aOut; 1800 + pLeft->nDoclist = nOut; 1801 + } 1802 + return rc; 1803 +} 1711 1804 1712 1805 /* 1713 1806 ** Evaluate the full-text expression pExpr against fts3 table pTab. Store 1714 1807 ** the resulting doclist in *paOut and *pnOut. 1715 1808 */ 1716 1809 static int evalFts3Expr( 1717 1810 Fts3Table *p, /* Virtual table handle */ ................................................................................ 1749 1842 || pExpr->eType==FTSQUERY_AND || pExpr->eType==FTSQUERY_NOT 1750 1843 ); 1751 1844 switch( pExpr->eType ){ 1752 1845 case FTSQUERY_NEAR: { 1753 1846 Fts3Expr *pLeft; 1754 1847 Fts3Expr *pRight; 1755 1848 int mergetype = isReqPos ? MERGE_POS_NEAR : MERGE_NEAR; 1756 - int nParam1; 1757 - int nParam2; 1758 - char *aBuffer; 1759 1849 1760 1850 if( pExpr->pParent && pExpr->pParent->eType==FTSQUERY_NEAR ){ 1761 1851 mergetype = MERGE_POS_NEAR; 1762 1852 } 1763 1853 pLeft = pExpr->pLeft; 1764 1854 while( pLeft->eType==FTSQUERY_NEAR ){ 1765 1855 pLeft=pLeft->pRight; 1766 1856 } 1767 1857 pRight = pExpr->pRight; 1768 1858 assert( pRight->eType==FTSQUERY_PHRASE ); 1769 1859 assert( pLeft->eType==FTSQUERY_PHRASE ); 1770 1860 1771 - nParam1 = pExpr->nNear+1; 1772 - nParam2 = nParam1+pLeft->pPhrase->nToken+pRight->pPhrase->nToken-2; 1773 - aBuffer = sqlite3_malloc(nLeft+nRight+1); 1774 - rc = fts3DoclistMerge(mergetype, nParam1, nParam2, aBuffer, 1775 - pnOut, aLeft, nLeft, aRight, nRight 1861 + rc = fts3NearMerge(mergetype, pExpr->nNear, 1862 + pLeft->pPhrase->nToken, aLeft, nLeft, 1863 + pRight->pPhrase->nToken, aRight, nRight, 1864 + paOut, pnOut 1776 1865 ); 1777 - if( rc!=SQLITE_OK ){ 1778 - sqlite3_free(aBuffer); 1779 - }else{ 1780 - *paOut = aBuffer; 1781 - } 1782 1866 sqlite3_free(aLeft); 1783 1867 break; 1784 1868 } 1785 1869 1786 1870 case FTSQUERY_OR: { 1787 1871 /* Allocate a buffer for the output. The maximum size is the 1788 1872 ** sum of the sizes of the two input buffers. The +1 term is ................................................................................ 2025 2109 */ 2026 2110 int sqlite3Fts3ExprLoadDoclist(Fts3Table *pTab, Fts3Expr *pExpr){ 2027 2111 return evalFts3Expr(pTab, pExpr, &pExpr->aDoclist, &pExpr->nDoclist, 1); 2028 2112 } 2029 2113 2030 2114 /* 2031 2115 ** After ExprLoadDoclist() (see above) has been called, this function is 2032 -** used to iterate through the position lists that make up the doclist 2116 +** used to iterate/search through the position lists that make up the doclist 2033 2117 ** stored in pExpr->aDoclist. 2034 2118 */ 2035 2119 char *sqlite3Fts3FindPositions( 2036 2120 Fts3Expr *pExpr, /* Access this expressions doclist */ 2037 2121 sqlite3_int64 iDocid, /* Docid associated with requested pos-list */ 2038 2122 int iCol /* Column of requested pos-list */ 2039 2123 ){ ................................................................................ 2060 2144 } 2061 2145 while( iThis<iCol ){ 2062 2146 fts3ColumnlistCopy(0, &pCsr); 2063 2147 if( *pCsr==0x00 ) return 0; 2064 2148 pCsr++; 2065 2149 pCsr += sqlite3Fts3GetVarint32(pCsr, &iThis); 2066 2150 } 2067 - if( iCol==iThis ) return pCsr; 2151 + if( iCol==iThis && (*pCsr&0xFE) ) return pCsr; 2068 2152 } 2069 2153 return 0; 2070 2154 } 2071 2155 } 2072 2156 } 2073 2157 2074 2158 return 0; ................................................................................ 2112 2196 int nVal, /* Size of apVal[] array */ 2113 2197 sqlite3_value **apVal /* Array of arguments */ 2114 2198 ){ 2115 2199 Fts3Cursor *pCsr; /* Cursor handle passed through apVal[0] */ 2116 2200 const char *zStart = "<b>"; 2117 2201 const char *zEnd = "</b>"; 2118 2202 const char *zEllipsis = "<b>...</b>"; 2119 - 2120 - /* There must be at least one argument passed to this function (otherwise 2121 - ** the non-overloaded version would have been called instead of this one). 2122 - */ 2123 - assert( nVal>=1 ); 2124 - 2125 - if( nVal>4 ){ 2126 - sqlite3_result_error(pContext, 2127 - "wrong number of arguments to function snippet()", -1); 2128 - return; 2129 - } 2130 - if( fts3FunctionArg(pContext, "snippet", apVal[0], &pCsr) ) return; 2131 - 2132 - switch( nVal ){ 2133 - case 4: zEllipsis = (const char*)sqlite3_value_text(apVal[3]); 2134 - case 3: zEnd = (const char*)sqlite3_value_text(apVal[2]); 2135 - case 2: zStart = (const char*)sqlite3_value_text(apVal[1]); 2136 - } 2137 - if( !zEllipsis || !zEnd || !zStart ){ 2138 - sqlite3_result_error_nomem(pContext); 2139 - }else if( SQLITE_OK==fts3CursorSeek(pContext, pCsr) ){ 2140 - sqlite3Fts3Snippet(pContext, pCsr, zStart, zEnd, zEllipsis); 2141 - } 2142 -} 2143 - 2144 -/* 2145 -** Implementation of the snippet2() function for FTS3 2146 -*/ 2147 -static void fts3Snippet2Func( 2148 - sqlite3_context *pContext, /* SQLite function call context */ 2149 - int nVal, /* Size of apVal[] array */ 2150 - sqlite3_value **apVal /* Array of arguments */ 2151 -){ 2152 - Fts3Cursor *pCsr; /* Cursor handle passed through apVal[0] */ 2153 - const char *zStart = "<b>"; 2154 - const char *zEnd = "</b>"; 2155 - const char *zEllipsis = "<b>...</b>"; 2156 2203 int iCol = -1; 2157 - int nToken = 10; 2204 + int nToken = 15; /* Default number of tokens in snippet */ 2158 2205 2159 2206 /* There must be at least one argument passed to this function (otherwise 2160 2207 ** the non-overloaded version would have been called instead of this one). 2161 2208 */ 2162 2209 assert( nVal>=1 ); 2163 2210 2164 2211 if( nVal>6 ){ ................................................................................ 2174 2221 case 4: zEllipsis = (const char*)sqlite3_value_text(apVal[3]); 2175 2222 case 3: zEnd = (const char*)sqlite3_value_text(apVal[2]); 2176 2223 case 2: zStart = (const char*)sqlite3_value_text(apVal[1]); 2177 2224 } 2178 2225 if( !zEllipsis || !zEnd || !zStart ){ 2179 2226 sqlite3_result_error_nomem(pContext); 2180 2227 }else if( SQLITE_OK==fts3CursorSeek(pContext, pCsr) ){ 2181 - sqlite3Fts3Snippet2(pContext, pCsr, zStart, zEnd, zEllipsis, iCol, nToken); 2228 + sqlite3Fts3Snippet(pContext, pCsr, zStart, zEnd, zEllipsis, iCol, nToken); 2182 2229 } 2183 2230 } 2184 2231 2185 2232 /* 2186 2233 ** Implementation of the offsets() function for FTS3 2187 2234 */ 2188 2235 static void fts3OffsetsFunc( ................................................................................ 2275 2322 void **ppArg /* Unused */ 2276 2323 ){ 2277 2324 struct Overloaded { 2278 2325 const char *zName; 2279 2326 void (*xFunc)(sqlite3_context*,int,sqlite3_value**); 2280 2327 } aOverload[] = { 2281 2328 { "snippet", fts3SnippetFunc }, 2282 - { "snippet2", fts3Snippet2Func }, 2283 2329 { "offsets", fts3OffsetsFunc }, 2284 2330 { "optimize", fts3OptimizeFunc }, 2285 2331 { "matchinfo", fts3MatchinfoFunc }, 2286 2332 }; 2287 2333 int i; /* Iterator variable */ 2288 2334 2289 2335 UNUSED_PARAMETER(pVtab); ................................................................................ 2425 2471 /* Create the virtual table wrapper around the hash-table and overload 2426 2472 ** the two scalar functions. If this is successful, register the 2427 2473 ** module with sqlite. 2428 2474 */ 2429 2475 if( SQLITE_OK==rc 2430 2476 && SQLITE_OK==(rc = sqlite3Fts3InitHashTable(db, pHash, "fts3_tokenizer")) 2431 2477 && SQLITE_OK==(rc = sqlite3_overload_function(db, "snippet", -1)) 2432 - && SQLITE_OK==(rc = sqlite3_overload_function(db, "snippet2", -1)) 2433 2478 && SQLITE_OK==(rc = sqlite3_overload_function(db, "offsets", 1)) 2434 2479 && SQLITE_OK==(rc = sqlite3_overload_function(db, "matchinfo", -1)) 2435 2480 && SQLITE_OK==(rc = sqlite3_overload_function(db, "optimize", 1)) 2436 2481 ){ 2437 2482 return sqlite3_create_module_v2( 2438 2483 db, "fts3", &fts3Module, (void *)pHash, hashDestroy 2439 2484 );
Changes to ext/fts3/fts3Int.h.
275 275 int sqlite3Fts3GetVarint(const char *, sqlite_int64 *); 276 276 int sqlite3Fts3GetVarint32(const char *, int *); 277 277 int sqlite3Fts3VarintLen(sqlite3_uint64); 278 278 void sqlite3Fts3Dequote(char *); 279 279 280 280 char *sqlite3Fts3FindPositions(Fts3Expr *, sqlite3_int64, int); 281 281 int sqlite3Fts3ExprLoadDoclist(Fts3Table *, Fts3Expr *); 282 +int sqlite3Fts3ExprNearTrim(Fts3Expr *, Fts3Expr *, int); 282 283 283 284 /* fts3_tokenizer.c */ 284 285 const char *sqlite3Fts3NextToken(const char *, int *); 285 286 int sqlite3Fts3InitHashTable(sqlite3 *, Fts3Hash *, const char *); 286 287 int sqlite3Fts3InitTokenizer(Fts3Hash *pHash, 287 288 const char *, sqlite3_tokenizer **, const char **, char ** 288 289 ); 289 290 290 291 /* fts3_snippet.c */ 291 292 void sqlite3Fts3Offsets(sqlite3_context*, Fts3Cursor*); 292 -void sqlite3Fts3Snippet(sqlite3_context*, Fts3Cursor*, 293 - const char *, const char *, const char * 294 -); 295 -void sqlite3Fts3Snippet2(sqlite3_context *, Fts3Cursor *, const char *, 293 +void sqlite3Fts3Snippet(sqlite3_context *, Fts3Cursor *, const char *, 296 294 const char *, const char *, int, int 297 295 ); 298 296 void sqlite3Fts3Matchinfo(sqlite3_context *, Fts3Cursor *); 299 297 300 298 /* fts3_expr.c */ 301 299 int sqlite3Fts3ExprParse(sqlite3_tokenizer *, 302 300 char **, int, int, const char *, int, Fts3Expr **
Changes to ext/fts3/fts3_snippet.c.
14 14 #if !defined(SQLITE_CORE) || defined(SQLITE_ENABLE_FTS3) 15 15 16 16 #include "fts3Int.h" 17 17 #include <string.h> 18 18 #include <assert.h> 19 19 #include <ctype.h> 20 20 21 -typedef struct Snippet Snippet; 22 - 23 -/* 24 -** An instance of the following structure keeps track of generated 25 -** matching-word offset information and snippets. 26 -*/ 27 -struct Snippet { 28 - int nMatch; /* Total number of matches */ 29 - int nAlloc; /* Space allocated for aMatch[] */ 30 - struct snippetMatch { /* One entry for each matching term */ 31 - char snStatus; /* Status flag for use while constructing snippets */ 32 - short int nByte; /* Number of bytes in the term */ 33 - short int iCol; /* The column that contains the match */ 34 - short int iTerm; /* The index in Query.pTerms[] of the matching term */ 35 - int iToken; /* The index of the matching document token */ 36 - int iStart; /* The offset to the first character of the term */ 37 - } *aMatch; /* Points to space obtained from malloc */ 38 - char *zOffset; /* Text rendering of aMatch[] */ 39 - int nOffset; /* strlen(zOffset) */ 40 - char *zSnippet; /* Snippet text */ 41 - int nSnippet; /* strlen(zSnippet) */ 21 + 22 +/* 23 +** Used as an fts3ExprIterate() context when loading phrase doclists to 24 +** Fts3Expr.aDoclist[]/nDoclist. 25 +*/ 26 +typedef struct LoadDoclistCtx LoadDoclistCtx; 27 +struct LoadDoclistCtx { 28 + Fts3Table *pTab; /* FTS3 Table */ 29 + int nPhrase; /* Number of phrases seen so far */ 30 + int nToken; /* Number of tokens seen so far */ 31 +}; 32 + 33 +/* 34 +** The following types are used as part of the implementation of the 35 +** fts3BestSnippet() routine. 36 +*/ 37 +typedef struct SnippetIter SnippetIter; 38 +typedef struct SnippetPhrase SnippetPhrase; 39 +typedef struct SnippetFragment SnippetFragment; 40 + 41 +struct SnippetIter { 42 + Fts3Cursor *pCsr; /* Cursor snippet is being generated from */ 43 + int iCol; /* Extract snippet from this column */ 44 + int nSnippet; /* Requested snippet length (in tokens) */ 45 + int nPhrase; /* Number of phrases in query */ 46 + SnippetPhrase *aPhrase; /* Array of size nPhrase */ 47 + int iCurrent; /* First token of current snippet */ 48 +}; 49 + 50 +struct SnippetPhrase { 51 + int nToken; /* Number of tokens in phrase */ 52 + char *pList; /* Pointer to start of phrase position list */ 53 + int iHead; /* Next value in position list */ 54 + char *pHead; /* Position list data following iHead */ 55 + int iTail; /* Next value in trailing position list */ 56 + char *pTail; /* Position list data following iTail */ 57 +}; 58 + 59 +struct SnippetFragment { 60 + int iCol; /* Column snippet is extracted from */ 61 + int iPos; /* Index of first token in snippet */ 62 + u64 covered; /* Mask of query phrases covered */ 63 + u64 hlmask; /* Mask of snippet terms to highlight */ 64 +}; 65 + 66 +/* 67 +** This type is used as an fts3ExprIterate() context object while 68 +** accumulating the data returned by the matchinfo() function. 69 +*/ 70 +typedef struct MatchInfo MatchInfo; 71 +struct MatchInfo { 72 + Fts3Cursor *pCursor; /* FTS3 Cursor */ 73 + int nCol; /* Number of columns in table */ 74 + u32 *aMatchinfo; /* Pre-allocated buffer */ 75 +}; 76 + 77 + 78 + 79 +/* 80 +** The snippet() and offsets() functions both return text values. An instance 81 +** of the following structure is used to accumulate those values while the 82 +** functions are running. See fts3StringAppend() for details. 83 +*/ 84 +typedef struct StrBuffer StrBuffer; 85 +struct StrBuffer { 86 + char *z; /* Pointer to buffer containing string */ 87 + int n; /* Length of z in bytes (excl. nul-term) */ 88 + int nAlloc; /* Allocated size of buffer z in bytes */ 42 89 }; 43 90 44 91 45 -/* It is not safe to call isspace(), tolower(), or isalnum() on 46 -** hi-bit-set characters. This is the same solution used in the 47 -** tokenizer. 48 -*/ 49 -static int fts3snippetIsspace(char c){ 50 - return (c&0x80)==0 ? isspace(c) : 0; 51 -} 52 - 53 - 54 -/* 55 -** A StringBuffer object holds a zero-terminated string that grows 56 -** arbitrarily by appending. Space to hold the string is obtained 57 -** from sqlite3_malloc(). After any memory allocation failure, 58 -** StringBuffer.z is set to NULL and no further allocation is attempted. 59 -*/ 60 -typedef struct StringBuffer { 61 - char *z; /* Text of the string. Space from malloc. */ 62 - int nUsed; /* Number bytes of z[] used, not counting \000 terminator */ 63 - int nAlloc; /* Bytes allocated for z[] */ 64 -} StringBuffer; 65 - 66 - 67 -/* 68 -** Initialize a new StringBuffer. 69 -*/ 70 -static void fts3SnippetSbInit(StringBuffer *p){ 71 - p->nAlloc = 100; 72 - p->nUsed = 0; 73 - p->z = sqlite3_malloc( p->nAlloc ); 74 -} 75 - 76 -/* 77 -** Append text to the string buffer. 78 -*/ 79 -static void fts3SnippetAppend(StringBuffer *p, const char *zNew, int nNew){ 80 - if( p->z==0 ) return; 81 - if( nNew<0 ) nNew = (int)strlen(zNew); 82 - if( p->nUsed + nNew >= p->nAlloc ){ 83 - int nAlloc; 84 - char *zNew; 85 - 86 - nAlloc = p->nUsed + nNew + p->nAlloc; 87 - zNew = sqlite3_realloc(p->z, nAlloc); 88 - if( zNew==0 ){ 89 - sqlite3_free(p->z); 90 - p->z = 0; 91 - return; 92 - } 93 - p->z = zNew; 94 - p->nAlloc = nAlloc; 95 - } 96 - memcpy(&p->z[p->nUsed], zNew, nNew); 97 - p->nUsed += nNew; 98 - p->z[p->nUsed] = 0; 99 -} 100 - 101 -/* If the StringBuffer ends in something other than white space, add a 102 -** single space character to the end. 103 -*/ 104 -static void fts3SnippetAppendWhiteSpace(StringBuffer *p){ 105 - if( p->z && p->nUsed && !fts3snippetIsspace(p->z[p->nUsed-1]) ){ 106 - fts3SnippetAppend(p, " ", 1); 107 - } 108 -} 109 - 110 -/* Remove white space from the end of the StringBuffer */ 111 -static void fts3SnippetTrimWhiteSpace(StringBuffer *p){ 112 - if( p->z ){ 113 - while( p->nUsed && fts3snippetIsspace(p->z[p->nUsed-1]) ){ 114 - p->nUsed--; 115 - } 116 - p->z[p->nUsed] = 0; 117 - } 118 -} 119 - 120 -/* 121 -** Release all memory associated with the Snippet structure passed as 122 -** an argument. 123 -*/ 124 -static void fts3SnippetFree(Snippet *p){ 125 - if( p ){ 126 - sqlite3_free(p->aMatch); 127 - sqlite3_free(p->zOffset); 128 - sqlite3_free(p->zSnippet); 129 - sqlite3_free(p); 130 - } 131 -} 132 - 133 -/* 134 -** Append a single entry to the p->aMatch[] log. 135 -*/ 136 -static int snippetAppendMatch( 137 - Snippet *p, /* Append the entry to this snippet */ 138 - int iCol, int iTerm, /* The column and query term */ 139 - int iToken, /* Matching token in document */ 140 - int iStart, int nByte /* Offset and size of the match */ 141 -){ 142 - int i; 143 - struct snippetMatch *pMatch; 144 - if( p->nMatch+1>=p->nAlloc ){ 145 - struct snippetMatch *pNew; 146 - p->nAlloc = p->nAlloc*2 + 10; 147 - pNew = sqlite3_realloc(p->aMatch, p->nAlloc*sizeof(p->aMatch[0]) ); 148 - if( pNew==0 ){ 149 - p->aMatch = 0; 150 - p->nMatch = 0; 151 - p->nAlloc = 0; 152 - return SQLITE_NOMEM; 153 - } 154 - p->aMatch = pNew; 155 - } 156 - i = p->nMatch++; 157 - pMatch = &p->aMatch[i]; 158 - pMatch->iCol = (short)iCol; 159 - pMatch->iTerm = (short)iTerm; 160 - pMatch->iToken = iToken; 161 - pMatch->iStart = iStart; 162 - pMatch->nByte = (short)nByte; 163 - return SQLITE_OK; 164 -} 165 - 166 -/* 167 -** Sizing information for the circular buffer used in snippetOffsetsOfColumn() 168 -*/ 169 -#define FTS3_ROTOR_SZ (32) 170 -#define FTS3_ROTOR_MASK (FTS3_ROTOR_SZ-1) 171 - 172 -/* 173 -** Function to iterate through the tokens of a compiled expression. 174 -** 175 -** Except, skip all tokens on the right-hand side of a NOT operator. 176 -** This function is used to find tokens as part of snippet and offset 177 -** generation and we do nt want snippets and offsets to report matches 178 -** for tokens on the RHS of a NOT. 179 -*/ 180 -static int fts3NextExprToken(Fts3Expr **ppExpr, int *piToken){ 181 - Fts3Expr *p = *ppExpr; 182 - int iToken = *piToken; 183 - if( iToken<0 ){ 184 - /* In this case the expression p is the root of an expression tree. 185 - ** Move to the first token in the expression tree. 186 - */ 187 - while( p->pLeft ){ 188 - p = p->pLeft; 189 - } 190 - iToken = 0; 191 - }else{ 192 - assert(p && p->eType==FTSQUERY_PHRASE ); 193 - if( iToken<(p->pPhrase->nToken-1) ){ 194 - iToken++; 195 - }else{ 196 - iToken = 0; 197 - while( p->pParent && p->pParent->pLeft!=p ){ 198 - assert( p->pParent->pRight==p ); 199 - p = p->pParent; 200 - } 201 - p = p->pParent; 202 - if( p ){ 203 - assert( p->pRight!=0 ); 204 - p = p->pRight; 205 - while( p->pLeft ){ 206 - p = p->pLeft; 207 - } 208 - } 209 - } 210 - } 211 - 212 - *ppExpr = p; 213 - *piToken = iToken; 214 - return p?1:0; 215 -} 216 - 217 -/* 218 -** Return TRUE if the expression node pExpr is located beneath the 219 -** RHS of a NOT operator. 220 -*/ 221 -static int fts3ExprBeneathNot(Fts3Expr *p){ 222 - Fts3Expr *pParent; 223 - while( p ){ 224 - pParent = p->pParent; 225 - if( pParent && pParent->eType==FTSQUERY_NOT && pParent->pRight==p ){ 226 - return 1; 227 - } 228 - p = pParent; 229 - } 230 - return 0; 231 -} 232 - 233 -/* 234 -** Add entries to pSnippet->aMatch[] for every match that occurs against 235 -** document zDoc[0..nDoc-1] which is stored in column iColumn. 236 -*/ 237 -static int snippetOffsetsOfColumn( 238 - Fts3Cursor *pCur, /* The fulltest search cursor */ 239 - Snippet *pSnippet, /* The Snippet object to be filled in */ 240 - int iColumn, /* Index of fulltext table column */ 241 - const char *zDoc, /* Text of the fulltext table column */ 242 - int nDoc /* Length of zDoc in bytes */ 243 -){ 244 - const sqlite3_tokenizer_module *pTModule; /* The tokenizer module */ 245 - sqlite3_tokenizer *pTokenizer; /* The specific tokenizer */ 246 - sqlite3_tokenizer_cursor *pTCursor; /* Tokenizer cursor */ 247 - Fts3Table *pVtab; /* The full text index */ 248 - int nColumn; /* Number of columns in the index */ 249 - int i, j; /* Loop counters */ 250 - int rc; /* Return code */ 251 - unsigned int match, prevMatch; /* Phrase search bitmasks */ 252 - const char *zToken; /* Next token from the tokenizer */ 253 - int nToken; /* Size of zToken */ 254 - int iBegin, iEnd, iPos; /* Offsets of beginning and end */ 255 - 256 - /* The following variables keep a circular buffer of the last 257 - ** few tokens */ 258 - unsigned int iRotor = 0; /* Index of current token */ 259 - int iRotorBegin[FTS3_ROTOR_SZ]; /* Beginning offset of token */ 260 - int iRotorLen[FTS3_ROTOR_SZ]; /* Length of token */ 261 - 262 - pVtab = (Fts3Table *)pCur->base.pVtab; 263 - nColumn = pVtab->nColumn; 264 - pTokenizer = pVtab->pTokenizer; 265 - pTModule = pTokenizer->pModule; 266 - rc = pTModule->xOpen(pTokenizer, zDoc, nDoc, &pTCursor); 267 - if( rc ) return rc; 268 - pTCursor->pTokenizer = pTokenizer; 269 - 270 - prevMatch = 0; 271 - while( (rc = pTModule->xNext(pTCursor, &zToken, &nToken, 272 - &iBegin, &iEnd, &iPos))==SQLITE_OK ){ 273 - Fts3Expr *pIter = pCur->pExpr; 274 - int iIter = -1; 275 - iRotorBegin[iRotor&FTS3_ROTOR_MASK] = iBegin; 276 - iRotorLen[iRotor&FTS3_ROTOR_MASK] = iEnd-iBegin; 277 - match = 0; 278 - for(i=0; i<(FTS3_ROTOR_SZ-1) && fts3NextExprToken(&pIter, &iIter); i++){ 279 - int nPhrase; /* Number of tokens in current phrase */ 280 - struct PhraseToken *pToken; /* Current token */ 281 - int iCol; /* Column index */ 282 - 283 - if( fts3ExprBeneathNot(pIter) ) continue; 284 - nPhrase = pIter->pPhrase->nToken; 285 - pToken = &pIter->pPhrase->aToken[iIter]; 286 - iCol = pIter->pPhrase->iColumn; 287 - if( iCol>=0 && iCol<nColumn && iCol!=iColumn ) continue; 288 - if( pToken->n>nToken ) continue; 289 - if( !pToken->isPrefix && pToken->n<nToken ) continue; 290 - assert( pToken->n<=nToken ); 291 - if( memcmp(pToken->z, zToken, pToken->n) ) continue; 292 - if( iIter>0 && (prevMatch & (1<<i))==0 ) continue; 293 - match |= 1<<i; 294 - if( i==(FTS3_ROTOR_SZ-2) || nPhrase==iIter+1 ){ 295 - for(j=nPhrase-1; j>=0; j--){ 296 - int k = (iRotor-j) & FTS3_ROTOR_MASK; 297 - rc = snippetAppendMatch(pSnippet, iColumn, i-j, iPos-j, 298 - iRotorBegin[k], iRotorLen[k]); 299 - if( rc ) goto end_offsets_of_column; 300 - } 301 - } 302 - } 303 - prevMatch = match<<1; 304 - iRotor++; 305 - } 306 -end_offsets_of_column: 307 - pTModule->xClose(pTCursor); 308 - return rc==SQLITE_DONE ? SQLITE_OK : rc; 309 -} 310 - 311 -/* 312 -** Remove entries from the pSnippet structure to account for the NEAR 313 -** operator. When this is called, pSnippet contains the list of token 314 -** offsets produced by treating all NEAR operators as AND operators. 315 -** This function removes any entries that should not be present after 316 -** accounting for the NEAR restriction. For example, if the queried 317 -** document is: 318 -** 319 -** "A B C D E A" 320 -** 321 -** and the query is: 322 -** 323 -** A NEAR/0 E 324 -** 325 -** then when this function is called the Snippet contains token offsets 326 -** 0, 4 and 5. This function removes the "0" entry (because the first A 327 -** is not near enough to an E). 328 -** 329 -** When this function is called, the value pointed to by parameter piLeft is 330 -** the integer id of the left-most token in the expression tree headed by 331 -** pExpr. This function increments *piLeft by the total number of tokens 332 -** in the expression tree headed by pExpr. 333 -** 334 -** Return 1 if any trimming occurs. Return 0 if no trimming is required. 335 -*/ 336 -static int trimSnippetOffsets( 337 - Fts3Expr *pExpr, /* The search expression */ 338 - Snippet *pSnippet, /* The set of snippet offsets to be trimmed */ 339 - int *piLeft /* Index of left-most token in pExpr */ 340 -){ 341 - if( pExpr ){ 342 - if( trimSnippetOffsets(pExpr->pLeft, pSnippet, piLeft) ){ 343 - return 1; 344 - } 345 - 346 - switch( pExpr->eType ){ 347 - case FTSQUERY_PHRASE: 348 - *piLeft += pExpr->pPhrase->nToken; 349 - break; 350 - case FTSQUERY_NEAR: { 351 - /* The right-hand-side of a NEAR operator is always a phrase. The 352 - ** left-hand-side is either a phrase or an expression tree that is 353 - ** itself headed by a NEAR operator. The following initializations 354 - ** set local variable iLeft to the token number of the left-most 355 - ** token in the right-hand phrase, and iRight to the right most 356 - ** token in the same phrase. For example, if we had: 357 - ** 358 - ** <col> MATCH '"abc def" NEAR/2 "ghi jkl"' 359 - ** 360 - ** then iLeft will be set to 2 (token number of ghi) and nToken will 361 - ** be set to 4. 362 - */ 363 - Fts3Expr *pLeft = pExpr->pLeft; 364 - Fts3Expr *pRight = pExpr->pRight; 365 - int iLeft = *piLeft; 366 - int nNear = pExpr->nNear; 367 - int nToken = pRight->pPhrase->nToken; 368 - int jj, ii; 369 - if( pLeft->eType==FTSQUERY_NEAR ){ 370 - pLeft = pLeft->pRight; 371 - } 372 - assert( pRight->eType==FTSQUERY_PHRASE ); 373 - assert( pLeft->eType==FTSQUERY_PHRASE ); 374 - nToken += pLeft->pPhrase->nToken; 375 - 376 - for(ii=0; ii<pSnippet->nMatch; ii++){ 377 - struct snippetMatch *p = &pSnippet->aMatch[ii]; 378 - if( p->iTerm==iLeft ){ 379 - int isOk = 0; 380 - /* Snippet ii is an occurence of query term iLeft in the document. 381 - ** It occurs at position (p->iToken) of the document. We now 382 - ** search for an instance of token (iLeft-1) somewhere in the 383 - ** range (p->iToken - nNear)...(p->iToken + nNear + nToken) within 384 - ** the set of snippetMatch structures. If one is found, proceed. 385 - ** If one cannot be found, then remove snippets ii..(ii+N-1) 386 - ** from the matching snippets, where N is the number of tokens 387 - ** in phrase pRight->pPhrase. 388 - */ 389 - for(jj=0; isOk==0 && jj<pSnippet->nMatch; jj++){ 390 - struct snippetMatch *p2 = &pSnippet->aMatch[jj]; 391 - if( p2->iTerm==(iLeft-1) ){ 392 - if( p2->iToken>=(p->iToken-nNear-1) 393 - && p2->iToken<(p->iToken+nNear+nToken) 394 - ){ 395 - isOk = 1; 396 - } 397 - } 398 - } 399 - if( !isOk ){ 400 - int kk; 401 - for(kk=0; kk<pRight->pPhrase->nToken; kk++){ 402 - pSnippet->aMatch[kk+ii].iTerm = -2; 403 - } 404 - return 1; 405 - } 406 - } 407 - if( p->iTerm==(iLeft-1) ){ 408 - int isOk = 0; 409 - for(jj=0; isOk==0 && jj<pSnippet->nMatch; jj++){ 410 - struct snippetMatch *p2 = &pSnippet->aMatch[jj]; 411 - if( p2->iTerm==iLeft ){ 412 - if( p2->iToken<=(p->iToken+nNear+1) 413 - && p2->iToken>(p->iToken-nNear-nToken) 414 - ){ 415 - isOk = 1; 416 - } 417 - } 418 - } 419 - if( !isOk ){ 420 - int kk; 421 - for(kk=0; kk<pLeft->pPhrase->nToken; kk++){ 422 - pSnippet->aMatch[ii-kk].iTerm = -2; 423 - } 424 - return 1; 425 - } 426 - } 427 - } 428 - break; 429 - } 430 - } 431 - 432 - if( trimSnippetOffsets(pExpr->pRight, pSnippet, piLeft) ){ 433 - return 1; 434 - } 435 - } 436 - return 0; 437 -} 438 - 439 -/* 440 -** Compute all offsets for the current row of the query. 441 -** If the offsets have already been computed, this routine is a no-op. 442 -*/ 443 -static int snippetAllOffsets(Fts3Cursor *pCsr, Snippet **ppSnippet){ 444 - Fts3Table *p = (Fts3Table *)pCsr->base.pVtab; /* The FTS3 virtual table */ 445 - int nColumn; /* Number of columns. Docid does count */ 446 - int iColumn; /* Index of of a column */ 447 - int i; /* Loop index */ 448 - int iFirst; /* First column to search */ 449 - int iLast; /* Last coumn to search */ 450 - int iTerm = 0; 451 - Snippet *pSnippet; 452 - int rc = SQLITE_OK; 453 - 454 - if( pCsr->pExpr==0 ){ 455 - return SQLITE_OK; 456 - } 457 - 458 - pSnippet = (Snippet *)sqlite3_malloc(sizeof(Snippet)); 459 - *ppSnippet = pSnippet; 460 - if( !pSnippet ){ 461 - return SQLITE_NOMEM; 462 - } 463 - memset(pSnippet, 0, sizeof(Snippet)); 464 - 465 - nColumn = p->nColumn; 466 - iColumn = (pCsr->eSearch - 2); 467 - if( iColumn<0 || iColumn>=nColumn ){ 468 - /* Look for matches over all columns of the full-text index */ 469 - iFirst = 0; 470 - iLast = nColumn-1; 471 - }else{ 472 - /* Look for matches in the iColumn-th column of the index only */ 473 - iFirst = iColumn; 474 - iLast = iColumn; 475 - } 476 - for(i=iFirst; rc==SQLITE_OK && i<=iLast; i++){ 477 - const char *zDoc; 478 - int nDoc; 479 - zDoc = (const char*)sqlite3_column_text(pCsr->pStmt, i+1); 480 - nDoc = sqlite3_column_bytes(pCsr->pStmt, i+1); 481 - if( zDoc==0 && sqlite3_column_type(pCsr->pStmt, i+1)!=SQLITE_NULL ){ 482 - rc = SQLITE_NOMEM; 483 - }else{ 484 - rc = snippetOffsetsOfColumn(pCsr, pSnippet, i, zDoc, nDoc); 485 - } 486 - } 487 - 488 - while( trimSnippetOffsets(pCsr->pExpr, pSnippet, &iTerm) ){ 489 - iTerm = 0; 490 - } 491 - 492 - return rc; 493 -} 494 - 495 -/* 496 -** Convert the information in the aMatch[] array of the snippet 497 -** into the string zOffset[0..nOffset-1]. This string is used as 498 -** the return of the SQL offsets() function. 499 -*/ 500 -static void snippetOffsetText(Snippet *p){ 501 - int i; 502 - int cnt = 0; 503 - StringBuffer sb; 504 - char zBuf[200]; 505 - if( p->zOffset ) return; 506 - fts3SnippetSbInit(&sb); 507 - for(i=0; i<p->nMatch; i++){ 508 - struct snippetMatch *pMatch = &p->aMatch[i]; 509 - if( pMatch->iTerm>=0 ){ 510 - /* If snippetMatch.iTerm is less than 0, then the match was 511 - ** discarded as part of processing the NEAR operator (see the 512 - ** trimSnippetOffsetsForNear() function for details). Ignore 513 - ** it in this case 514 - */ 515 - zBuf[0] = ' '; 516 - sqlite3_snprintf(sizeof(zBuf)-1, &zBuf[cnt>0], "%d %d %d %d", 517 - pMatch->iCol, pMatch->iTerm, pMatch->iStart, pMatch->nByte); 518 - fts3SnippetAppend(&sb, zBuf, -1); 519 - cnt++; 520 - } 521 - } 522 - p->zOffset = sb.z; 523 - p->nOffset = sb.z ? sb.nUsed : 0; 524 -} 525 - 526 -/* 527 -** zDoc[0..nDoc-1] is phrase of text. aMatch[0..nMatch-1] are a set 528 -** of matching words some of which might be in zDoc. zDoc is column 529 -** number iCol. 530 -** 531 -** iBreak is suggested spot in zDoc where we could begin or end an 532 -** excerpt. Return a value similar to iBreak but possibly adjusted 533 -** to be a little left or right so that the break point is better. 534 -*/ 535 -static int wordBoundary( 536 - int iBreak, /* The suggested break point */ 537 - const char *zDoc, /* Document text */ 538 - int nDoc, /* Number of bytes in zDoc[] */ 539 - struct snippetMatch *aMatch, /* Matching words */ 540 - int nMatch, /* Number of entries in aMatch[] */ 541 - int iCol /* The column number for zDoc[] */ 542 -){ 543 - int i; 544 - if( iBreak<=10 ){ 545 - return 0; 546 - } 547 - if( iBreak>=nDoc-10 ){ 548 - return nDoc; 549 - } 550 - for(i=0; ALWAYS(i<nMatch) && aMatch[i].iCol<iCol; i++){} 551 - while( i<nMatch && aMatch[i].iStart+aMatch[i].nByte<iBreak ){ i++; } 552 - if( i<nMatch ){ 553 - if( aMatch[i].iStart<iBreak+10 ){ 554 - return aMatch[i].iStart; 555 - } 556 - if( i>0 && aMatch[i-1].iStart+aMatch[i-1].nByte>=iBreak ){ 557 - return aMatch[i-1].iStart; 558 - } 559 - } 560 - for(i=1; i<=10; i++){ 561 - if( fts3snippetIsspace(zDoc[iBreak-i]) ){ 562 - return iBreak - i + 1; 563 - } 564 - if( fts3snippetIsspace(zDoc[iBreak+i]) ){ 565 - return iBreak + i + 1; 566 - } 567 - } 568 - return iBreak; 569 -} 570 - 571 - 572 - 573 -/* 574 -** Allowed values for Snippet.aMatch[].snStatus 575 -*/ 576 -#define SNIPPET_IGNORE 0 /* It is ok to omit this match from the snippet */ 577 -#define SNIPPET_DESIRED 1 /* We want to include this match in the snippet */ 578 - 579 -/* 580 -** Generate the text of a snippet. 581 -*/ 582 -static void snippetText( 583 - Fts3Cursor *pCursor, /* The cursor we need the snippet for */ 584 - Snippet *pSnippet, 585 - const char *zStartMark, /* Markup to appear before each match */ 586 - const char *zEndMark, /* Markup to appear after each match */ 587 - const char *zEllipsis /* Ellipsis mark */ 588 -){ 589 - int i, j; 590 - struct snippetMatch *aMatch; 591 - int nMatch; 592 - int nDesired; 593 - StringBuffer sb; 594 - int tailCol; 595 - int tailOffset; 596 - int iCol; 597 - int nDoc; 598 - const char *zDoc; 599 - int iStart, iEnd; 600 - int tailEllipsis = 0; 601 - int iMatch; 602 - 603 - 604 - sqlite3_free(pSnippet->zSnippet); 605 - pSnippet->zSnippet = 0; 606 - aMatch = pSnippet->aMatch; 607 - nMatch = pSnippet->nMatch; 608 - fts3SnippetSbInit(&sb); 609 - 610 - for(i=0; i<nMatch; i++){ 611 - aMatch[i].snStatus = SNIPPET_IGNORE; 612 - } 613 - nDesired = 0; 614 - for(i=0; i<FTS3_ROTOR_SZ; i++){ 615 - for(j=0; j<nMatch; j++){ 616 - if( aMatch[j].iTerm==i ){ 617 - aMatch[j].snStatus = SNIPPET_DESIRED; 618 - nDesired++; 619 - break; 620 - } 621 - } 622 - } 623 - 624 - iMatch = 0; 625 - tailCol = -1; 626 - tailOffset = 0; 627 - for(i=0; i<nMatch && nDesired>0; i++){ 628 - if( aMatch[i].snStatus!=SNIPPET_DESIRED ) continue; 629 - nDesired--; 630 - iCol = aMatch[i].iCol; 631 - zDoc = (const char*)sqlite3_column_text(pCursor->pStmt, iCol+1); 632 - nDoc = sqlite3_column_bytes(pCursor->pStmt, iCol+1); 633 - iStart = aMatch[i].iStart - 40; 634 - iStart = wordBoundary(iStart, zDoc, nDoc, aMatch, nMatch, iCol); 635 - if( iStart<=10 ){ 636 - iStart = 0; 637 - } 638 - if( iCol==tailCol && iStart<=tailOffset+20 ){ 639 - iStart = tailOffset; 640 - } 641 - if( (iCol!=tailCol && tailCol>=0) || iStart!=tailOffset ){ 642 - fts3SnippetTrimWhiteSpace(&sb); 643 - fts3SnippetAppendWhiteSpace(&sb); 644 - fts3SnippetAppend(&sb, zEllipsis, -1); 645 - fts3SnippetAppendWhiteSpace(&sb); 646 - } 647 - iEnd = aMatch[i].iStart + aMatch[i].nByte + 40; 648 - iEnd = wordBoundary(iEnd, zDoc, nDoc, aMatch, nMatch, iCol); 649 - if( iEnd>=nDoc-10 ){ 650 - iEnd = nDoc; 651 - tailEllipsis = 0; 652 - }else{ 653 - tailEllipsis = 1; 654 - } 655 - while( iMatch<nMatch && aMatch[iMatch].iCol<iCol ){ iMatch++; } 656 - while( iStart<iEnd ){ 657 - while( iMatch<nMatch && aMatch[iMatch].iStart<iStart 658 - && aMatch[iMatch].iCol<=iCol ){ 659 - iMatch++; 660 - } 661 - if( iMatch<nMatch && aMatch[iMatch].iStart<iEnd 662 - && aMatch[iMatch].iCol==iCol ){ 663 - fts3SnippetAppend(&sb, &zDoc[iStart], aMatch[iMatch].iStart - iStart); 664 - iStart = aMatch[iMatch].iStart; 665 - fts3SnippetAppend(&sb, zStartMark, -1); 666 - fts3SnippetAppend(&sb, &zDoc[iStart], aMatch[iMatch].nByte); 667 - fts3SnippetAppend(&sb, zEndMark, -1); 668 - iStart += aMatch[iMatch].nByte; 669 - for(j=iMatch+1; j<nMatch; j++){ 670 - if( aMatch[j].iTerm==aMatch[iMatch].iTerm 671 - && aMatch[j].snStatus==SNIPPET_DESIRED ){ 672 - nDesired--; 673 - aMatch[j].snStatus = SNIPPET_IGNORE; 674 - } 675 - } 676 - }else{ 677 - fts3SnippetAppend(&sb, &zDoc[iStart], iEnd - iStart); 678 - iStart = iEnd; 679 - } 680 - } 681 - tailCol = iCol; 682 - tailOffset = iEnd; 683 - } 684 - fts3SnippetTrimWhiteSpace(&sb); 685 - if( tailEllipsis ){ 686 - fts3SnippetAppendWhiteSpace(&sb); 687 - fts3SnippetAppend(&sb, zEllipsis, -1); 688 - } 689 - pSnippet->zSnippet = sb.z; 690 - pSnippet->nSnippet = sb.z ? sb.nUsed : 0; 691 -} 692 - 693 -void sqlite3Fts3Offsets( 694 - sqlite3_context *pCtx, /* SQLite function call context */ 695 - Fts3Cursor *pCsr /* Cursor object */ 696 -){ 697 - Snippet *p; /* Snippet structure */ 698 - int rc = snippetAllOffsets(pCsr, &p); 699 - if( rc==SQLITE_OK ){ 700 - snippetOffsetText(p); 701 - if( p->zOffset ){ 702 - sqlite3_result_text(pCtx, p->zOffset, p->nOffset, SQLITE_TRANSIENT); 703 - }else{ 704 - sqlite3_result_error_nomem(pCtx); 705 - } 706 - }else{ 707 - sqlite3_result_error_nomem(pCtx); 708 - } 709 - fts3SnippetFree(p); 710 -} 711 - 712 -void sqlite3Fts3Snippet( 713 - sqlite3_context *pCtx, /* SQLite function call context */ 714 - Fts3Cursor *pCsr, /* Cursor object */ 715 - const char *zStart, /* Snippet start text - "<b>" */ 716 - const char *zEnd, /* Snippet end text - "</b>" */ 717 - const char *zEllipsis /* Snippet ellipsis text - "<b>...</b>" */ 718 -){ 719 - Snippet *p; /* Snippet structure */ 720 - int rc = snippetAllOffsets(pCsr, &p); 721 - if( rc==SQLITE_OK ){ 722 - snippetText(pCsr, p, zStart, zEnd, zEllipsis); 723 - if( p->zSnippet ){ 724 - sqlite3_result_text(pCtx, p->zSnippet, p->nSnippet, SQLITE_TRANSIENT); 725 - }else{ 726 - sqlite3_result_error_nomem(pCtx); 727 - } 728 - }else{ 729 - sqlite3_result_error_nomem(pCtx); 730 - } 731 - fts3SnippetFree(p); 732 -} 733 - 734 -/************************************************************************* 735 -** Below this point is the alternative, experimental snippet() implementation. 736 -*/ 737 - 738 -#define SNIPPET_BUFFER_CHUNK 64 739 -#define SNIPPET_BUFFER_SIZE SNIPPET_BUFFER_CHUNK*4 740 -#define SNIPPET_BUFFER_MASK (SNIPPET_BUFFER_SIZE-1) 741 - 92 +/* 93 +** This function is used to help iterate through a position-list. A position 94 +** list is a list of unique integers, sorted from smallest to largest. Each 95 +** element of the list is represented by an FTS3 varint that takes the value 96 +** of the difference between the current element and the previous one plus 97 +** two. For example, to store the position-list: 98 +** 99 +** 4 9 113 100 +** 101 +** the three varints: 102 +** 103 +** 6 7 106 104 +** 105 +** are encoded. 106 +** 107 +** When this function is called, *pp points to the start of an element of 108 +** the list. *piPos contains the value of the previous entry in the list. 109 +** After it returns, *piPos contains the value of the next element of the 110 +** list and *pp is advanced to the following varint. 111 +*/ 742 112 static void fts3GetDeltaPosition(char **pp, int *piPos){ 743 113 int iVal; 744 114 *pp += sqlite3Fts3GetVarint32(*pp, &iVal); 745 115 *piPos += (iVal-2); 746 116 } 117 + 118 +/* 119 +** Helper function for fts3ExprIterate() (see below). 120 +*/ 121 +static int fts3ExprIterate2( 122 + Fts3Expr *pExpr, /* Expression to iterate phrases of */ 123 + int *piPhrase, /* Pointer to phrase counter */ 124 + int (*x)(Fts3Expr*,int,void*), /* Callback function to invoke for phrases */ 125 + void *pCtx /* Second argument to pass to callback */ 126 +){ 127 + int rc; /* Return code */ 128 + int eType = pExpr->eType; /* Type of expression node pExpr */ 129 + 130 + if( eType!=FTSQUERY_PHRASE ){ 131 + assert( pExpr->pLeft && pExpr->pRight ); 132 + rc = fts3ExprIterate2(pExpr->pLeft, piPhrase, x, pCtx); 133 + if( rc==SQLITE_OK && eType!=FTSQUERY_NOT ){ 134 + rc = fts3ExprIterate2(pExpr->pRight, piPhrase, x, pCtx); 135 + } 136 + }else{ 137 + rc = x(pExpr, *piPhrase, pCtx); 138 + (*piPhrase)++; 139 + } 140 + return rc; 141 +} 747 142 748 143 /* 749 144 ** Iterate through all phrase nodes in an FTS3 query, except those that 750 145 ** are part of a sub-tree that is the right-hand-side of a NOT operator. 751 146 ** For each phrase node found, the supplied callback function is invoked. 752 147 ** 753 148 ** If the callback function returns anything other than SQLITE_OK, 754 149 ** the iteration is abandoned and the error code returned immediately. 755 150 ** Otherwise, SQLITE_OK is returned after a callback has been made for 756 151 ** all eligible phrase nodes. 757 152 */ 758 153 static int fts3ExprIterate( 759 154 Fts3Expr *pExpr, /* Expression to iterate phrases of */ 760 - int (*x)(Fts3Expr *, void *), /* Callback function to invoke for phrases */ 155 + int (*x)(Fts3Expr*,int,void*), /* Callback function to invoke for phrases */ 761 156 void *pCtx /* Second argument to pass to callback */ 762 157 ){ 763 - int rc; 764 - int eType = pExpr->eType; 765 - if( eType==FTSQUERY_NOT ){ 766 - rc = SQLITE_OK; 767 - }else if( eType!=FTSQUERY_PHRASE ){ 768 - assert( pExpr->pLeft && pExpr->pRight ); 769 - rc = fts3ExprIterate(pExpr->pLeft, x, pCtx); 770 - if( rc==SQLITE_OK ){ 771 - rc = fts3ExprIterate(pExpr->pRight, x, pCtx); 772 - } 773 - }else{ 774 - rc = x(pExpr, pCtx); 775 - } 158 + int iPhrase = 0; /* Variable used as the phrase counter */ 159 + return fts3ExprIterate2(pExpr, &iPhrase, x, pCtx); 160 +} 161 + 162 +/* 163 +** The argument to this function is always a phrase node. Its doclist 164 +** (Fts3Expr.aDoclist[]) and the doclists associated with all phrase nodes 165 +** to the left of this one in the query tree have already been loaded. 166 +** 167 +** If this phrase node is part of a series of phrase nodes joined by 168 +** NEAR operators (and is not the left-most of said series), then elements are 169 +** removed from the phrases doclist consistent with the NEAR restriction. If 170 +** required, elements may be removed from the doclists of phrases to the 171 +** left of this one that are part of the same series of NEAR operator 172 +** connected phrases. 173 +** 174 +** If an OOM error occurs, SQLITE_NOMEM is returned. Otherwise, SQLITE_OK. 175 +*/ 176 +static int fts3ExprNearTrim(Fts3Expr *pExpr){ 177 + int rc = SQLITE_OK; 178 + Fts3Expr *pParent = pExpr->pParent; 179 + 180 + assert( pExpr->eType==FTSQUERY_PHRASE ); 181 + while( rc==SQLITE_OK 182 + && pParent 183 + && pParent->eType==FTSQUERY_NEAR 184 + && pParent->pRight==pExpr 185 + ){ 186 + /* This expression (pExpr) is the right-hand-side of a NEAR operator. 187 + ** Find the expression to the left of the same operator. 188 + */ 189 + int nNear = pParent->nNear; 190 + Fts3Expr *pLeft = pParent->pLeft; 191 + 192 + if( pLeft->eType!=FTSQUERY_PHRASE ){ 193 + assert( pLeft->eType==FTSQUERY_NEAR ); 194 + assert( pLeft->pRight->eType==FTSQUERY_PHRASE ); 195 + pLeft = pLeft->pRight; 196 + } 197 + 198 + rc = sqlite3Fts3ExprNearTrim(pLeft, pExpr, nNear); 199 + 200 + pExpr = pLeft; 201 + pParent = pExpr->pParent; 202 + } 203 + 776 204 return rc; 777 205 } 778 206 779 -typedef struct LoadDoclistCtx LoadDoclistCtx; 780 -struct LoadDoclistCtx { 781 - Fts3Table *pTab; /* FTS3 Table */ 782 - int nPhrase; /* Number of phrases so far */ 783 -}; 784 - 785 -static int fts3ExprLoadDoclistsCb(Fts3Expr *pExpr, void *ctx){ 207 +/* 208 +** This is an fts3ExprIterate() callback used while loading the doclists 209 +** for each phrase into Fts3Expr.aDoclist[]/nDoclist. See also 210 +** fts3ExprLoadDoclists(). 211 +*/ 212 +static int fts3ExprLoadDoclistsCb1(Fts3Expr *pExpr, int iPhrase, void *ctx){ 786 213 int rc = SQLITE_OK; 787 214 LoadDoclistCtx *p = (LoadDoclistCtx *)ctx; 215 + 788 216 p->nPhrase++; 217 + p->nToken += pExpr->pPhrase->nToken; 218 + 789 219 if( pExpr->isLoaded==0 ){ 790 220 rc = sqlite3Fts3ExprLoadDoclist(p->pTab, pExpr); 791 221 pExpr->isLoaded = 1; 792 - if( rc==SQLITE_OK && pExpr->aDoclist ){ 793 - pExpr->pCurrent = pExpr->aDoclist; 794 - pExpr->pCurrent += sqlite3Fts3GetVarint(pExpr->pCurrent,&pExpr->iCurrent); 222 + if( rc==SQLITE_OK ){ 223 + rc = fts3ExprNearTrim(pExpr); 795 224 } 796 225 } 226 + 797 227 return rc; 798 228 } 799 229 800 -static int fts3ExprLoadDoclists(Fts3Cursor *pCsr, int *pnPhrase){ 801 - int rc; 802 - LoadDoclistCtx sCtx = {0, 0}; 230 +/* 231 +** This is an fts3ExprIterate() callback used while loading the doclists 232 +** for each phrase into Fts3Expr.aDoclist[]/nDoclist. See also 233 +** fts3ExprLoadDoclists(). 234 +*/ 235 +static int fts3ExprLoadDoclistsCb2(Fts3Expr *pExpr, int iPhrase, void *ctx){ 236 + if( pExpr->aDoclist ){ 237 + pExpr->pCurrent = pExpr->aDoclist; 238 + pExpr->iCurrent = 0; 239 + pExpr->pCurrent += sqlite3Fts3GetVarint(pExpr->pCurrent, &pExpr->iCurrent); 240 + } 241 + return SQLITE_OK; 242 +} 243 + 244 +/* 245 +** Load the doclists for each phrase in the query associated with FTS3 cursor 246 +** pCsr. 247 +** 248 +** If pnPhrase is not NULL, then *pnPhrase is set to the number of matchable 249 +** phrases in the expression (all phrases except those directly or 250 +** indirectly descended from the right-hand-side of a NOT operator). If 251 +** pnToken is not NULL, then it is set to the number of tokens in all 252 +** matchable phrases of the expression. 253 +*/ 254 +static int fts3ExprLoadDoclists( 255 + Fts3Cursor *pCsr, /* Fts3 cursor for current query */ 256 + int *pnPhrase, /* OUT: Number of phrases in query */ 257 + int *pnToken /* OUT: Number of tokens in query */ 258 +){ 259 + int rc; /* Return Code */ 260 + LoadDoclistCtx sCtx = {0,0,0}; /* Context for fts3ExprIterate() */ 803 261 sCtx.pTab = (Fts3Table *)pCsr->base.pVtab; 804 - rc = fts3ExprIterate(pCsr->pExpr, fts3ExprLoadDoclistsCb, (void *)&sCtx); 805 - *pnPhrase = sCtx.nPhrase; 262 + rc = fts3ExprIterate(pCsr->pExpr, fts3ExprLoadDoclistsCb1, (void *)&sCtx); 263 + if( rc==SQLITE_OK ){ 264 + (void)fts3ExprIterate(pCsr->pExpr, fts3ExprLoadDoclistsCb2, 0); 265 + } 266 + if( pnPhrase ) *pnPhrase = sCtx.nPhrase; 267 + if( pnToken ) *pnToken = sCtx.nToken; 806 268 return rc; 807 269 } 808 270 809 271 /* 810 -** Each call to this function populates a chunk of a snippet-buffer 811 -** SNIPPET_BUFFER_CHUNK bytes in size. 812 -** 813 -** Return true if the end of the data has been reached (and all subsequent 814 -** calls to fts3LoadSnippetBuffer() with the same arguments will be no-ops), 815 -** or false otherwise. 816 -*/ 817 -static int fts3LoadSnippetBuffer( 818 - int iPos, /* Document token offset to load data for */ 819 - u8 *aBuffer, /* Circular snippet buffer to populate */ 820 - int nList, /* Number of position lists in appList */ 821 - char **apList, /* IN/OUT: nList position list pointers */ 822 - int *aiPrev /* IN/OUT: Previous positions read */ 823 -){ 824 - int i; 825 - int nFin = 0; 826 - 827 - assert( (iPos&(SNIPPET_BUFFER_CHUNK-1))==0 ); 828 - 829 - memset(&aBuffer[iPos&SNIPPET_BUFFER_MASK], 0, SNIPPET_BUFFER_CHUNK); 830 - 831 - for(i=0; i<nList; i++){ 832 - int iPrev = aiPrev[i]; 833 - char *pList = apList[i]; 834 - 835 - if( !pList ){ 836 - nFin++; 837 - continue; 838 - } 839 - 840 - while( iPrev<(iPos+SNIPPET_BUFFER_CHUNK) ){ 841 - if( iPrev>=iPos ){ 842 - aBuffer[iPrev&SNIPPET_BUFFER_MASK] = (u8)(i+1); 843 - } 844 - if( 0==((*pList)&0xFE) ){ 845 - nFin++; 272 +** Advance the position list iterator specified by the first two 273 +** arguments so that it points to the first element with a value greater 274 +** than or equal to parameter iNext. 275 +*/ 276 +static void fts3SnippetAdvance(char **ppIter, int *piIter, int iNext){ 277 + char *pIter = *ppIter; 278 + if( pIter ){ 279 + int iIter = *piIter; 280 + 281 + while( iIter<iNext ){ 282 + if( 0==(*pIter & 0xFE) ){ 283 + iIter = -1; 284 + pIter = 0; 846 285 break; 847 286 } 848 - fts3GetDeltaPosition(&pList, &iPrev); 849 - } 850 - 851 - aiPrev[i] = iPrev; 852 - apList[i] = pList; 853 - } 854 - 855 - return (nFin==nList); 856 -} 857 - 858 -typedef struct SnippetCtx SnippetCtx; 859 -struct SnippetCtx { 860 - Fts3Cursor *pCsr; 861 - int iCol; 862 - int iPhrase; 863 - int *aiPrev; 864 - int *anToken; 865 - char **apList; 866 -}; 867 - 868 -static int fts3SnippetFindPositions(Fts3Expr *pExpr, void *ctx){ 869 - SnippetCtx *p = (SnippetCtx *)ctx; 870 - int iPhrase = p->iPhrase++; 287 + fts3GetDeltaPosition(&pIter, &iIter); 288 + } 289 + 290 + *piIter = iIter; 291 + *ppIter = pIter; 292 + } 293 +} 294 + 295 +/* 296 +** Advance the snippet iterator to the next candidate snippet. 297 +*/ 298 +static int fts3SnippetNextCandidate(SnippetIter *pIter){ 299 + int i; /* Loop counter */ 300 + 301 + if( pIter->iCurrent<0 ){ 302 + /* The SnippetIter object has just been initialized. The first snippet 303 + ** candidate always starts at offset 0 (even if this candidate has a 304 + ** score of 0.0). 305 + */ 306 + pIter->iCurrent = 0; 307 + 308 + /* Advance the 'head' iterator of each phrase to the first offset that 309 + ** is greater than or equal to (iNext+nSnippet). 310 + */ 311 + for(i=0; i<pIter->nPhrase; i++){ 312 + SnippetPhrase *pPhrase = &pIter->aPhrase[i]; 313 + fts3SnippetAdvance(&pPhrase->pHead, &pPhrase->iHead, pIter->nSnippet); 314 + } 315 + }else{ 316 + int iStart; 317 + int iEnd = 0x7FFFFFFF; 318 + 319 + for(i=0; i<pIter->nPhrase; i++){ 320 + SnippetPhrase *pPhrase = &pIter->aPhrase[i]; 321 + if( pPhrase->pHead && pPhrase->iHead<iEnd ){ 322 + iEnd = pPhrase->iHead; 323 + } 324 + } 325 + if( iEnd==0x7FFFFFFF ){ 326 + return 1; 327 + } 328 + 329 + pIter->iCurrent = iStart = iEnd - pIter->nSnippet + 1; 330 + for(i=0; i<pIter->nPhrase; i++){ 331 + SnippetPhrase *pPhrase = &pIter->aPhrase[i]; 332 + fts3SnippetAdvance(&pPhrase->pHead, &pPhrase->iHead, iEnd+1); 333 + fts3SnippetAdvance(&pPhrase->pTail, &pPhrase->iTail, iStart); 334 + } 335 + } 336 + 337 + return 0; 338 +} 339 + 340 +/* 341 +** Retrieve information about the current candidate snippet of snippet 342 +** iterator pIter. 343 +*/ 344 +static void fts3SnippetDetails( 345 + SnippetIter *pIter, /* Snippet iterator */ 346 + u64 mCovered, /* Bitmask of phrases already covered */ 347 + int *piToken, /* OUT: First token of proposed snippet */ 348 + int *piScore, /* OUT: "Score" for this snippet */ 349 + u64 *pmCover, /* OUT: Bitmask of phrases covered */ 350 + u64 *pmHighlight /* OUT: Bitmask of terms to highlight */ 351 +){ 352 + int iStart = pIter->iCurrent; /* First token of snippet */ 353 + int iScore = 0; /* Score of this snippet */ 354 + int i; /* Loop counter */ 355 + u64 mCover = 0; /* Mask of phrases covered by this snippet */ 356 + u64 mHighlight = 0; /* Mask of tokens to highlight in snippet */ 357 + 358 + for(i=0; i<pIter->nPhrase; i++){ 359 + SnippetPhrase *pPhrase = &pIter->aPhrase[i]; 360 + if( pPhrase->pTail ){ 361 + char *pCsr = pPhrase->pTail; 362 + int iCsr = pPhrase->iTail; 363 + 364 + while( iCsr<(iStart+pIter->nSnippet) ){ 365 + int j; 366 + u64 mPhrase = (u64)1 << i; 367 + u64 mPos = (u64)1 << (iCsr - iStart); 368 + assert( iCsr>=iStart ); 369 + if( (mCover|mCovered)&mPhrase ){ 370 + iScore++; 371 + }else{ 372 + iScore += 1000; 373 + } 374 + mCover |= mPhrase; 375 + 376 + for(j=0; j<pPhrase->nToken; j++){ 377 + mHighlight |= (mPos>>j); 378 + } 379 + 380 + if( 0==(*pCsr & 0x0FE) ) break; 381 + fts3GetDeltaPosition(&pCsr, &iCsr); 382 + } 383 + } 384 + } 385 + 386 + /* Set the output variables before returning. */ 387 + *piToken = iStart; 388 + *piScore = iScore; 389 + *pmCover = mCover; 390 + *pmHighlight = mHighlight; 391 +} 392 + 393 +/* 394 +** This function is an fts3ExprIterate() callback used by fts3BestSnippet(). 395 +** Each invocation populates an element of the SnippetIter.aPhrase[] array. 396 +*/ 397 +static int fts3SnippetFindPositions(Fts3Expr *pExpr, int iPhrase, void *ctx){ 398 + SnippetIter *p = (SnippetIter *)ctx; 399 + SnippetPhrase *pPhrase = &p->aPhrase[iPhrase]; 871 400 char *pCsr; 872 401 873 - p->anToken[iPhrase] = pExpr->pPhrase->nToken; 402 + pPhrase->nToken = pExpr->pPhrase->nToken; 403 + 874 404 pCsr = sqlite3Fts3FindPositions(pExpr, p->pCsr->iPrevId, p->iCol); 875 - 876 405 if( pCsr ){ 877 - int iVal; 878 - pCsr += sqlite3Fts3GetVarint32(pCsr, &iVal); 879 - p->apList[iPhrase] = pCsr; 880 - p->aiPrev[iPhrase] = iVal-2; 406 + int iFirst = 0; 407 + pPhrase->pList = pCsr; 408 + fts3GetDeltaPosition(&pCsr, &iFirst); 409 + pPhrase->pHead = pCsr; 410 + pPhrase->pTail = pCsr; 411 + pPhrase->iHead = iFirst; 412 + pPhrase->iTail = iFirst; 413 + }else{ 414 + assert( pPhrase->pList==0 && pPhrase->pHead==0 && pPhrase->pTail==0 ); 881 415 } 416 + 882 417 return SQLITE_OK; 883 418 } 884 419 885 -static void fts3SnippetCnt( 886 - int iIdx, 887 - int nSnippet, 888 - int *anCnt, 889 - u8 *aBuffer, 890 - int *anToken, 891 - u64 *pHlmask 892 -){ 893 - int iSub = (iIdx-1)&SNIPPET_BUFFER_MASK; 894 - int iAdd = (iIdx+nSnippet-1)&SNIPPET_BUFFER_MASK; 895 - int iSub2 = (iIdx+(nSnippet/3)-1)&SNIPPET_BUFFER_MASK; 896 - int iAdd2 = (iIdx+(nSnippet*2/3)-1)&SNIPPET_BUFFER_MASK; 897 - 898 - u64 h = *pHlmask; 899 - 900 - anCnt[ aBuffer[iSub] ]--; 901 - anCnt[ aBuffer[iSub2] ]--; 902 - anCnt[ aBuffer[iAdd] ]++; 903 - anCnt[ aBuffer[iAdd2] ]++; 904 - 905 - h = h >> 1; 906 - if( aBuffer[iAdd] ){ 907 - int j; 908 - for(j=anToken[aBuffer[iAdd]-1]; j>=1; j--){ 909 - h |= (u64)1 << (nSnippet-j); 910 - } 911 - } 912 - *pHlmask = h; 913 -} 914 - 915 -static int fts3SnippetScore(int n, int *anCnt){ 916 - int j; 917 - int iScore = 0; 918 - for(j=1; j<=n; j++){ 919 - int nCnt = anCnt[j]; 920 - iScore += nCnt + (nCnt ? 1000 : 0); 921 - } 922 - return iScore; 923 -} 924 - 420 +/* 421 +** Select the fragment of text consisting of nFragment contiguous tokens 422 +** from column iCol that represent the "best" snippet. The best snippet 423 +** is the snippet with the highest score, where scores are calculated 424 +** by adding: 425 +** 426 +** (a) +1 point for each occurence of a matchable phrase in the snippet. 427 +** 428 +** (b) +1000 points for the first occurence of each matchable phrase in 429 +** the snippet for which the corresponding mCovered bit is not set. 430 +** 431 +** The selected snippet parameters are stored in structure *pFragment before 432 +** returning. The score of the selected snippet is stored in *piScore 433 +** before returning. 434 +*/ 925 435 static int fts3BestSnippet( 926 436 int nSnippet, /* Desired snippet length */ 927 437 Fts3Cursor *pCsr, /* Cursor to create snippet for */ 928 438 int iCol, /* Index of column to create snippet from */ 929 - int *piPos, /* OUT: Starting token for best snippet */ 930 - u64 *pHlmask /* OUT: Highlight mask for best snippet */ 439 + u64 mCovered, /* Mask of phrases already covered */ 440 + u64 *pmSeen, /* IN/OUT: Mask of phrases seen */ 441 + SnippetFragment *pFragment, /* OUT: Best snippet found */ 442 + int *piScore /* OUT: Score of snippet pFragment */ 931 443 ){ 932 444 int rc; /* Return Code */ 933 - u8 aBuffer[SNIPPET_BUFFER_SIZE];/* Circular snippet buffer */ 934 - int *aiPrev; /* Used by fts3LoadSnippetBuffer() */ 935 - int *anToken; /* Number of tokens in each phrase */ 936 - char **apList; /* Array of position lists */ 937 - int *anCnt; /* Running totals of phrase occurences */ 938 - int nList; 445 + int nList; /* Number of phrases in expression */ 446 + SnippetIter sIter; /* Iterates through snippet candidates */ 447 + int nByte; /* Number of bytes of space to allocate */ 448 + int iBestScore = -1; /* Best snippet score found so far */ 449 + int i; /* Loop counter */ 939 450 940 - int i; 941 - 942 - u64 hlmask = 0; /* Current mask of highlighted terms */ 943 - u64 besthlmask = 0; /* Mask of highlighted terms for iBestPos */ 944 - int iBestPos = 0; /* Starting position of 'best' snippet */ 945 - int iBestScore = 0; /* Score of best snippet higher->better */ 946 - SnippetCtx sCtx; 451 + memset(&sIter, 0, sizeof(sIter)); 947 452 948 453 /* Iterate through the phrases in the expression to count them. The same 949 454 ** callback makes sure the doclists are loaded for each phrase. 950 455 */ 951 - rc = fts3ExprLoadDoclists(pCsr, &nList); 456 + rc = fts3ExprLoadDoclists(pCsr, &nList, 0); 952 457 if( rc!=SQLITE_OK ){ 953 458 return rc; 954 459 } 955 460 956 461 /* Now that it is known how many phrases there are, allocate and zero 957 - ** the required arrays using malloc(). 462 + ** the required space using malloc(). 958 463 */ 959 - apList = sqlite3_malloc( 960 - sizeof(u8*)*nList + /* apList */ 961 - sizeof(int)*(nList) + /* anToken */ 962 - sizeof(int)*nList + /* aiPrev */ 963 - sizeof(int)*(nList+1) /* anCnt */ 964 - ); 965 - if( !apList ){ 464 + nByte = sizeof(SnippetPhrase) * nList; 465 + sIter.aPhrase = (SnippetPhrase *)sqlite3_malloc(nByte); 466 + if( !sIter.aPhrase ){ 966 467 return SQLITE_NOMEM; 967 468 } 968 - memset(apList, 0, sizeof(u8*)*nList+sizeof(int)*nList+sizeof(int)*nList); 969 - anToken = (int *)&apList[nList]; 970 - aiPrev = &anToken[nList]; 971 - anCnt = &aiPrev[nList]; 972 - 973 - /* Initialize the contents of the aiPrev and aiList arrays. */ 974 - sCtx.pCsr = pCsr; 975 - sCtx.iCol = iCol; 976 - sCtx.apList = apList; 977 - sCtx.aiPrev = aiPrev; 978 - sCtx.anToken = anToken; 979 - sCtx.iPhrase = 0; 980 - (void)fts3ExprIterate(pCsr->pExpr, fts3SnippetFindPositions, (void *)&sCtx); 981 - 982 - /* Load the first two chunks of data into the buffer. */ 983 - memset(aBuffer, 0, SNIPPET_BUFFER_SIZE); 984 - fts3LoadSnippetBuffer(0, aBuffer, nList, apList, aiPrev); 985 - fts3LoadSnippetBuffer(SNIPPET_BUFFER_CHUNK, aBuffer, nList, apList, aiPrev); 986 - 987 - /* Set the initial contents of the highlight-mask and anCnt[] array. */ 988 - for(i=1-nSnippet; i<=0; i++){ 989 - fts3SnippetCnt(i, nSnippet, anCnt, aBuffer, anToken, &hlmask); 990 - } 991 - iBestScore = fts3SnippetScore(nList, anCnt); 992 - besthlmask = hlmask; 993 - iBestPos = 0; 994 - 995 - for(i=1; 1; i++){ 469 + memset(sIter.aPhrase, 0, nByte); 470 + 471 + /* Initialize the contents of the SnippetIter object. Then iterate through 472 + ** the set of phrases in the expression to populate the aPhrase[] array. 473 + */ 474 + sIter.pCsr = pCsr; 475 + sIter.iCol = iCol; 476 + sIter.nSnippet = nSnippet; 477 + sIter.nPhrase = nList; 478 + sIter.iCurrent = -1; 479 + (void)fts3ExprIterate(pCsr->pExpr, fts3SnippetFindPositions, (void *)&sIter); 480 + 481 + /* Set the *pmSeen output variable. */ 482 + for(i=0; i<nList; i++){ 483 + if( sIter.aPhrase[i].pHead ){ 484 + *pmSeen |= (u64)1 << i; 485 + } 486 + } 487 + 488 + /* Loop through all candidate snippets. Store the best snippet in 489 + ** *pFragment. Store its associated 'score' in iBestScore. 490 + */ 491 + pFragment->iCol = iCol; 492 + while( !fts3SnippetNextCandidate(&sIter) ){ 493 + int iPos; 996 494 int iScore; 997 - 998 - if( 0==(i&(SNIPPET_BUFFER_CHUNK-1)) ){ 999 - int iLoad = i + SNIPPET_BUFFER_CHUNK; 1000 - if( fts3LoadSnippetBuffer(iLoad, aBuffer, nList, apList, aiPrev) ) break; 1001 - } 1002 - 1003 - /* Figure out how highly a snippet starting at token offset i scores 1004 - ** according to fts3SnippetScore(). If it is higher than any previously 1005 - ** considered position, save the current position, score and hlmask as 1006 - ** the best snippet candidate found so far. 1007 - */ 1008 - fts3SnippetCnt(i, nSnippet, anCnt, aBuffer, anToken, &hlmask); 1009 - iScore = fts3SnippetScore(nList, anCnt); 495 + u64 mCover; 496 + u64 mHighlight; 497 + fts3SnippetDetails(&sIter, mCovered, &iPos, &iScore, &mCover, &mHighlight); 498 + assert( iScore>=0 ); 1010 499 if( iScore>iBestScore ){ 1011 - iBestPos = i; 500 + pFragment->iPos = iPos; 501 + pFragment->hlmask = mHighlight; 502 + pFragment->covered = mCover; 1012 503 iBestScore = iScore; 1013 - besthlmask = hlmask; 1014 504 } 1015 505 } 1016 506 1017 - sqlite3_free(apList); 1018 - *piPos = iBestPos; 1019 - *pHlmask = besthlmask; 507 + sqlite3_free(sIter.aPhrase); 508 + *piScore = iBestScore; 1020 509 return SQLITE_OK; 1021 510 } 1022 511 1023 -typedef struct StrBuffer StrBuffer; 1024 -struct StrBuffer { 1025 - char *z; 1026 - int n; 1027 - int nAlloc; 1028 -}; 1029 512 513 +/* 514 +** Append a string to the string-buffer passed as the first argument. 515 +** 516 +** If nAppend is negative, then the length of the string zAppend is 517 +** determined using strlen(). 518 +*/ 1030 519 static int fts3StringAppend( 1031 - StrBuffer *pStr, 1032 - const char *zAppend, 1033 - int nAppend 520 + StrBuffer *pStr, /* Buffer to append to */ 521 + const char *zAppend, /* Pointer to data to append to buffer */ 522 + int nAppend /* Size of zAppend in bytes (or -1) */ 1034 523 ){ 1035 524 if( nAppend<0 ){ 1036 - nAppend = (int)strlen(zAppend); 525 + nAppend = strlen(zAppend); 1037 526 } 1038 527 528 + /* If there is insufficient space allocated at StrBuffer.z, use realloc() 529 + ** to grow the buffer until so that it is big enough to accomadate the 530 + ** appended data. 531 + */ 1039 532 if( pStr->n+nAppend+1>=pStr->nAlloc ){ 1040 533 int nAlloc = pStr->nAlloc+nAppend+100; 1041 534 char *zNew = sqlite3_realloc(pStr->z, nAlloc); 1042 535 if( !zNew ){ 1043 536 return SQLITE_NOMEM; 1044 537 } 1045 538 pStr->z = zNew; 1046 539 pStr->nAlloc = nAlloc; 1047 540 } 1048 541 542 + /* Append the data to the string buffer. */ 1049 543 memcpy(&pStr->z[pStr->n], zAppend, nAppend); 1050 544 pStr->n += nAppend; 1051 545 pStr->z[pStr->n] = '\0'; 1052 546 1053 547 return SQLITE_OK; 1054 548 } 1055 549 550 +/* 551 +** The fts3BestSnippet() function often selects snippets that end with a 552 +** query term. That is, the final term of the snippet is always a term 553 +** that requires highlighting. For example, if 'X' is a highlighted term 554 +** and '.' is a non-highlighted term, BestSnippet() may select: 555 +** 556 +** ........X.....X 557 +** 558 +** This function "shifts" the beginning of the snippet forward in the 559 +** document so that there are approximately the same number of 560 +** non-highlighted terms to the right of the final highlighted term as there 561 +** are to the left of the first highlighted term. For example, to this: 562 +** 563 +** ....X.....X.... 564 +** 565 +** This is done as part of extracting the snippet text, not when selecting 566 +** the snippet. Snippet selection is done based on doclists only, so there 567 +** is no way for fts3BestSnippet() to know whether or not the document 568 +** actually contains terms that follow the final highlighted term. 569 +*/ 570 +int fts3SnippetShift( 571 + Fts3Table *pTab, /* FTS3 table snippet comes from */ 572 + int nSnippet, /* Number of tokens desired for snippet */ 573 + const char *zDoc, /* Document text to extract snippet from */ 574 + int nDoc, /* Size of buffer zDoc in bytes */ 575 + int *piPos, /* IN/OUT: First token of snippet */ 576 + u64 *pHlmask /* IN/OUT: Mask of tokens to highlight */ 577 +){ 578 + u64 hlmask = *pHlmask; /* Local copy of initial highlight-mask */ 579 + 580 + if( hlmask ){ 581 + int nLeft; /* Tokens to the left of first highlight */ 582 + int nRight; /* Tokens to the right of last highlight */ 583 + int nDesired; /* Ideal number of tokens to shift forward */ 584 + 585 + for(nLeft=0; !(hlmask & ((u64)1 << nLeft)); nLeft++); 586 + for(nRight=0; !(hlmask & ((u64)1 << (nSnippet-1-nRight))); nRight++); 587 + nDesired = (nLeft-nRight)/2; 588 + 589 + /* Ideally, the start of the snippet should be pushed forward in the 590 + ** document nDesired tokens. This block checks if there are actually 591 + ** nDesired tokens to the right of the snippet. If so, *piPos and 592 + ** *pHlMask are updated to shift the snippet nDesired tokens to the 593 + ** right. Otherwise, the snippet is shifted by the number of tokens 594 + ** available. 595 + */ 596 + if( nDesired>0 ){ 597 + int nShift; /* Number of tokens to shift snippet by */ 598 + int iCurrent = 0; /* Token counter */ 599 + int rc; /* Return Code */ 600 + sqlite3_tokenizer_module *pMod; 601 + sqlite3_tokenizer_cursor *pC; 602 + pMod = (sqlite3_tokenizer_module *)pTab->pTokenizer->pModule; 603 + 604 + /* Open a cursor on zDoc/nDoc. Check if there are (nSnippet+nDesired) 605 + ** or more tokens in zDoc/nDoc. 606 + */ 607 + rc = pMod->xOpen(pTab->pTokenizer, zDoc, nDoc, &pC); 608 + if( rc!=SQLITE_OK ){ 609 + return rc; 610 + } 611 + pC->pTokenizer = pTab->pTokenizer; 612 + while( rc==SQLITE_OK && iCurrent<(nSnippet+nDesired) ){ 613 + const char *ZDUMMY; int DUMMY1, DUMMY2, DUMMY3; 614 + rc = pMod->xNext(pC, &ZDUMMY, &DUMMY1, &DUMMY2, &DUMMY3, &iCurrent); 615 + } 616 + pMod->xClose(pC); 617 + if( rc!=SQLITE_OK && rc!=SQLITE_DONE ){ return rc; } 618 + 619 + nShift = (rc==SQLITE_DONE)+iCurrent-nSnippet; 620 + assert( nShift<=nDesired ); 621 + if( nShift>0 ){ 622 + *piPos += nShift; 623 + *pHlmask = hlmask >> nShift; 624 + } 625 + } 626 + } 627 + return SQLITE_OK; 628 +} 629 + 630 +/* 631 +** Extract the snippet text for fragment pFragment from cursor pCsr and 632 +** append it to string buffer pOut. 633 +*/ 1056 634 static int fts3SnippetText( 1057 635 Fts3Cursor *pCsr, /* FTS3 Cursor */ 1058 - const char *zDoc, /* Document to extract snippet from */ 1059 - int nDoc, /* Size of zDoc in bytes */ 636 + SnippetFragment *pFragment, /* Snippet to extract */ 637 + int iFragment, /* Fragment number */ 638 + int isLast, /* True for final fragment in snippet */ 1060 639 int nSnippet, /* Number of tokens in extracted snippet */ 1061 - int iPos, /* Index of first document token in snippet */ 1062 - u64 hlmask, /* Bitmask of terms to highlight in snippet */ 1063 640 const char *zOpen, /* String inserted before highlighted term */ 1064 641 const char *zClose, /* String inserted after highlighted term */ 1065 - const char *zEllipsis, 1066 - char **pzSnippet /* OUT: Snippet text */ 642 + const char *zEllipsis, /* String inserted between snippets */ 643 + StrBuffer *pOut /* Write output here */ 1067 644 ){ 1068 645 Fts3Table *pTab = (Fts3Table *)pCsr->base.pVtab; 1069 646 int rc; /* Return code */ 1070 - int iCurrent = 0; 1071 - int iStart = 0; 1072 - int iEnd; 1073 - 647 + const char *zDoc; /* Document text to extract snippet from */ 648 + int nDoc; /* Size of zDoc in bytes */ 649 + int iCurrent = 0; /* Current token number of document */ 650 + int iEnd = 0; /* Byte offset of end of current token */ 651 + int isShiftDone = 0; /* True after snippet is shifted */ 652 + int iPos = pFragment->iPos; /* First token of snippet */ 653 + u64 hlmask = pFragment->hlmask; /* Highlight-mask for snippet */ 654 + int iCol = pFragment->iCol+1; /* Query column to extract text from */ 1074 655 sqlite3_tokenizer_module *pMod; /* Tokenizer module methods object */ 1075 656 sqlite3_tokenizer_cursor *pC; /* Tokenizer cursor open on zDoc/nDoc */ 1076 - const char *ZDUMMY; /* Dummy arguments used with tokenizer */ 1077 - int DUMMY1, DUMMY2, DUMMY3; /* Dummy arguments used with tokenizer */ 657 + const char *ZDUMMY; /* Dummy argument used with tokenizer */ 658 + int DUMMY1; /* Dummy argument used with tokenizer */ 659 + 660 + zDoc = (const char *)sqlite3_column_text(pCsr->pStmt, iCol); 661 + if( zDoc==0 ){ 662 + if( sqlite3_column_type(pCsr->pStmt, iCol)!=SQLITE_NULL ){ 663 + return SQLITE_NOMEM; 664 + } 665 + return SQLITE_OK; 666 + } 667 + nDoc = sqlite3_column_bytes(pCsr->pStmt, iCol); 1078 668 1079 - StrBuffer res = {0, 0, 0}; /* Result string */ 1080 - 1081 - /* Open a token cursor on the document. Read all tokens up to and 1082 - ** including token iPos (the first token of the snippet). Set variable 1083 - ** iStart to the byte offset in zDoc of the start of token iPos. 1084 - */ 669 + /* Open a token cursor on the document. */ 1085 670 pMod = (sqlite3_tokenizer_module *)pTab->pTokenizer->pModule; 1086 671 rc = pMod->xOpen(pTab->pTokenizer, zDoc, nDoc, &pC); 1087 - while( rc==SQLITE_OK && iCurrent<iPos ){ 1088 - rc = pMod->xNext(pC, &ZDUMMY, &DUMMY1, &iStart, &DUMMY2, &iCurrent); 672 + if( rc!=SQLITE_OK ){ 673 + return rc; 1089 674 } 1090 - iEnd = iStart; 1091 - 1092 - if( rc==SQLITE_OK && iStart>0 ){ 1093 - rc = fts3StringAppend(&res, zEllipsis, -1); 1094 - } 675 + pC->pTokenizer = pTab->pTokenizer; 1095 676 1096 677 while( rc==SQLITE_OK ){ 1097 - int iBegin; 1098 - int iFin; 678 + int iBegin; /* Offset in zDoc of start of token */ 679 + int iFin; /* Offset in zDoc of end of token */ 680 + int isHighlight; /* True for highlighted terms */ 681 + 1099 682 rc = pMod->xNext(pC, &ZDUMMY, &DUMMY1, &iBegin, &iFin, &iCurrent); 683 + if( rc!=SQLITE_OK ){ 684 + if( rc==SQLITE_DONE ){ 685 + /* Special case - the last token of the snippet is also the last token 686 + ** of the column. Append any punctuation that occurred between the end 687 + ** of the previous token and the end of the document to the output. 688 + ** Then break out of the loop. */ 689 + rc = fts3StringAppend(pOut, &zDoc[iEnd], -1); 690 + } 691 + break; 692 + } 693 + if( iCurrent<iPos ){ continue; } 1100 694 1101 - if( rc==SQLITE_OK ){ 1102 - if( iCurrent>=(iPos+nSnippet) ){ 1103 - rc = SQLITE_DONE; 1104 - }else{ 1105 - iEnd = iFin; 1106 - if( hlmask & ((u64)1 << (iCurrent-iPos)) ){ 1107 - if( fts3StringAppend(&res, &zDoc[iStart], iBegin-iStart) 1108 - || fts3StringAppend(&res, zOpen, -1) 1109 - || fts3StringAppend(&res, &zDoc[iBegin], iEnd-iBegin) 1110 - || fts3StringAppend(&res, zClose, -1) 1111 - ){ 1112 - rc = SQLITE_NOMEM; 1113 - } 1114 - iStart = iEnd; 1115 - } 695 + if( !isShiftDone ){ 696 + int n = nDoc - iBegin; 697 + rc = fts3SnippetShift(pTab, nSnippet, &zDoc[iBegin], n, &iPos, &hlmask); 698 + isShiftDone = 1; 699 + 700 + /* Now that the shift has been done, check if the initial "..." are 701 + ** required. They are required if (a) this is not the first fragment, 702 + ** or (b) this fragment does not begin at position 0 of its column. 703 + */ 704 + if( rc==SQLITE_OK && (iPos>0 || iFragment>0) ){ 705 + rc = fts3StringAppend(pOut, zEllipsis, -1); 1116 706 } 707 + if( rc!=SQLITE_OK || iCurrent<iPos ) continue; 1117 708 } 1118 - } 1119 - assert( rc!=SQLITE_OK ); 1120 - if( rc==SQLITE_DONE ){ 1121 - rc = fts3StringAppend(&res, &zDoc[iStart], iEnd-iStart); 1122 - if( rc==SQLITE_OK ){ 1123 - rc = pMod->xNext(pC, &ZDUMMY, &DUMMY1, &DUMMY2, &DUMMY3, &iCurrent); 1124 - if( rc==SQLITE_OK ){ 1125 - rc = fts3StringAppend(&res, zEllipsis, -1); 1126 - }else if( rc==SQLITE_DONE ){ 1127 - rc = fts3StringAppend(&res, &zDoc[iEnd], -1); 709 + 710 + if( iCurrent>=(iPos+nSnippet) ){ 711 + if( isLast ){ 712 + rc = fts3StringAppend(pOut, zEllipsis, -1); 1128 713 } 714 + break; 1129 715 } 716 + 717 + /* Set isHighlight to true if this term should be highlighted. */ 718 + isHighlight = (hlmask & ((u64)1 << (iCurrent-iPos)))!=0; 719 + 720 + if( iCurrent>iPos ) rc = fts3StringAppend(pOut, &zDoc[iEnd], iBegin-iEnd); 721 + if( rc==SQLITE_OK && isHighlight ) rc = fts3StringAppend(pOut, zOpen, -1); 722 + if( rc==SQLITE_OK ) rc = fts3StringAppend(pOut, &zDoc[iBegin], iFin-iBegin); 723 + if( rc==SQLITE_OK && isHighlight ) rc = fts3StringAppend(pOut, zClose, -1); 724 + 725 + iEnd = iFin; 1130 726 } 1131 727 1132 728 pMod->xClose(pC); 1133 - if( rc!=SQLITE_OK ){ 1134 - sqlite3_free(res.z); 1135 - }else{ 1136 - *pzSnippet = res.z; 1137 - } 1138 729 return rc; 1139 730 } 1140 731 1141 732 1142 733 /* 1143 -** An instance of this structure is used to collect the 'global' part of 1144 -** the matchinfo statistics. The 'global' part consists of the following: 1145 -** 1146 -** 1. The number of phrases in the query (nPhrase). 1147 -** 1148 -** 2. The number of columns in the FTS3 table (nCol). 1149 -** 1150 -** 3. A matrix of (nPhrase*nCol) integers containing the sum of the 1151 -** number of hits for each phrase in each column across all rows 1152 -** of the table. 734 +** This function is used to count the entries in a column-list (a 735 +** delta-encoded list of term offsets within a single column of a single 736 +** row). When this function is called, *ppCollist should point to the 737 +** beginning of the first varint in the column-list (the varint that 738 +** contains the position of the first matching term in the column data). 739 +** Before returning, *ppCollist is set to point to the first byte after 740 +** the last varint in the column-list (either the 0x00 signifying the end 741 +** of the position-list, or the 0x01 that precedes the column number of 742 +** the next column in the position-list). 1153 743 ** 1154 -** The total size of the global matchinfo array, assuming the number of 1155 -** columns is N and the number of phrases is P is: 1156 -** 1157 -** 2 + P*(N+1) 1158 -** 1159 -** The number of hits for the 3rd phrase in the second column is found 1160 -** using the expression: 1161 -** 1162 -** aGlobal[2 + P*(1+2) + 1] 1163 -*/ 1164 -typedef struct MatchInfo MatchInfo; 1165 -struct MatchInfo { 1166 - Fts3Table *pTab; /* FTS3 Table */ 1167 - Fts3Cursor *pCursor; /* FTS3 Cursor */ 1168 - int iPhrase; /* Number of phrases so far */ 1169 - int nCol; /* Number of columns in table */ 1170 - u32 *aGlobal; /* Pre-allocated buffer */ 1171 -}; 1172 - 1173 -/* 1174 -** This function is used to count the entries in a column-list (delta-encoded 1175 -** list of term offsets within a single column of a single row). 744 +** The number of elements in the column-list is returned. 1176 745 */ 1177 746 static int fts3ColumnlistCount(char **ppCollist){ 1178 747 char *pEnd = *ppCollist; 1179 748 char c = 0; 1180 749 int nEntry = 0; 1181 750 1182 751 /* A column-list is terminated by either a 0x01 or 0x00. */ ................................................................................ 1185 754 if( !c ) nEntry++; 1186 755 } 1187 756 1188 757 *ppCollist = pEnd; 1189 758 return nEntry; 1190 759 } 1191 760 1192 -static void fts3LoadColumnlistCounts(char **pp, u32 *aOut){ 761 +static void fts3LoadColumnlistCounts(char **pp, u32 *aOut, int isGlobal){ 1193 762 char *pCsr = *pp; 1194 763 while( *pCsr ){ 764 + int nHit; 1195 765 sqlite3_int64 iCol = 0; 1196 766 if( *pCsr==0x01 ){ 1197 767 pCsr++; 1198 768 pCsr += sqlite3Fts3GetVarint(pCsr, &iCol); 1199 769 } 1200 - aOut[iCol] += fts3ColumnlistCount(&pCsr); 770 + nHit = fts3ColumnlistCount(&pCsr); 771 + assert( nHit>0 ); 772 + if( isGlobal ){ 773 + aOut[iCol*3+1]++; 774 + } 775 + aOut[iCol*3] += nHit; 1201 776 } 1202 777 pCsr++; 1203 778 *pp = pCsr; 1204 779 } 1205 780 1206 781 /* 1207 782 ** fts3ExprIterate() callback used to collect the "global" matchinfo stats 1208 -** for a single query. 783 +** for a single query. The "global" stats are those elements of the matchinfo 784 +** array that are constant for all rows returned by the current query. 1209 785 */ 1210 786 static int fts3ExprGlobalMatchinfoCb( 1211 787 Fts3Expr *pExpr, /* Phrase expression node */ 788 + int iPhrase, /* Phrase number (numbered from zero) */ 1212 789 void *pCtx /* Pointer to MatchInfo structure */ 1213 790 ){ 1214 791 MatchInfo *p = (MatchInfo *)pCtx; 1215 792 char *pCsr; 1216 793 char *pEnd; 1217 - const int iStart = 2 + p->nCol*p->iPhrase; 794 + const int iStart = 2 + (iPhrase * p->nCol * 3) + 1; 1218 795 1219 796 assert( pExpr->isLoaded ); 1220 797 1221 798 /* Fill in the global hit count matrix row for this phrase. */ 1222 799 pCsr = pExpr->aDoclist; 1223 800 pEnd = &pExpr->aDoclist[pExpr->nDoclist]; 1224 801 while( pCsr<pEnd ){ 1225 - while( *pCsr++ & 0x80 ); 1226 - fts3LoadColumnlistCounts(&pCsr, &p->aGlobal[iStart]); 802 + while( *pCsr++ & 0x80 ); /* Skip past docid. */ 803 + fts3LoadColumnlistCounts(&pCsr, &p->aMatchinfo[iStart], 1); 1227 804 } 1228 805 1229 - p->iPhrase++; 1230 806 return SQLITE_OK; 1231 807 } 1232 808 809 +/* 810 +** fts3ExprIterate() callback used to collect the "local" matchinfo stats 811 +** for a single query. The "local" stats are those elements of the matchinfo 812 +** array that are different for each row returned by the query. 813 +*/ 1233 814 static int fts3ExprLocalMatchinfoCb( 1234 815 Fts3Expr *pExpr, /* Phrase expression node */ 816 + int iPhrase, /* Phrase number */ 1235 817 void *pCtx /* Pointer to MatchInfo structure */ 1236 818 ){ 1237 819 MatchInfo *p = (MatchInfo *)pCtx; 1238 - int iPhrase = p->iPhrase++; 1239 820 1240 821 if( pExpr->aDoclist ){ 1241 822 char *pCsr; 1242 - int iOffset = 2 + p->nCol*(p->aGlobal[0]+iPhrase); 823 + int iStart = 2 + (iPhrase * p->nCol * 3); 824 + int i; 1243 825 1244 - memset(&p->aGlobal[iOffset], 0, p->nCol*sizeof(u32)); 826 + for(i=0; i<p->nCol; i++) p->aMatchinfo[iStart+i*3] = 0; 827 + 1245 828 pCsr = sqlite3Fts3FindPositions(pExpr, p->pCursor->iPrevId, -1); 1246 - if( pCsr ) fts3LoadColumnlistCounts(&pCsr, &p->aGlobal[iOffset]); 829 + if( pCsr ){ 830 + fts3LoadColumnlistCounts(&pCsr, &p->aMatchinfo[iStart], 0); 831 + } 1247 832 } 1248 833 1249 834 return SQLITE_OK; 1250 835 } 1251 836 1252 837 /* 1253 -** Populate pCsr->aMatchinfo[] with data for the current row. The 'matchinfo' 1254 -** data is an array of 32-bit unsigned integers (C type u32). 838 +** Populate pCsr->aMatchinfo[] with data for the current row. The 839 +** 'matchinfo' data is an array of 32-bit unsigned integers (C type u32). 1255 840 */ 1256 841 static int fts3GetMatchinfo(Fts3Cursor *pCsr){ 1257 - MatchInfo g; 842 + MatchInfo sInfo; 1258 843 Fts3Table *pTab = (Fts3Table *)pCsr->base.pVtab; 844 + 845 + sInfo.pCursor = pCsr; 846 + sInfo.nCol = pTab->nColumn; 847 + 1259 848 if( pCsr->aMatchinfo==0 ){ 1260 - int rc; 1261 - int nPhrase; 1262 - int nMatchinfo; 849 + /* If Fts3Cursor.aMatchinfo[] is NULL, then this is the first time the 850 + ** matchinfo function has been called for this query. In this case 851 + ** allocate the array used to accumulate the matchinfo data and 852 + ** initialize those elements that are constant for every row. 853 + */ 854 + int rc; /* Return Code */ 855 + int nPhrase; /* Number of phrases */ 856 + int nMatchinfo; /* Number of u32 elements in match-info */ 1263 857 1264 - g.pTab = pTab; 1265 - g.nCol = pTab->nColumn; 1266 - g.iPhrase = 0; 1267 - rc = fts3ExprLoadDoclists(pCsr, &nPhrase); 858 + /* Load doclists for each phrase in the query. */ 859 + rc = fts3ExprLoadDoclists(pCsr, &nPhrase, 0); 1268 860 if( rc!=SQLITE_OK ){ 1269 861 return rc; 1270 862 } 863 + nMatchinfo = 2 + 3*sInfo.nCol*nPhrase; 1271 864 1272 - nMatchinfo = 2 + 2*g.nCol*nPhrase; 1273 - 1274 - g.iPhrase = 0; 1275 - g.aGlobal = (u32 *)sqlite3_malloc(sizeof(u32)*nMatchinfo); 1276 - if( !g.aGlobal ){ 865 + sInfo.aMatchinfo = (u32 *)sqlite3_malloc(sizeof(u32)*nMatchinfo); 866 + if( !sInfo.aMatchinfo ){ 1277 867 return SQLITE_NOMEM; 1278 868 } 1279 - memset(g.aGlobal, 0, sizeof(u32)*nMatchinfo); 869 + memset(sInfo.aMatchinfo, 0, sizeof(u32)*nMatchinfo); 1280 870 1281 - g.aGlobal[0] = nPhrase; 1282 - g.aGlobal[1] = g.nCol; 1283 - (void)fts3ExprIterate(pCsr->pExpr, fts3ExprGlobalMatchinfoCb, (void *)&g); 1284 871 1285 - pCsr->aMatchinfo = g.aGlobal; 872 + /* First element of match-info is the number of phrases in the query */ 873 + sInfo.aMatchinfo[0] = nPhrase; 874 + sInfo.aMatchinfo[1] = sInfo.nCol; 875 + (void)fts3ExprIterate(pCsr->pExpr, fts3ExprGlobalMatchinfoCb,(void*)&sInfo); 876 + 877 + pCsr->aMatchinfo = sInfo.aMatchinfo; 1286 878 } 1287 879 1288 - g.pTab = pTab; 1289 - g.pCursor = pCsr; 1290 - g.nCol = pTab->nColumn; 1291 - g.iPhrase = 0; 1292 - g.aGlobal = pCsr->aMatchinfo; 1293 - 880 + sInfo.aMatchinfo = pCsr->aMatchinfo; 1294 881 if( pCsr->isMatchinfoOk ){ 1295 - (void)fts3ExprIterate(pCsr->pExpr, fts3ExprLocalMatchinfoCb, (void *)&g); 882 + (void)fts3ExprIterate(pCsr->pExpr, fts3ExprLocalMatchinfoCb, (void*)&sInfo); 1296 883 pCsr->isMatchinfoOk = 0; 1297 884 } 1298 885 1299 886 return SQLITE_OK; 1300 887 } 1301 888 1302 -void sqlite3Fts3Snippet2( 889 +/* 890 +** Implementation of snippet() function. 891 +*/ 892 +void sqlite3Fts3Snippet( 1303 893 sqlite3_context *pCtx, /* SQLite function call context */ 1304 894 Fts3Cursor *pCsr, /* Cursor object */ 1305 895 const char *zStart, /* Snippet start text - "<b>" */ 1306 896 const char *zEnd, /* Snippet end text - "</b>" */ 1307 897 const char *zEllipsis, /* Snippet ellipsis text - "<b>...</b>" */ 1308 898 int iCol, /* Extract snippet from this column */ 1309 899 int nToken /* Approximate number of tokens in snippet */ 1310 900 ){ 1311 - int rc; 1312 - int iPos = 0; 1313 - u64 hlmask = 0; 1314 - char *z = 0; 1315 - int nDoc; 1316 - const char *zDoc; 1317 - 1318 - rc = fts3BestSnippet(nToken, pCsr, iCol, &iPos, &hlmask); 1319 - 1320 - nDoc = sqlite3_column_bytes(pCsr->pStmt, iCol+1); 1321 - zDoc = (const char *)sqlite3_column_text(pCsr->pStmt, iCol+1); 1322 - 1323 - if( rc==SQLITE_OK ){ 1324 - rc = fts3SnippetText( 1325 - pCsr, zDoc, nDoc, nToken, iPos, hlmask, zStart, zEnd, zEllipsis, &z); 901 + Fts3Table *pTab = (Fts3Table *)pCsr->base.pVtab; 902 + int rc = SQLITE_OK; 903 + int i; 904 + StrBuffer res = {0, 0, 0}; 905 + 906 + /* The returned text includes up to four fragments of text extracted from 907 + ** the data in the current row. The first iteration of the for(...) loop 908 + ** below attempts to locate a single fragment of text nToken tokens in 909 + ** size that contains at least one instance of all phrases in the query 910 + ** expression that appear in the current row. If such a fragment of text 911 + ** cannot be found, the second iteration of the loop attempts to locate 912 + ** a pair of fragments, and so on. 913 + */ 914 + int nSnippet = 0; /* Number of fragments in this snippet */ 915 + SnippetFragment aSnippet[4]; /* Maximum of 4 fragments per snippet */ 916 + int nFToken = -1; /* Number of tokens in each fragment */ 917 + 918 + for(nSnippet=1; 1; nSnippet++){ 919 + 920 + int iSnip; /* Loop counter 0..nSnippet-1 */ 921 + u64 mCovered = 0; /* Bitmask of phrases covered by snippet */ 922 + u64 mSeen = 0; /* Bitmask of phrases seen by BestSnippet() */ 923 + 924 + if( nToken>=0 ){ 925 + nFToken = (nToken+nSnippet-1) / nSnippet; 926 + }else{ 927 + nFToken = -1 * nToken; 928 + } 929 + 930 + for(iSnip=0; iSnip<nSnippet; iSnip++){ 931 + int iBestScore = -1; /* Best score of columns checked so far */ 932 + int iRead; /* Used to iterate through columns */ 933 + SnippetFragment *pFragment = &aSnippet[iSnip]; 934 + 935 + memset(pFragment, 0, sizeof(*pFragment)); 936 + 937 + /* Loop through all columns of the table being considered for snippets. 938 + ** If the iCol argument to this function was negative, this means all 939 + ** columns of the FTS3 table. Otherwise, only column iCol is considered. 940 + */ 941 + for(iRead=0; iRead<pTab->nColumn; iRead++){ 942 + SnippetFragment sF; 943 + int iS; 944 + if( iCol>=0 && iRead!=iCol ) continue; 945 + 946 + /* Find the best snippet of nFToken tokens in column iRead. */ 947 + rc = fts3BestSnippet(nFToken, pCsr, iRead, mCovered, &mSeen, &sF, &iS); 948 + if( rc!=SQLITE_OK ){ 949 + goto snippet_out; 950 + } 951 + if( iS>iBestScore ){ 952 + *pFragment = sF; 953 + iBestScore = iS; 954 + } 955 + } 956 + 957 + mCovered |= pFragment->covered; 958 + } 959 + 960 + /* If all query phrases seen by fts3BestSnippet() are present in at least 961 + ** one of the nSnippet snippet fragments, break out of the loop. 962 + */ 963 + assert( (mCovered&mSeen)==mCovered ); 964 + if( mSeen==mCovered || nSnippet==SizeofArray(aSnippet) ) break; 965 + } 966 + 967 + assert( nFToken>0 ); 968 + 969 + for(i=0; i<nSnippet && rc==SQLITE_OK; i++){ 970 + rc = fts3SnippetText(pCsr, &aSnippet[i], 971 + i, (i==nSnippet-1), nFToken, zStart, zEnd, zEllipsis, &res 972 + ); 1326 973 } 974 + 975 + snippet_out: 1327 976 if( rc!=SQLITE_OK ){ 1328 977 sqlite3_result_error_code(pCtx, rc); 978 + sqlite3_free(res.z); 979 + }else{ 980 + sqlite3_result_text(pCtx, res.z, -1, sqlite3_free); 981 + } 982 +} 983 + 984 + 985 +typedef struct TermOffset TermOffset; 986 +typedef struct TermOffsetCtx TermOffsetCtx; 987 + 988 +struct TermOffset { 989 + char *pList; /* Position-list */ 990 + int iPos; /* Position just read from pList */ 991 + int iOff; /* Offset of this term from read positions */ 992 +}; 993 + 994 +struct TermOffsetCtx { 995 + int iCol; /* Column of table to populate aTerm for */ 996 + int iTerm; 997 + sqlite3_int64 iDocid; 998 + TermOffset *aTerm; 999 +}; 1000 + 1001 +/* 1002 +** This function is an fts3ExprIterate() callback used by sqlite3Fts3Offsets(). 1003 +*/ 1004 +static int fts3ExprTermOffsetInit(Fts3Expr *pExpr, int iPhrase, void *ctx){ 1005 + TermOffsetCtx *p = (TermOffsetCtx *)ctx; 1006 + int nTerm; /* Number of tokens in phrase */ 1007 + int iTerm; /* For looping through nTerm phrase terms */ 1008 + char *pList; /* Pointer to position list for phrase */ 1009 + int iPos = 0; /* First position in position-list */ 1010 + 1011 + pList = sqlite3Fts3FindPositions(pExpr, p->iDocid, p->iCol); 1012 + nTerm = pExpr->pPhrase->nToken; 1013 + if( pList ){ 1014 + fts3GetDeltaPosition(&pList, &iPos); 1015 + assert( iPos>=0 ); 1016 + } 1017 + 1018 + for(iTerm=0; iTerm<nTerm; iTerm++){ 1019 + TermOffset *pT = &p->aTerm[p->iTerm++]; 1020 + pT->iOff = nTerm-iTerm-1; 1021 + pT->pList = pList; 1022 + pT->iPos = iPos; 1023 + } 1024 + 1025 + return SQLITE_OK; 1026 +} 1027 + 1028 +/* 1029 +** Implementation of offsets() function. 1030 +*/ 1031 +void sqlite3Fts3Offsets( 1032 + sqlite3_context *pCtx, /* SQLite function call context */ 1033 + Fts3Cursor *pCsr /* Cursor object */ 1034 +){ 1035 + Fts3Table *pTab = (Fts3Table *)pCsr->base.pVtab; 1036 + sqlite3_tokenizer_module const *pMod = pTab->pTokenizer->pModule; 1037 + const char *ZDUMMY; /* Dummy argument used with xNext() */ 1038 + int NDUMMY; /* Dummy argument used with xNext() */ 1039 + int rc; /* Return Code */ 1040 + int nToken; /* Number of tokens in query */ 1041 + int iCol; /* Column currently being processed */ 1042 + StrBuffer res = {0, 0, 0}; /* Result string */ 1043 + TermOffsetCtx sCtx; /* Context for fts3ExprTermOffsetInit() */ 1044 + 1045 + memset(&sCtx, 0, sizeof(sCtx)); 1046 + assert( pCsr->isRequireSeek==0 ); 1047 + 1048 + /* Count the number of terms in the query */ 1049 + rc = fts3ExprLoadDoclists(pCsr, 0, &nToken); 1050 + if( rc!=SQLITE_OK ) goto offsets_out; 1051 + 1052 + /* Allocate the array of TermOffset iterators. */ 1053 + sCtx.aTerm = (TermOffset *)sqlite3_malloc(sizeof(TermOffset)*nToken); 1054 + if( 0==sCtx.aTerm ){ 1055 + rc = SQLITE_NOMEM; 1056 + goto offsets_out; 1057 + } 1058 + sCtx.iDocid = pCsr->iPrevId; 1059 + 1060 + /* Loop through the table columns, appending offset information to 1061 + ** string-buffer res for each column. 1062 + */ 1063 + for(iCol=0; iCol<pTab->nColumn; iCol++){ 1064 + sqlite3_tokenizer_cursor *pC; /* Tokenizer cursor */ 1065 + int iStart; 1066 + int iEnd; 1067 + int iCurrent; 1068 + const char *zDoc; 1069 + int nDoc; 1070 + 1071 + /* Initialize the contents of sCtx.aTerm[] for column iCol. There is 1072 + ** no way that this operation can fail, so the return code from 1073 + ** fts3ExprIterate() can be discarded. 1074 + */ 1075 + sCtx.iCol = iCol; 1076 + sCtx.iTerm = 0; 1077 + (void)fts3ExprIterate(pCsr->pExpr, fts3ExprTermOffsetInit, (void *)&sCtx); 1078 + 1079 + /* Retreive the text stored in column iCol. If an SQL NULL is stored 1080 + ** in column iCol, jump immediately to the next iteration of the loop. 1081 + ** If an OOM occurs while retrieving the data (this can happen if SQLite 1082 + ** needs to transform the data from utf-16 to utf-8), return SQLITE_NOMEM 1083 + ** to the caller. 1084 + */ 1085 + zDoc = (const char *)sqlite3_column_text(pCsr->pStmt, iCol+1); 1086 + nDoc = sqlite3_column_bytes(pCsr->pStmt, iCol+1); 1087 + if( zDoc==0 ){ 1088 + if( sqlite3_column_type(pCsr->pStmt, iCol+1)==SQLITE_NULL ){ 1089 + continue; 1090 + } 1091 + rc = SQLITE_NOMEM; 1092 + goto offsets_out; 1093 + } 1094 + 1095 + /* Initialize a tokenizer iterator to iterate through column iCol. */ 1096 + rc = pMod->xOpen(pTab->pTokenizer, zDoc, nDoc, &pC); 1097 + if( rc!=SQLITE_OK ) goto offsets_out; 1098 + pC->pTokenizer = pTab->pTokenizer; 1099 + 1100 + rc = pMod->xNext(pC, &ZDUMMY, &NDUMMY, &iStart, &iEnd, &iCurrent); 1101 + while( rc==SQLITE_OK ){ 1102 + int i; /* Used to loop through terms */ 1103 + int iMinPos = 0x7FFFFFFF; /* Position of next token */ 1104 + TermOffset *pTerm = 0; /* TermOffset associated with next token */ 1105 + 1106 + for(i=0; i<nToken; i++){ 1107 + TermOffset *pT = &sCtx.aTerm[i]; 1108 + if( pT->pList && (pT->iPos-pT->iOff)<iMinPos ){ 1109 + iMinPos = pT->iPos-pT->iOff; 1110 + pTerm = pT; 1111 + } 1112 + } 1113 + 1114 + if( !pTerm ){ 1115 + /* All offsets for this column have been gathered. */ 1116 + break; 1117 + }else{ 1118 + assert( iCurrent<=iMinPos ); 1119 + if( 0==(0xFE&*pTerm->pList) ){ 1120 + pTerm->pList = 0; 1121 + }else{ 1122 + fts3GetDeltaPosition(&pTerm->pList, &pTerm->iPos); 1123 + } 1124 + while( rc==SQLITE_OK && iCurrent<iMinPos ){ 1125 + rc = pMod->xNext(pC, &ZDUMMY, &NDUMMY, &iStart, &iEnd, &iCurrent); 1126 + } 1127 + if( rc==SQLITE_OK ){ 1128 + char aBuffer[64]; 1129 + sqlite3_snprintf(sizeof(aBuffer), aBuffer, 1130 + "%d %d %d %d ", iCol, pTerm-sCtx.aTerm, iStart, iEnd-iStart 1131 + ); 1132 + rc = fts3StringAppend(&res, aBuffer, -1); 1133 + } 1134 + } 1135 + } 1136 + if( rc==SQLITE_DONE ){ 1137 + rc = SQLITE_CORRUPT; 1138 + } 1139 + 1140 + pMod->xClose(pC); 1141 + if( rc!=SQLITE_OK ) goto offsets_out; 1142 + } 1143 + 1144 + offsets_out: 1145 + sqlite3_free(sCtx.aTerm); 1146 + assert( rc!=SQLITE_DONE ); 1147 + if( rc!=SQLITE_OK ){ 1148 + sqlite3_result_error_code(pCtx, rc); 1149 + sqlite3_free(res.z); 1329 1150 }else{ 1330 - sqlite3_result_text(pCtx, z, -1, sqlite3_free); 1151 + sqlite3_result_text(pCtx, res.z, res.n-1, sqlite3_free); 1331 1152 } 1153 + return; 1332 1154 } 1333 1155 1156 +/* 1157 +** Implementation of matchinfo() function. 1158 +*/ 1334 1159 void sqlite3Fts3Matchinfo(sqlite3_context *pContext, Fts3Cursor *pCsr){ 1335 1160 int rc = fts3GetMatchinfo(pCsr); 1336 1161 if( rc!=SQLITE_OK ){ 1337 1162 sqlite3_result_error_code(pContext, rc); 1338 1163 }else{ 1339 - int n = sizeof(u32)*(2+pCsr->aMatchinfo[0]*pCsr->aMatchinfo[1]*2); 1164 + int n = sizeof(u32)*(2+pCsr->aMatchinfo[0]*pCsr->aMatchinfo[1]*3); 1340 1165 sqlite3_result_blob(pContext, pCsr->aMatchinfo, n, SQLITE_TRANSIENT); 1341 1166 } 1342 1167 } 1343 1168 1344 1169 #endif
Changes to src/date.c.
1088 1088 FUNCTION(datetime, -1, 0, 0, datetimeFunc ), 1089 1089 FUNCTION(strftime, -1, 0, 0, strftimeFunc ), 1090 1090 FUNCTION(current_time, 0, 0, 0, ctimeFunc ), 1091 1091 FUNCTION(current_timestamp, 0, 0, 0, ctimestampFunc), 1092 1092 FUNCTION(current_date, 0, 0, 0, cdateFunc ), 1093 1093 #else 1094 1094 STR_FUNCTION(current_time, 0, "%H:%M:%S", 0, currentTimeFunc), 1095 - STR_FUNCTION(current_timestamp, 0, "%Y-%m-%d", 0, currentTimeFunc), 1096 - STR_FUNCTION(current_date, 0, "%Y-%m-%d %H:%M:%S", 0, currentTimeFunc), 1095 + STR_FUNCTION(current_date, 0, "%Y-%m-%d", 0, currentTimeFunc), 1096 + STR_FUNCTION(current_timestamp, 0, "%Y-%m-%d %H:%M:%S", 0, currentTimeFunc), 1097 1097 #endif 1098 1098 }; 1099 1099 int i; 1100 1100 FuncDefHash *pHash = &GLOBAL(FuncDefHash, sqlite3GlobalFunctions); 1101 1101 FuncDef *aFunc = (FuncDef*)&GLOBAL(FuncDef, aDateTimeFuncs); 1102 1102 1103 1103 for(i=0; i<ArraySize(aDateTimeFuncs); i++){ 1104 1104 sqlite3FuncDefInsert(pHash, &aFunc[i]); 1105 1105 } 1106 1106 }
Changes to src/expr.c.
1878 1878 char *out = sqlite3DbMallocRaw(sqlite3VdbeDb(v), 8); 1879 1879 if( out ){ 1880 1880 memcpy(out, in, 8); 1881 1881 } 1882 1882 return out; 1883 1883 } 1884 1884 1885 +#ifndef SQLITE_OMIT_FLOATING_POINT 1885 1886 /* 1886 1887 ** Generate an instruction that will put the floating point 1887 1888 ** value described by z[0..n-1] into register iMem. 1888 1889 ** 1889 1890 ** The z[] string will probably not be zero-terminated. But the 1890 1891 ** z[n] character is guaranteed to be something that does not look 1891 1892 ** like the continuation of the number. ................................................................................ 1897 1898 sqlite3AtoF(z, &value); 1898 1899 assert( !sqlite3IsNaN(value) ); /* The new AtoF never returns NaN */ 1899 1900 if( negateFlag ) value = -value; 1900 1901 zV = dup8bytes(v, (char*)&value); 1901 1902 sqlite3VdbeAddOp4(v, OP_Real, 0, iMem, 0, zV, P4_REAL); 1902 1903 } 1903 1904 } 1905 +#endif 1904 1906 1905 1907 1906 1908 /* 1907 1909 ** Generate an instruction that will put the integer describe by 1908 1910 ** text z[0..n-1] into register iMem. 1909 1911 ** 1910 1912 ** The z[] string will probably not be zero-terminated. But the 1911 1913 ** z[n] character is guaranteed to be something that does not look 1912 1914 ** like the continuation of the number. 1913 1915 */ 1914 -static void codeInteger(Vdbe *v, Expr *pExpr, int negFlag, int iMem){ 1916 +static void codeInteger(Parse *pParse, Expr *pExpr, int negFlag, int iMem){ 1917 + Vdbe *v = pParse->pVdbe; 1915 1918 if( pExpr->flags & EP_IntValue ){ 1916 1919 int i = pExpr->u.iValue; 1917 1920 if( negFlag ) i = -i; 1918 1921 sqlite3VdbeAddOp2(v, OP_Integer, i, iMem); 1919 1922 }else{ 1920 1923 const char *z = pExpr->u.zToken; 1921 1924 assert( z!=0 ); ................................................................................ 1923 1926 i64 value; 1924 1927 char *zV; 1925 1928 sqlite3Atoi64(z, &value); 1926 1929 if( negFlag ) value = -value; 1927 1930 zV = dup8bytes(v, (char*)&value); 1928 1931 sqlite3VdbeAddOp4(v, OP_Int64, 0, iMem, 0, zV, P4_INT64); 1929 1932 }else{ 1933 +#ifdef SQLITE_OMIT_FLOATING_POINT 1934 + sqlite3ErrorMsg(pParse, "oversized integer: %s%s", negFlag ? "-" : "", z); 1935 +#else 1930 1936 codeReal(v, z, negFlag, iMem); 1937 +#endif 1931 1938 } 1932 1939 } 1933 1940 } 1934 1941 1935 1942 /* 1936 1943 ** Clear a cache entry. 1937 1944 */ ................................................................................ 2310 2317 }else{ 2311 2318 inReg = sqlite3ExprCodeGetColumn(pParse, pExpr->pTab, 2312 2319 pExpr->iColumn, pExpr->iTable, target); 2313 2320 } 2314 2321 break; 2315 2322 } 2316 2323 case TK_INTEGER: { 2317 - codeInteger(v, pExpr, 0, target); 2324 + codeInteger(pParse, pExpr, 0, target); 2318 2325 break; 2319 2326 } 2327 +#ifndef SQLITE_OMIT_FLOATING_POINT 2320 2328 case TK_FLOAT: { 2321 2329 assert( !ExprHasProperty(pExpr, EP_IntValue) ); 2322 2330 codeReal(v, pExpr->u.zToken, 0, target); 2323 2331 break; 2324 2332 } 2333 +#endif 2325 2334 case TK_STRING: { 2326 2335 assert( !ExprHasProperty(pExpr, EP_IntValue) ); 2327 2336 sqlite3VdbeAddOp4(v, OP_String8, 0, target, 0, pExpr->u.zToken, 0); 2328 2337 break; 2329 2338 } 2330 2339 case TK_NULL: { 2331 2340 sqlite3VdbeAddOp2(v, OP_Null, 0, target); ................................................................................ 2487 2496 testcase( regFree1==0 ); 2488 2497 testcase( regFree2==0 ); 2489 2498 break; 2490 2499 } 2491 2500 case TK_UMINUS: { 2492 2501 Expr *pLeft = pExpr->pLeft; 2493 2502 assert( pLeft ); 2494 - if( pLeft->op==TK_FLOAT ){ 2503 + if( pLeft->op==TK_INTEGER ){ 2504 + codeInteger(pParse, pLeft, 1, target); 2505 +#ifndef SQLITE_OMIT_FLOATING_POINT 2506 + }else if( pLeft->op==TK_FLOAT ){ 2495 2507 assert( !ExprHasProperty(pExpr, EP_IntValue) ); 2496 2508 codeReal(v, pLeft->u.zToken, 1, target); 2497 - }else if( pLeft->op==TK_INTEGER ){ 2498 - codeInteger(v, pLeft, 1, target); 2509 +#endif 2499 2510 }else{ 2500 2511 regFree1 = r1 = sqlite3GetTempReg(pParse); 2501 2512 sqlite3VdbeAddOp2(v, OP_Integer, 0, r1); 2502 2513 r2 = sqlite3ExprCodeTemp(pParse, pExpr->pLeft, ®Free2); 2503 2514 sqlite3VdbeAddOp3(v, OP_Subtract, r2, r1, target); 2504 2515 testcase( regFree2==0 ); 2505 2516 } ................................................................................ 2739 2750 sqlite3VdbeAddOp2(v, OP_Param, p1, target); 2740 2751 VdbeComment((v, "%s.%s -> $%d", 2741 2752 (pExpr->iTable ? "new" : "old"), 2742 2753 (pExpr->iColumn<0 ? "rowid" : pExpr->pTab->aCol[pExpr->iColumn].zName), 2743 2754 target 2744 2755 )); 2745 2756 2757 +#ifndef SQLITE_OMIT_FLOATING_POINT 2746 2758 /* If the column has REAL affinity, it may currently be stored as an 2747 2759 ** integer. Use OP_RealAffinity to make sure it is really real. */ 2748 2760 if( pExpr->iColumn>=0 2749 2761 && pTab->aCol[pExpr->iColumn].affinity==SQLITE_AFF_REAL 2750 2762 ){ 2751 2763 sqlite3VdbeAddOp1(v, OP_RealAffinity, target); 2752 2764 } 2765 +#endif 2753 2766 break; 2754 2767 } 2755 2768 2756 2769 2757 2770 /* 2758 2771 ** Form A: 2759 2772 ** CASE x WHEN e1 THEN r1 WHEN e2 THEN r2 ... WHEN eN THEN rN ELSE y END ................................................................................ 3411 3424 } 3412 3425 } 3413 3426 sqlite3ReleaseTempReg(pParse, regFree1); 3414 3427 sqlite3ReleaseTempReg(pParse, regFree2); 3415 3428 } 3416 3429 3417 3430 /* 3418 -** Do a deep comparison of two expression trees. Return TRUE (non-zero) 3419 -** if they are identical and return FALSE if they differ in any way. 3431 +** Do a deep comparison of two expression trees. Return 0 if the two 3432 +** expressions are completely identical. Return 1 if they differ only 3433 +** by a COLLATE operator at the top level. Return 2 if there are differences 3434 +** other than the top-level COLLATE operator. 3420 3435 ** 3421 -** Sometimes this routine will return FALSE even if the two expressions 3436 +** Sometimes this routine will return 2 even if the two expressions 3422 3437 ** really are equivalent. If we cannot prove that the expressions are 3423 -** identical, we return FALSE just to be safe. So if this routine 3424 -** returns false, then you do not really know for certain if the two 3425 -** expressions are the same. But if you get a TRUE return, then you 3438 +** identical, we return 2 just to be safe. So if this routine 3439 +** returns 2, then you do not really know for certain if the two 3440 +** expressions are the same. But if you get a 0 or 1 return, then you 3426 3441 ** can be sure the expressions are the same. In the places where 3427 -** this routine is used, it does not hurt to get an extra FALSE - that 3442 +** this routine is used, it does not hurt to get an extra 2 - that 3428 3443 ** just might result in some slightly slower code. But returning 3429 -** an incorrect TRUE could lead to a malfunction. 3444 +** an incorrect 0 or 1 could lead to a malfunction. 3430 3445 */ 3431 3446 int sqlite3ExprCompare(Expr *pA, Expr *pB){ 3432 3447 int i; 3433 3448 if( pA==0||pB==0 ){ 3434 - return pB==pA; 3449 + return pB==pA ? 0 : 2; 3435 3450 } 3436 3451 assert( !ExprHasAnyProperty(pA, EP_TokenOnly|EP_Reduced) ); 3437 3452 assert( !ExprHasAnyProperty(pB, EP_TokenOnly|EP_Reduced) ); 3438 3453 if( ExprHasProperty(pA, EP_xIsSelect) || ExprHasProperty(pB, EP_xIsSelect) ){ 3439 - return 0; 3454 + return 2; 3440 3455 } 3441 - if( (pA->flags & EP_Distinct)!=(pB->flags & EP_Distinct) ) return 0; 3442 - if( pA->op!=pB->op ) return 0; 3443 - if( !sqlite3ExprCompare(pA->pLeft, pB->pLeft) ) return 0; 3444 - if( !sqlite3ExprCompare(pA->pRight, pB->pRight) ) return 0; 3456 + if( (pA->flags & EP_Distinct)!=(pB->flags & EP_Distinct) ) return 2; 3457 + if( pA->op!=pB->op ) return 2; 3458 + if( sqlite3ExprCompare(pA->pLeft, pB->pLeft) ) return 2; 3459 + if( sqlite3ExprCompare(pA->pRight, pB->pRight) ) return 2; 3445 3460 3446 3461 if( pA->x.pList && pB->x.pList ){ 3447 - if( pA->x.pList->nExpr!=pB->x.pList->nExpr ) return 0; 3462 + if( pA->x.pList->nExpr!=pB->x.pList->nExpr ) return 2; 3448 3463 for(i=0; i<pA->x.pList->nExpr; i++){ 3449 3464 Expr *pExprA = pA->x.pList->a[i].pExpr; 3450 3465 Expr *pExprB = pB->x.pList->a[i].pExpr; 3451 - if( !sqlite3ExprCompare(pExprA, pExprB) ) return 0; 3466 + if( sqlite3ExprCompare(pExprA, pExprB) ) return 2; 3452 3467 } 3453 3468 }else if( pA->x.pList || pB->x.pList ){ 3454 - return 0; 3469 + return 2; 3455 3470 } 3456 3471 3457 - if( pA->iTable!=pB->iTable || pA->iColumn!=pB->iColumn ) return 0; 3472 + if( pA->iTable!=pB->iTable || pA->iColumn!=pB->iColumn ) return 2; 3458 3473 if( ExprHasProperty(pA, EP_IntValue) ){ 3459 3474 if( !ExprHasProperty(pB, EP_IntValue) || pA->u.iValue!=pB->u.iValue ){ 3460 - return 0; 3475 + return 2; 3461 3476 } 3462 3477 }else if( pA->op!=TK_COLUMN && pA->u.zToken ){ 3463 - if( ExprHasProperty(pB, EP_IntValue) || NEVER(pB->u.zToken==0) ) return 0; 3478 + if( ExprHasProperty(pB, EP_IntValue) || NEVER(pB->u.zToken==0) ) return 2; 3464 3479 if( sqlite3StrICmp(pA->u.zToken,pB->u.zToken)!=0 ){ 3465 - return 0; 3480 + return 2; 3466 3481 } 3467 3482 } 3468 - return 1; 3483 + if( (pA->flags & EP_ExpCollate)!=(pB->flags & EP_ExpCollate) ) return 1; 3484 + if( (pA->flags & EP_ExpCollate)!=0 && pA->pColl!=pB->pColl ) return 2; 3485 + return 0; 3469 3486 } 3470 3487 3471 3488 3472 3489 /* 3473 3490 ** Add a new element to the pAggInfo->aCol[] array. Return the index of 3474 3491 ** the new element. Return a negative number if malloc fails. 3475 3492 */ ................................................................................ 3592 3609 ** to be ignored */ 3593 3610 if( pNC->nDepth==0 ){ 3594 3611 /* Check to see if pExpr is a duplicate of another aggregate 3595 3612 ** function that is already in the pAggInfo structure 3596 3613 */ 3597 3614 struct AggInfo_func *pItem = pAggInfo->aFunc; 3598 3615 for(i=0; i<pAggInfo->nFunc; i++, pItem++){ 3599 - if( sqlite3ExprCompare(pItem->pExpr, pExpr) ){ 3616 + if( sqlite3ExprCompare(pItem->pExpr, pExpr)==0 ){ 3600 3617 break; 3601 3618 } 3602 3619 } 3603 3620 if( i>=pAggInfo->nFunc ){ 3604 3621 /* pExpr is original. Make a new entry in pAggInfo->aFunc[] 3605 3622 */ 3606 3623 u8 enc = ENC(pParse->db);
Changes to src/func.c.
439 439 static void last_insert_rowid( 440 440 sqlite3_context *context, 441 441 int NotUsed, 442 442 sqlite3_value **NotUsed2 443 443 ){ 444 444 sqlite3 *db = sqlite3_context_db_handle(context); 445 445 UNUSED_PARAMETER2(NotUsed, NotUsed2); 446 + /* IMP: R-51513-12026 The last_insert_rowid() SQL function is a 447 + ** wrapper around the sqlite3_last_insert_rowid() C/C++ interface 448 + ** function. */ 446 449 sqlite3_result_int64(context, sqlite3_last_insert_rowid(db)); 447 450 } 448 451 449 452 /* 450 -** Implementation of the changes() SQL function. The return value is the 451 -** same as the sqlite3_changes() API function. 453 +** Implementation of the changes() SQL function. 454 +** 455 +** IMP: R-62073-11209 The changes() SQL function is a wrapper 456 +** around the sqlite3_changes() C/C++ function and hence follows the same 457 +** rules for counting changes. 452 458 */ 453 459 static void changes( 454 460 sqlite3_context *context, 455 461 int NotUsed, 456 462 sqlite3_value **NotUsed2 457 463 ){ 458 464 sqlite3 *db = sqlite3_context_db_handle(context); ................................................................................ 467 473 static void total_changes( 468 474 sqlite3_context *context, 469 475 int NotUsed, 470 476 sqlite3_value **NotUsed2 471 477 ){ 472 478 sqlite3 *db = sqlite3_context_db_handle(context); 473 479 UNUSED_PARAMETER2(NotUsed, NotUsed2); 480 + /* IMP: R-52756-41993 This function is a wrapper around the 481 + ** sqlite3_total_changes() C/C++ interface. */ 474 482 sqlite3_result_int(context, sqlite3_total_changes(db)); 475 483 } 476 484 477 485 /* 478 486 ** A structure defining how to do GLOB-style comparisons. 479 487 */ 480 488 struct compareInfo { ................................................................................ 734 742 */ 735 743 static void versionFunc( 736 744 sqlite3_context *context, 737 745 int NotUsed, 738 746 sqlite3_value **NotUsed2 739 747 ){ 740 748 UNUSED_PARAMETER2(NotUsed, NotUsed2); 741 - sqlite3_result_text(context, sqlite3_version, -1, SQLITE_STATIC); 749 + /* IMP: R-48699-48617 This function is an SQL wrapper around the 750 + ** sqlite3_libversion() C-interface. */ 751 + sqlite3_result_text(context, sqlite3_libversion(), -1, SQLITE_STATIC); 742 752 } 743 753 744 754 /* 745 755 ** Implementation of the sqlite_source_id() function. The result is a string 746 756 ** that identifies the particular version of the source code used to build 747 757 ** SQLite. 748 758 */ 749 759 static void sourceidFunc( 750 760 sqlite3_context *context, 751 761 int NotUsed, 752 762 sqlite3_value **NotUsed2 753 763 ){ 754 764 UNUSED_PARAMETER2(NotUsed, NotUsed2); 755 - sqlite3_result_text(context, SQLITE_SOURCE_ID, -1, SQLITE_STATIC); 765 + /* IMP: R-24470-31136 This function is an SQL wrapper around the 766 + ** sqlite3_sourceid() C interface. */ 767 + sqlite3_result_text(context, sqlite3_sourceid(), -1, SQLITE_STATIC); 756 768 } 757 769 758 770 /* Array for converting from half-bytes (nybbles) into ASCII hex 759 771 ** digits. */ 760 772 static const char hexdigits[] = { 761 773 '0', '1', '2', '3', '4', '5', '6', '7', 762 774 '8', '9', 'A', 'B', 'C', 'D', 'E', 'F' ................................................................................ 877 889 UNUSED_PARAMETER(argc); 878 890 n = sqlite3_value_int64(argv[0]); 879 891 testcase( n==db->aLimit[SQLITE_LIMIT_LENGTH] ); 880 892 testcase( n==db->aLimit[SQLITE_LIMIT_LENGTH]+1 ); 881 893 if( n>db->aLimit[SQLITE_LIMIT_LENGTH] ){ 882 894 sqlite3_result_error_toobig(context); 883 895 }else{ 884 - sqlite3_result_zeroblob(context, (int)n); 896 + sqlite3_result_zeroblob(context, (int)n); /* IMP: R-00293-64994 */ 885 897 } 886 898 } 887 899 888 900 /* 889 901 ** The replace() function. Three arguments are all strings: call 890 902 ** them A, B, and C. The result is also a string which is derived 891 903 ** from A by replacing every occurance of B with C. The match
Changes to src/insert.c.
1711 1711 if( xferCompatibleIndex(pDestIdx, pSrcIdx) ) break; 1712 1712 } 1713 1713 if( pSrcIdx==0 ){ 1714 1714 return 0; /* pDestIdx has no corresponding index in pSrc */ 1715 1715 } 1716 1716 } 1717 1717 #ifndef SQLITE_OMIT_CHECK 1718 - if( pDest->pCheck && !sqlite3ExprCompare(pSrc->pCheck, pDest->pCheck) ){ 1718 + if( pDest->pCheck && sqlite3ExprCompare(pSrc->pCheck, pDest->pCheck) ){ 1719 1719 return 0; /* Tables have different CHECK constraints. Ticket #2252 */ 1720 1720 } 1721 1721 #endif 1722 1722 1723 1723 /* If we get this far, it means either: 1724 1724 ** 1725 1725 ** * We can always do the transfer if the table contains an
Changes to src/legacy.c.
40 40 int rc = SQLITE_OK; /* Return code */ 41 41 const char *zLeftover; /* Tail of unprocessed SQL */ 42 42 sqlite3_stmt *pStmt = 0; /* The current SQL statement */ 43 43 char **azCols = 0; /* Names of result columns */ 44 44 int nRetry = 0; /* Number of retry attempts */ 45 45 int callbackIsInit; /* True if callback data is initialized */ 46 46 47 - if (!sqlite3SafetyCheckOk(db)) { 48 - return SQLITE_MISUSE; 49 - } 50 - 47 + if( !sqlite3SafetyCheckOk(db) ) return SQLITE_MISUSE; 51 48 if( zSql==0 ) zSql = ""; 52 49 #ifdef SQLITE_ENABLE_SQLRR 53 50 SRRecExec(db, zSql); 54 51 #endif 55 52 sqlite3_mutex_enter(db->mutex); 56 53 sqlite3Error(db, SQLITE_OK, 0); 57 54 while( (rc==SQLITE_OK || (rc==SQLITE_SCHEMA && (++nRetry)<2)) && zSql[0] ){
Changes to src/prepare.c.
314 314 315 315 /* Read the schema information out of the schema tables 316 316 */ 317 317 assert( db->init.busy ); 318 318 { 319 319 char *zSql; 320 320 zSql = sqlite3MPrintf(db, 321 - "SELECT name, rootpage, sql FROM '%q'.%s", 321 + "SELECT name, rootpage, sql FROM '%q'.%s ORDER BY rowid", 322 322 db->aDb[iDb].zName, zMasterName); 323 323 (void)sqlite3SafetyOff(db); 324 324 #ifndef SQLITE_OMIT_AUTHORIZATION 325 325 { 326 326 int (*xAuth)(void*,int,const char*,const char*,const char*,const char*); 327 327 xAuth = db->xAuth; 328 328 db->xAuth = 0;
Changes to src/printf.c.
602 602 for(i=width; i>=nPad; i--){ 603 603 bufpt[i] = bufpt[i-nPad]; 604 604 } 605 605 i = prefix!=0; 606 606 while( nPad-- ) bufpt[i++] = '0'; 607 607 length = width; 608 608 } 609 -#endif 609 +#else 610 + length = 0; 611 +#endif /* SQLITE_OMIT_FLOATING_POINT */ 610 612 break; 611 613 case etSIZE: 612 614 *(va_arg(ap,int*)) = pAccum->nChar; 613 615 length = width = 0; 614 616 break; 615 617 case etPERCENT: 616 618 buf[0] = '%';
Changes to src/resolve.c.
682 682 } 683 683 684 684 /* Try to match the ORDER BY expression against an expression 685 685 ** in the result set. Return an 1-based index of the matching 686 686 ** result-set entry. 687 687 */ 688 688 for(i=0; i<pEList->nExpr; i++){ 689 - if( sqlite3ExprCompare(pEList->a[i].pExpr, pE) ){ 689 + if( sqlite3ExprCompare(pEList->a[i].pExpr, pE)<2 ){ 690 690 return i+1; 691 691 } 692 692 } 693 693 694 694 /* If no match, return 0. */ 695 695 return 0; 696 696 }
Changes to src/shell.c.
1497 1497 ** This is the callback routine that the shell 1498 1498 ** invokes for each row of a query result. 1499 1499 */ 1500 1500 static int shell_callback(void *pArg, int nArg, char **azArg, char **azCol, int *aiType){ 1501 1501 int i; 1502 1502 struct callback_data *p = (struct callback_data*)pArg; 1503 1503 1504 - if( p->echoOn && p->cnt==0 && p->pStmt){ 1505 - printf("%s\n", sqlite3_sql(p->pStmt)); 1506 - } 1507 - 1508 1504 switch( p->mode ){ 1509 1505 case MODE_Line: { 1510 1506 int w = 5; 1511 1507 if( azArg==0 ) break; 1512 1508 for(i=0; i<nArg; i++){ 1513 1509 int len = strlen30(azCol[i] ? azCol[i] : ""); 1514 1510 if( len>w ) w = len; ................................................................................ 1850 1846 }else{ 1851 1847 if( !pStmt ){ 1852 1848 /* this happens for a comment or white-space */ 1853 1849 zSql = zLeftover; 1854 1850 while( isspace(zSql[0]) ) zSql++; 1855 1851 continue; 1856 1852 } 1853 + 1854 + /* echo the sql statement if echo on */ 1855 + if( pArg->echoOn ){ 1856 + char *zStmtSql = sqlite3_sql(pStmt); 1857 + fprintf(pArg->out,"%s\n", zStmtSql ? zStmtSql : zSql); 1858 + } 1857 1859 1858 1860 /* perform the first step. this will tell us if we 1859 1861 ** have a result set or not and how wide it is. 1860 1862 */ 1861 1863 rc = sqlite3_step(pStmt); 1862 1864 /* if we have a result set... */ 1863 1865 if( SQLITE_ROW == rc ){
Changes to src/sqliteInt.h.
72 72 #ifdef HAVE_STDINT_H 73 73 #include <stdint.h> 74 74 #endif 75 75 #ifdef HAVE_INTTYPES_H 76 76 #include <inttypes.h> 77 77 #endif 78 78 79 +/* 80 +** The number of samples of an index that SQLite takes in order to 81 +** construct a histogram of the table content when running ANALYZE 82 +** and with SQLITE_ENABLE_STAT2 83 +*/ 79 84 #define SQLITE_INDEX_SAMPLES 10 80 85 81 86 /* 82 87 ** This macro is used to "hide" some ugliness in casting an int 83 88 ** value to a ptr value under the MSVC 64-bit compiler. Casting 84 89 ** non 64-bit values to ptr types results in a "hard" error with 85 90 ** the MSVC 64-bit compiler which this attempts to avoid. ................................................................................ 136 141 137 142 /* 138 143 ** Exactly one of the following macros must be defined in order to 139 144 ** specify which memory allocation subsystem to use. 140 145 ** 141 146 ** SQLITE_SYSTEM_MALLOC // Use normal system malloc() 142 147 ** SQLITE_MEMDEBUG // Debugging version of system malloc() 143 -** SQLITE_MEMORY_SIZE // internal allocator #1 144 -** SQLITE_MMAP_HEAP_SIZE // internal mmap() allocator 145 -** SQLITE_POW2_MEMORY_SIZE // internal power-of-two allocator 148 +** 149 +** (Historical note: There used to be several other options, but we've 150 +** pared it down to just these two.) 146 151 ** 147 152 ** If none of the above are defined, then set SQLITE_SYSTEM_MALLOC as 148 153 ** the default. 149 154 */ 150 -#if defined(SQLITE_SYSTEM_MALLOC)+defined(SQLITE_MEMDEBUG)+\ 151 - defined(SQLITE_MEMORY_SIZE)+defined(SQLITE_MMAP_HEAP_SIZE)+\ 152 - defined(SQLITE_POW2_MEMORY_SIZE)>1 155 +#if defined(SQLITE_SYSTEM_MALLOC)+defined(SQLITE_MEMDEBUG)>1 153 156 # error "At most one of the following compile-time configuration options\ 154 - is allows: SQLITE_SYSTEM_MALLOC, SQLITE_MEMDEBUG, SQLITE_MEMORY_SIZE,\ 155 - SQLITE_MMAP_HEAP_SIZE, SQLITE_POW2_MEMORY_SIZE" 157 + is allows: SQLITE_SYSTEM_MALLOC, SQLITE_MEMDEBUG" 156 158 #endif 157 -#if defined(SQLITE_SYSTEM_MALLOC)+defined(SQLITE_MEMDEBUG)+\ 158 - defined(SQLITE_MEMORY_SIZE)+defined(SQLITE_MMAP_HEAP_SIZE)+\ 159 - defined(SQLITE_POW2_MEMORY_SIZE)==0 159 +#if defined(SQLITE_SYSTEM_MALLOC)+defined(SQLITE_MEMDEBUG)==0 160 160 # define SQLITE_SYSTEM_MALLOC 1 161 161 #endif 162 162 163 163 /* 164 164 ** If SQLITE_MALLOC_SOFT_LIMIT is not zero, then try to keep the 165 165 ** sizes of memory allocations below this value where possible. 166 166 */ ................................................................................ 324 324 */ 325 325 #ifdef SQLITE_OMIT_TEMPDB 326 326 #define OMIT_TEMPDB 1 327 327 #else 328 328 #define OMIT_TEMPDB 0 329 329 #endif 330 330 331 -/* 332 -** If the following macro is set to 1, then NULL values are considered 333 -** distinct when determining whether or not two entries are the same 334 -** in a UNIQUE index. This is the way PostgreSQL, Oracle, DB2, MySQL, 335 -** OCELOT, and Firebird all work. The SQL92 spec explicitly says this 336 -** is the way things are suppose to work. 337 -** 338 -** If the following macro is set to 0, the NULLs are indistinct for 339 -** a UNIQUE index. In this mode, you can only have a single NULL entry 340 -** for a column declared UNIQUE. This is the way Informix and SQL Server 341 -** work. 342 -*/ 343 -#define NULL_DISTINCT_FOR_UNIQUE 1 344 - 345 331 /* 346 332 ** The "file format" number is an integer that is incremented whenever 347 333 ** the VDBE-level file format changes. The following macros define the 348 334 ** the default file format for new databases and the maximum file format 349 335 ** that the library can read. 350 336 */ 351 337 #define SQLITE_MAX_FILE_FORMAT 4 352 338 #ifndef SQLITE_DEFAULT_FILE_FORMAT 353 339 # define SQLITE_DEFAULT_FILE_FORMAT 1 354 340 #endif 355 341 342 +/* 343 +** Determine whether triggers are recursive by default. This can be 344 +** changed at run-time using a pragma. 345 +*/ 356 346 #ifndef SQLITE_DEFAULT_RECURSIVE_TRIGGERS 357 347 # define SQLITE_DEFAULT_RECURSIVE_TRIGGERS 0 358 348 #endif 359 349 360 350 /* 361 351 ** Provide a default value for SQLITE_TEMP_STORE in case it is not specified 362 352 ** on the command-line ................................................................................ 593 583 /* 594 584 ** Forward references to structures 595 585 */ 596 586 typedef struct AggInfo AggInfo; 597 587 typedef struct AuthContext AuthContext; 598 588 typedef struct AutoincInfo AutoincInfo; 599 589 typedef struct Bitvec Bitvec; 600 -typedef struct RowSet RowSet; 601 590 typedef struct CollSeq CollSeq; 602 591 typedef struct Column Column; 603 592 typedef struct Db Db; 604 593 typedef struct Schema Schema; 605 594 typedef struct Expr Expr; 606 595 typedef struct ExprList ExprList; 607 596 typedef struct ExprSpan ExprSpan; ................................................................................ 614 603 typedef struct KeyClass KeyClass; 615 604 typedef struct KeyInfo KeyInfo; 616 605 typedef struct Lookaside Lookaside; 617 606 typedef struct LookasideSlot LookasideSlot; 618 607 typedef struct Module Module; 619 608 typedef struct NameContext NameContext; 620 609 typedef struct Parse Parse; 610 +typedef struct RowSet RowSet; 621 611 typedef struct Savepoint Savepoint; 622 612 typedef struct Select Select; 623 613 typedef struct SrcList SrcList; 624 614 typedef struct StrAccum StrAccum; 625 615 typedef struct Table Table; 626 616 typedef struct TableLock TableLock; 627 617 typedef struct Token Token; 618 +typedef struct Trigger Trigger; 628 619 typedef struct TriggerPrg TriggerPrg; 629 620 typedef struct TriggerStep TriggerStep; 630 -typedef struct Trigger Trigger; 631 621 typedef struct UnpackedRecord UnpackedRecord; 632 622 typedef struct VTable VTable; 633 623 typedef struct Walker Walker; 634 624 typedef struct WherePlan WherePlan; 635 625 typedef struct WhereInfo WhereInfo; 636 626 typedef struct WhereLevel WhereLevel; 637 627 ................................................................................ 689 679 #ifndef SQLITE_OMIT_VIRTUALTABLE 690 680 sqlite3 *db; /* "Owner" connection. See comment above */ 691 681 #endif 692 682 }; 693 683 694 684 /* 695 685 ** These macros can be used to test, set, or clear bits in the 696 -** Db.flags field. 686 +** Db.pSchema->flags field. 697 687 */ 698 688 #define DbHasProperty(D,I,P) (((D)->aDb[I].pSchema->flags&(P))==(P)) 699 689 #define DbHasAnyProperty(D,I,P) (((D)->aDb[I].pSchema->flags&(P))!=0) 700 690 #define DbSetProperty(D,I,P) (D)->aDb[I].pSchema->flags|=(P) 701 691 #define DbClearProperty(D,I,P) (D)->aDb[I].pSchema->flags&=~(P) 702 692 703 693 /* 704 -** Allowed values for the DB.flags field. 694 +** Allowed values for the DB.pSchema->flags field. 705 695 ** 706 696 ** The DB_SchemaLoaded flag is set after the database schema has been 707 697 ** read into internal hash tables. 708 698 ** 709 699 ** DB_UnresetViews means that one or more views have column names that 710 700 ** have been filled out. If the schema changes, these column names might 711 701 ** changes and so the view will need to be reset. ................................................................................ 761 751 ** Collisions are on the FuncDef.pHash chain. 762 752 */ 763 753 struct FuncDefHash { 764 754 FuncDef *a[23]; /* Hash table for functions */ 765 755 }; 766 756 767 757 /* 768 -** Each database is an instance of the following structure. 758 +** Each database connection is an instance of the following structure. 769 759 ** 770 760 ** The sqlite.lastRowid records the last insert rowid generated by an 771 761 ** insert statement. Inserts on views do not affect its value. Each 772 762 ** trigger has its own context, so that lastRowid can be updated inside 773 763 ** triggers as usual. The previous value will be restored once the trigger 774 764 ** exits. Upon entering a before or instead of trigger, lastRowid is no 775 765 ** longer (since after version 2.8.12) reset to -1. ................................................................................ 2528 2518 int sqlite3MutexEnd(void); 2529 2519 #endif 2530 2520 2531 2521 int sqlite3StatusValue(int); 2532 2522 void sqlite3StatusAdd(int, int); 2533 2523 void sqlite3StatusSet(int, int); 2534 2524 2535 -int sqlite3IsNaN(double); 2525 +#ifndef SQLITE_OMIT_FLOATING_POINT 2526 + int sqlite3IsNaN(double); 2527 +#else 2528 +# define sqlite3IsNaN(X) 0 2529 +#endif 2536 2530 2537 2531 void sqlite3VXPrintf(StrAccum*, int, const char*, va_list); 2538 2532 #ifndef SQLITE_OMIT_TRACE 2539 2533 void sqlite3XPrintf(StrAccum*, const char*, ...); 2540 2534 #endif 2541 2535 char *sqlite3MPrintf(sqlite3*,const char*, ...); 2542 2536 char *sqlite3VMPrintf(sqlite3*,const char*, va_list);
Changes to src/test3.c.
215 215 Tcl_Interp *interp, /* The TCL interpreter that invoked this command */ 216 216 int argc, /* Number of arguments */ 217 217 const char **argv /* Text of each argument */ 218 218 ){ 219 219 Btree *pBt; 220 220 int iTable; 221 221 BtCursor *pCur; 222 - int rc; 222 + int rc = SQLITE_OK; 223 223 int wrFlag; 224 224 char zBuf[30]; 225 225 226 226 if( argc!=4 ){ 227 227 Tcl_AppendResult(interp, "wrong # args: should be \"", argv[0], 228 228 " ID TABLENUM WRITEABLE\"", 0); 229 229 return TCL_ERROR; ................................................................................ 230 230 } 231 231 pBt = sqlite3TestTextToPtr(argv[1]); 232 232 if( Tcl_GetInt(interp, argv[2], &iTable) ) return TCL_ERROR; 233 233 if( Tcl_GetBoolean(interp, argv[3], &wrFlag) ) return TCL_ERROR; 234 234 pCur = (BtCursor *)ckalloc(sqlite3BtreeCursorSize()); 235 235 memset(pCur, 0, sqlite3BtreeCursorSize()); 236 236 sqlite3BtreeEnter(pBt); 237 +#ifndef SQLITE_OMIT_SHARED_CACHE 237 238 rc = sqlite3BtreeLockTable(pBt, iTable, wrFlag); 239 +#endif 238 240 if( rc==SQLITE_OK ){ 239 241 rc = sqlite3BtreeCursor(pBt, iTable, wrFlag, 0, pCur); 240 242 } 241 243 sqlite3BtreeLeave(pBt); 242 244 if( rc ){ 243 245 ckfree((char *)pCur); 244 246 Tcl_AppendResult(interp, errorName(rc), 0);
Changes to src/test_func.c.
337 337 338 338 /* 339 339 ** hex_to_utf16be(HEX) 340 340 ** 341 341 ** Convert the input string from HEX into binary. Then return the 342 342 ** result using sqlite3_result_text16le(). 343 343 */ 344 +#ifndef SQLITE_OMIT_UTF16 344 345 static void testHexToUtf16be( 345 346 sqlite3_context *pCtx, 346 347 int nArg, 347 348 sqlite3_value **argv 348 349 ){ 349 350 int n; 350 351 const char *zIn; ................................................................................ 356 357 if( zOut==0 ){ 357 358 sqlite3_result_error_nomem(pCtx); 358 359 }else{ 359 360 testHexToBin(zIn, zOut); 360 361 sqlite3_result_text16be(pCtx, zOut, n/2, sqlite3_free); 361 362 } 362 363 } 364 +#endif 363 365 364 366 /* 365 367 ** hex_to_utf8(HEX) 366 368 ** 367 369 ** Convert the input string from HEX into binary. Then return the 368 370 ** result using sqlite3_result_text16le(). 369 371 */ ................................................................................ 389 391 390 392 /* 391 393 ** hex_to_utf16le(HEX) 392 394 ** 393 395 ** Convert the input string from HEX into binary. Then return the 394 396 ** result using sqlite3_result_text16le(). 395 397 */ 398 +#ifndef SQLITE_OMIT_UTF16 396 399 static void testHexToUtf16le( 397 400 sqlite3_context *pCtx, 398 401 int nArg, 399 402 sqlite3_value **argv 400 403 ){ 401 404 int n; 402 405 const char *zIn; ................................................................................ 408 411 if( zOut==0 ){ 409 412 sqlite3_result_error_nomem(pCtx); 410 413 }else{ 411 414 testHexToBin(zIn, zOut); 412 415 sqlite3_result_text16le(pCtx, zOut, n/2, sqlite3_free); 413 416 } 414 417 } 418 +#endif 415 419 416 420 static int registerTestFunctions(sqlite3 *db){ 417 421 static const struct { 418 422 char *zName; 419 423 signed char nArg; 420 424 unsigned char eTextRep; /* 1: UTF-16. 0: UTF-8 */ 421 425 void (*xFunc)(sqlite3_context*,int,sqlite3_value **);
Changes to src/util.c.
27 27 #ifdef SQLITE_COVERAGE_TEST 28 28 void sqlite3Coverage(int x){ 29 29 static int dummy = 0; 30 30 dummy += x; 31 31 } 32 32 #endif 33 33 34 +#ifndef SQLITE_OMIT_FLOATING_POINT 34 35 /* 35 36 ** Return true if the floating point value is Not a Number (NaN). 36 37 ** 37 38 ** Use the math library isnan() function if compiled with SQLITE_HAVE_ISNAN. 38 39 ** Otherwise, we have our own implementation that works on most systems. 39 40 */ 40 41 int sqlite3IsNaN(double x){ ................................................................................ 71 72 rc = (y!=z); 72 73 #else /* if defined(SQLITE_HAVE_ISNAN) */ 73 74 rc = isnan(x); 74 75 #endif /* SQLITE_HAVE_ISNAN */ 75 76 testcase( rc ); 76 77 return rc; 77 78 } 79 +#endif /* SQLITE_OMIT_FLOATING_POINT */ 78 80 79 81 /* 80 82 ** Compute a string length that is limited to what can be stored in 81 83 ** lower 30 bits of a 32-bit signed integer. 82 84 ** 83 85 ** The value returned will never be negative. Nor will it ever be greater 84 86 ** than the actual length of the string. For very long strings (greater ................................................................................ 251 253 if( *z=='-' || *z=='+' ) z += incr; 252 254 if( !sqlite3Isdigit(*z) ){ 253 255 return 0; 254 256 } 255 257 z += incr; 256 258 *realnum = 0; 257 259 while( sqlite3Isdigit(*z) ){ z += incr; } 260 +#ifndef SQLITE_OMIT_FLOATING_POINT 258 261 if( *z=='.' ){ 259 262 z += incr; 260 263 if( !sqlite3Isdigit(*z) ) return 0; 261 264 while( sqlite3Isdigit(*z) ){ z += incr; } 262 265 *realnum = 1; 263 266 } 264 267 if( *z=='e' || *z=='E' ){ 265 268 z += incr; 266 269 if( *z=='+' || *z=='-' ) z += incr; 267 270 if( !sqlite3Isdigit(*z) ) return 0; 268 271 while( sqlite3Isdigit(*z) ){ z += incr; } 269 272 *realnum = 1; 270 273 } 274 +#endif 271 275 return *z==0; 272 276 } 273 277 274 278 /* 275 279 ** The string z[] is an ASCII representation of a real number. 276 280 ** Convert this string to a double. 277 281 ** ................................................................................ 425 429 ** will return -8. 426 430 */ 427 431 static int compare2pow63(const char *zNum){ 428 432 int c; 429 433 c = memcmp(zNum,"922337203685477580",18)*10; 430 434 if( c==0 ){ 431 435 c = zNum[18] - '8'; 436 + testcase( c==(-1) ); 437 + testcase( c==0 ); 438 + testcase( c==(+1) ); 432 439 } 433 440 return c; 434 441 } 435 442 436 443 437 444 /* 438 445 ** Return TRUE if zNum is a 64-bit signed integer and write ................................................................................ 461 468 } 462 469 zStart = zNum; 463 470 while( zNum[0]=='0' ){ zNum++; } /* Skip over leading zeros. Ticket #2454 */ 464 471 for(i=0; (c=zNum[i])>='0' && c<='9'; i++){ 465 472 v = v*10 + c - '0'; 466 473 } 467 474 *pNum = neg ? -v : v; 475 + testcase( i==18 ); 476 + testcase( i==19 ); 477 + testcase( i==20 ); 468 478 if( c!=0 || (i==0 && zStart==zNum) || i>19 ){ 469 479 /* zNum is empty or contains non-numeric text or is longer 470 480 ** than 19 digits (thus guaranting that it is too large) */ 471 481 return 0; 472 482 }else if( i<19 ){ 473 483 /* Less than 19 digits, so we know that it fits in 64 bits */ 474 484 return 1; ................................................................................ 504 514 assert( zNum[0]>='0' && zNum[0]<='9' ); /* zNum is an unsigned number */ 505 515 506 516 if( negFlag ) neg = 1-neg; 507 517 while( *zNum=='0' ){ 508 518 zNum++; /* Skip leading zeros. Ticket #2454 */ 509 519 } 510 520 for(i=0; zNum[i]; i++){ assert( zNum[i]>='0' && zNum[i]<='9' ); } 521 + testcase( i==18 ); 522 + testcase( i==19 ); 523 + testcase( i==20 ); 511 524 if( i<19 ){ 512 525 /* Guaranteed to fit if less than 19 digits */ 513 526 return 1; 514 527 }else if( i>19 ){ 515 528 /* Guaranteed to be too big if greater than 19 digits */ 516 529 return 0; 517 530 }else{ ................................................................................ 544 557 } 545 558 546 559 /* The longest decimal representation of a 32 bit integer is 10 digits: 547 560 ** 548 561 ** 1234567890 549 562 ** 2^31 -> 2147483648 550 563 */ 564 + testcase( i==10 ); 551 565 if( i>10 ){ 552 566 return 0; 553 567 } 568 + testcase( v-neg==2147483647 ); 554 569 if( v-neg>2147483647 ){ 555 570 return 0; 556 571 } 557 572 if( neg ){ 558 573 v = -v; 559 574 } 560 575 *pValue = (int)v;
Changes to src/vacuum.c.
100 100 ** restored before returning. Then set the writable-schema flag, and 101 101 ** disable CHECK and foreign key constraints. */ 102 102 saved_flags = db->flags; 103 103 saved_nChange = db->nChange; 104 104 saved_nTotalChange = db->nTotalChange; 105 105 saved_xTrace = db->xTrace; 106 106 db->flags |= SQLITE_WriteSchema | SQLITE_IgnoreChecks; 107 - db->flags &= ~SQLITE_ForeignKeys; 107 + db->flags &= ~(SQLITE_ForeignKeys | SQLITE_ReverseOrder); 108 108 db->xTrace = 0; 109 109 110 110 pMain = db->aDb[0].pBt; 111 111 isMemDb = sqlite3PagerIsMemdb(sqlite3BtreePager(pMain)); 112 112 113 113 /* Attach the temporary database as 'vacuum_db'. The synchronous pragma 114 114 ** can be set to 'off' for this file, as it is not recovered if a crash
Changes to src/vdbe.c.
880 880 */ 881 881 case OP_Int64: { /* out2-prerelease */ 882 882 assert( pOp->p4.pI64!=0 ); 883 883 pOut->u.i = *pOp->p4.pI64; 884 884 break; 885 885 } 886 886 887 +#ifndef SQLITE_OMIT_FLOATING_POINT 887 888 /* Opcode: Real * P2 * P4 * 888 889 ** 889 890 ** P4 is a pointer to a 64-bit floating point value. 890 891 ** Write that value into register P2. 891 892 */ 892 893 case OP_Real: { /* same as TK_FLOAT, out2-prerelease */ 893 894 pOut->flags = MEM_Real; 894 895 assert( !sqlite3IsNaN(*pOp->p4.pReal) ); 895 896 pOut->r = *pOp->p4.pReal; 896 897 break; 897 898 } 899 +#endif 898 900 899 901 /* Opcode: String8 * P2 * P4 * 900 902 ** 901 903 ** P4 points to a nul terminated UTF-8 string. This opcode is transformed 902 904 ** into an OP_String before it is executed for the first time. 903 905 */ 904 906 case OP_String8: { /* same as TK_STRING, out2-prerelease */ ................................................................................ 1291 1293 iB = (i64)rB; 1292 1294 if( iA==0 ) goto arithmetic_result_is_null; 1293 1295 if( iA==-1 ) iA = 1; 1294 1296 rB = (double)(iB % iA); 1295 1297 break; 1296 1298 } 1297 1299 } 1300 +#ifdef SQLITE_OMIT_FLOATING_POINT 1301 + pOut->u.i = rB; 1302 + MemSetTypeFlag(pOut, MEM_Int); 1303 +#else 1298 1304 if( sqlite3IsNaN(rB) ){ 1299 1305 goto arithmetic_result_is_null; 1300 1306 } 1301 1307 pOut->r = rB; 1302 1308 MemSetTypeFlag(pOut, MEM_Real); 1303 1309 if( (flags & MEM_Real)==0 ){ 1304 1310 sqlite3VdbeIntegerAffinity(pOut); 1305 1311 } 1312 +#endif 1306 1313 } 1307 1314 break; 1308 1315 1309 1316 arithmetic_result_is_null: 1310 1317 sqlite3VdbeMemSetNull(pOut); 1311 1318 break; 1312 1319 } ................................................................................ 1526 1533 } 1527 1534 }else{ 1528 1535 MemSetTypeFlag(pIn1, MEM_Int); 1529 1536 } 1530 1537 break; 1531 1538 } 1532 1539 1540 +#ifndef SQLITE_OMIT_FLOATING_POINT 1533 1541 /* Opcode: RealAffinity P1 * * * * 1534 1542 ** 1535 1543 ** If register P1 holds an integer convert it to a real value. 1536 1544 ** 1537 1545 ** This opcode is used when extracting information from a column that 1538 1546 ** has REAL affinity. Such column values may still be stored as 1539 1547 ** integers, for space efficiency, but after extraction we want them ................................................................................ 1542 1550 case OP_RealAffinity: { /* in1 */ 1543 1551 pIn1 = &aMem[pOp->p1]; 1544 1552 if( pIn1->flags & MEM_Int ){ 1545 1553 sqlite3VdbeMemRealify(pIn1); 1546 1554 } 1547 1555 break; 1548 1556 } 1557 +#endif 1549 1558 1550 1559 #ifndef SQLITE_OMIT_CAST 1551 1560 /* Opcode: ToText P1 * * * * 1552 1561 ** 1553 1562 ** Force the value in register P1 to be text. 1554 1563 ** If the value is numeric, convert it to a string using the 1555 1564 ** equivalent of printf(). Blob values are unchanged and ................................................................................ 1625 1634 pIn1 = &aMem[pOp->p1]; 1626 1635 if( (pIn1->flags & MEM_Null)==0 ){ 1627 1636 sqlite3VdbeMemIntegerify(pIn1); 1628 1637 } 1629 1638 break; 1630 1639 } 1631 1640 1632 -#ifndef SQLITE_OMIT_CAST 1641 +#if !defined(SQLITE_OMIT_CAST) && !defined(SQLITE_OMIT_FLOATING_POINT) 1633 1642 /* Opcode: ToReal P1 * * * * 1634 1643 ** 1635 1644 ** Force the value in register P1 to be a floating point number. 1636 1645 ** If The value is currently an integer, convert it. 1637 1646 ** If the value is text or blob, try to convert it to an integer using the 1638 1647 ** equivalent of atoi() and store 0.0 if no such conversion is possible. 1639 1648 ** ................................................................................ 1642 1651 case OP_ToReal: { /* same as TK_TO_REAL, in1 */ 1643 1652 pIn1 = &aMem[pOp->p1]; 1644 1653 if( (pIn1->flags & MEM_Null)==0 ){ 1645 1654 sqlite3VdbeMemRealify(pIn1); 1646 1655 } 1647 1656 break; 1648 1657 } 1649 -#endif /* SQLITE_OMIT_CAST */ 1658 +#endif /* !defined(SQLITE_OMIT_CAST) && !defined(SQLITE_OMIT_FLOATING_POINT) */ 1650 1659 1651 1660 /* Opcode: Lt P1 P2 P3 P4 P5 1652 1661 ** 1653 1662 ** Compare the values in register P1 and P3. If reg(P3)<reg(P1) then 1654 1663 ** jump to address P2. 1655 1664 ** 1656 1665 ** If the SQLITE_JUMPIFNULL bit of P5 is set and either reg(P1) or ................................................................................ 4568 4577 sqlite3BtreeEnterAll(db); 4569 4578 if( pOp->p2 || DbHasProperty(db, iDb, DB_SchemaLoaded) ){ 4570 4579 zMaster = SCHEMA_TABLE(iDb); 4571 4580 initData.db = db; 4572 4581 initData.iDb = pOp->p1; 4573 4582 initData.pzErrMsg = &p->zErrMsg; 4574 4583 zSql = sqlite3MPrintf(db, 4575 - "SELECT name, rootpage, sql FROM '%q'.%s WHERE %s", 4584 + "SELECT name, rootpage, sql FROM '%q'.%s WHERE %s ORDER BY rowid", 4576 4585 db->aDb[iDb].zName, zMaster, pOp->p4.z); 4577 4586 if( zSql==0 ){ 4578 4587 rc = SQLITE_NOMEM; 4579 4588 }else{ 4580 4589 (void)sqlite3SafetyOff(db); 4581 4590 assert( db->init.busy==0 ); 4582 4591 db->init.busy = 1; ................................................................................ 5660 5669 /* 5661 5670 ** The magic Explain opcode are only inserted when explain==2 (which 5662 5671 ** is to say when the EXPLAIN QUERY PLAN syntax is used.) 5663 5672 ** This opcode records information from the optimizer. It is the 5664 5673 ** the same as a no-op. This opcodesnever appears in a real VM program. 5665 5674 */ 5666 5675 default: { /* This is really OP_Noop and OP_Explain */ 5676 + assert( pOp->opcode==OP_Noop || pOp->opcode==OP_Explain ); 5667 5677 break; 5668 5678 } 5669 5679 5670 5680 /***************************************************************************** 5671 5681 ** The cases of the switch statement above this line should all be indented 5672 5682 ** by 6 spaces. But the left-most 6 spaces have been removed to improve the 5673 5683 ** readability. From this point on down, the normal indentation rules are
Changes to src/vdbeInt.h.
358 358 int sqlite3VdbeMemTooBig(Mem*); 359 359 int sqlite3VdbeMemCopy(Mem*, const Mem*); 360 360 void sqlite3VdbeMemShallowCopy(Mem*, const Mem*, int); 361 361 void sqlite3VdbeMemMove(Mem*, Mem*); 362 362 int sqlite3VdbeMemNulTerminate(Mem*); 363 363 int sqlite3VdbeMemSetStr(Mem*, const char*, int, u8, void(*)(void*)); 364 364 void sqlite3VdbeMemSetInt64(Mem*, i64); 365 -void sqlite3VdbeMemSetDouble(Mem*, double); 365 +#ifdef SQLITE_OMIT_FLOATING_POINT 366 +# define sqlite3VdbeMemSetDouble sqlite3VdbeMemSetInt64 367 +#else 368 + void sqlite3VdbeMemSetDouble(Mem*, double); 369 +#endif 366 370 void sqlite3VdbeMemSetNull(Mem*); 367 371 void sqlite3VdbeMemSetZeroBlob(Mem*,int); 368 372 void sqlite3VdbeMemSetRowSet(Mem*); 369 373 int sqlite3VdbeMemMakeWriteable(Mem*); 370 374 int sqlite3VdbeMemStringify(Mem*, int); 371 375 i64 sqlite3VdbeIntValue(Mem*); 372 376 int sqlite3VdbeMemIntegerify(Mem*);
Changes to src/vdbeapi.c.
49 49 rc = SQLITE_OK; 50 50 }else{ 51 51 Vdbe *v = (Vdbe*)pStmt; 52 52 #ifdef SQLITE_ENABLE_SQLRR 53 53 SRRecFinalize(pStmt); 54 54 #endif 55 55 sqlite3 *db = v->db; 56 - if( db==0 ){ 57 - return SQLITE_MISUSE; 58 - } 56 + if( db==0 ) return SQLITE_MISUSE; 59 57 #if SQLITE_THREADSAFE 60 58 sqlite3_mutex *mutex = v->db->mutex; 61 59 #endif 62 60 sqlite3_mutex_enter(mutex); 63 61 rc = sqlite3VdbeFinalize(v); 64 62 rc = sqlite3ApiExit(db, rc); 65 63 sqlite3_mutex_leave(mutex); ................................................................................ 414 412 ** This is the top-level implementation of sqlite3_step(). Call 415 413 ** sqlite3Step() to do most of the work. If a schema error occurs, 416 414 ** call sqlite3Reprepare() and try again. 417 415 */ 418 416 int sqlite3_step(sqlite3_stmt *pStmt){ 419 417 int rc = SQLITE_MISUSE; 420 418 Vdbe *v = (Vdbe*)pStmt; 421 - sqlite3 *db; 422 - if( v && ((db = v->db) != NULL)){ 419 + if( v && (v->db)!=0 ){ 423 420 int cnt = 0; 421 + sqlite3 *db = v->db; 424 422 #ifdef SQLITE_ENABLE_SQLRR 425 423 SRRecStep(pStmt); 426 424 #endif 427 - 428 425 sqlite3_mutex_enter(db->mutex); 429 426 while( (rc = sqlite3Step(v))==SQLITE_SCHEMA 430 427 && cnt++ < 5 431 428 && (rc = sqlite3Reprepare(v))==SQLITE_OK ){ 432 429 sqlite3_reset(pStmt); 433 430 v->expired = 0; 434 431 }
Changes to src/vdbeaux.c.
62 62 } 63 63 64 64 /* 65 65 ** Return the SQL associated with a prepared statement 66 66 */ 67 67 const char *sqlite3_sql(sqlite3_stmt *pStmt){ 68 68 Vdbe *p = (Vdbe *)pStmt; 69 - return ((p && p->isPrepareV2) ? p->zSql : 0); 69 + return (p && p->isPrepareV2) ? p->zSql : 0; 70 70 } 71 71 72 72 /* 73 73 ** Swap all content between two VDBE structures. 74 74 */ 75 75 void sqlite3VdbeSwap(Vdbe *pA, Vdbe *pB){ 76 76 Vdbe tmp, *pTmp; ................................................................................ 2332 2332 releaseMemArray(p->aColName, p->nResColumn*COLNAME_N); 2333 2333 vdbeFreeOpArray(db, p->aOp, p->nOp); 2334 2334 sqlite3DbFree(db, p->aLabel); 2335 2335 sqlite3DbFree(db, p->aColName); 2336 2336 sqlite3DbFree(db, p->zSql); 2337 2337 p->magic = VDBE_MAGIC_DEAD; 2338 2338 sqlite3DbFree(db, p->pFree); 2339 - p->db = NULL; 2339 + p->db = 0; 2340 2340 sqlite3DbFree(db, p); 2341 2341 } 2342 2342 2343 2343 /* 2344 2344 ** Make sure the cursor p is ready to read or write the row to which it 2345 2345 ** was last positioned. Return an error code if an OOM fault or I/O error 2346 2346 ** prevents us from positioning the cursor to its correct position.
Changes to src/vdbemem.c.
311 311 ** there are reports that windows throws an expection 312 312 ** if the floating point value is out of range. (See ticket #2880.) 313 313 ** Because we do not completely understand the problem, we will 314 314 ** take the conservative approach and always do range tests 315 315 ** before attempting the conversion. 316 316 */ 317 317 static i64 doubleToInt64(double r){ 318 +#ifdef SQLITE_OMIT_FLOATING_POINT 319 + /* When floating-point is omitted, double and int64 are the same thing */ 320 + return r; 321 +#else 318 322 /* 319 323 ** Many compilers we encounter do not define constants for the 320 324 ** minimum and maximum 64-bit integers, or they define them 321 325 ** inconsistently. And many do not understand the "LL" notation. 322 326 ** So we define our own static constants here using nothing 323 327 ** larger than a 32-bit integer constant. 324 328 */ ................................................................................ 332 336 ** a very large positive number to an integer results in a very large 333 337 ** negative integer. This makes no sense, but it is what x86 hardware 334 338 ** does so for compatibility we will do the same in software. */ 335 339 return minInt; 336 340 }else{ 337 341 return (i64)r; 338 342 } 343 +#endif 339 344 } 340 345 341 346 /* 342 347 ** Return some kind of integer value which is the best we can do 343 348 ** at representing the value that *pMem describes as an integer. 344 349 ** If pMem is an integer, then the value is exact. If pMem is 345 350 ** a floating-point then the value returned is the integer part. ................................................................................ 525 530 void sqlite3VdbeMemSetInt64(Mem *pMem, i64 val){ 526 531 sqlite3VdbeMemRelease(pMem); 527 532 pMem->u.i = val; 528 533 pMem->flags = MEM_Int; 529 534 pMem->type = SQLITE_INTEGER; 530 535 } 531 536 537 +#ifndef SQLITE_OMIT_FLOATING_POINT 532 538 /* 533 539 ** Delete any previous value and set the value stored in *pMem to val, 534 540 ** manifest type REAL. 535 541 */ 536 542 void sqlite3VdbeMemSetDouble(Mem *pMem, double val){ 537 543 if( sqlite3IsNaN(val) ){ 538 544 sqlite3VdbeMemSetNull(pMem); ................................................................................ 539 545 }else{ 540 546 sqlite3VdbeMemRelease(pMem); 541 547 pMem->r = val; 542 548 pMem->flags = MEM_Real; 543 549 pMem->type = SQLITE_FLOAT; 544 550 } 545 551 } 552 +#endif 546 553 547 554 /* 548 555 ** Delete any previous value and set the value of pMem to be an 549 556 ** empty boolean index. 550 557 */ 551 558 void sqlite3VdbeMemSetRowSet(Mem *pMem){ 552 559 sqlite3 *db = pMem->db;
Changes to test/auth2.test.
98 98 SQLITE_UPDATE sqlite_master rootpage main {} 99 99 SQLITE_UPDATE sqlite_master sql main {} 100 100 SQLITE_READ sqlite_master ROWID main {} 101 101 SQLITE_READ sqlite_master name main {} 102 102 SQLITE_READ sqlite_master rootpage main {} 103 103 SQLITE_READ sqlite_master sql main {} 104 104 SQLITE_READ sqlite_master tbl_name main {} 105 +SQLITE_READ sqlite_master ROWID main {} 105 106 } 106 107 do_test auth2-2.2 { 107 108 set ::authargs {} 108 109 db eval { 109 110 CREATE VIEW v2 AS SELECT x+y AS a, y+z AS b from t2; 110 111 } 111 112 set ::authargs ................................................................................ 117 118 SQLITE_UPDATE sqlite_master rootpage main {} 118 119 SQLITE_UPDATE sqlite_master sql main {} 119 120 SQLITE_READ sqlite_master ROWID main {} 120 121 SQLITE_READ sqlite_master name main {} 121 122 SQLITE_READ sqlite_master rootpage main {} 122 123 SQLITE_READ sqlite_master sql main {} 123 124 SQLITE_READ sqlite_master tbl_name main {} 125 +SQLITE_READ sqlite_master ROWID main {} 124 126 } 125 127 do_test auth2-2.3 { 126 128 set ::authargs {} 127 129 db eval { 128 130 SELECT a, b FROM v2; 129 131 } 130 132 set ::authargs
Changes to test/backup.test.
900 900 execsql { PRAGMA integrity_check } db3 901 901 } {ok} 902 902 903 903 db2 close 904 904 db3 close 905 905 } 906 906 907 + 908 +# Test that if the database is written to via the same database handle being 909 +# used as the source by a backup operation: 910 +# 911 +# 10.1.*: If the db is in-memory, the backup is restarted. 912 +# 10.2.*: If the db is a file, the backup is not restarted. 913 +# 914 +db close 915 +file delete -force test.db test.db-journal 916 +foreach {tn file rc} { 917 + 1 test.db SQLITE_DONE 918 + 2 :memory: SQLITE_OK 919 +} { 920 + do_test backup-10.$tn.1 { 921 + sqlite3 db $file 922 + execsql { 923 + CREATE TABLE t1(a INTEGER PRIMARY KEY, b BLOB); 924 + BEGIN; 925 + INSERT INTO t1 VALUES(NULL, randomblob(200)); 926 + INSERT INTO t1 SELECT NULL, randomblob(200) FROM t1; 927 + INSERT INTO t1 SELECT NULL, randomblob(200) FROM t1; 928 + INSERT INTO t1 SELECT NULL, randomblob(200) FROM t1; 929 + INSERT INTO t1 SELECT NULL, randomblob(200) FROM t1; 930 + INSERT INTO t1 SELECT NULL, randomblob(200) FROM t1; 931 + INSERT INTO t1 SELECT NULL, randomblob(200) FROM t1; 932 + INSERT INTO t1 SELECT NULL, randomblob(200) FROM t1; 933 + INSERT INTO t1 SELECT NULL, randomblob(200) FROM t1; 934 + COMMIT; 935 + SELECT count(*) FROM t1; 936 + } 937 + } {256} 938 + 939 + do_test backup-10.$tn.2 { 940 + set pgs [execsql {pragma page_count}] 941 + expr {$pgs > 50 && $pgs < 75} 942 + } {1} 943 + 944 + do_test backup-10.$tn.3 { 945 + file delete -force bak.db bak.db-journal 946 + sqlite3 db2 bak.db 947 + sqlite3_backup B db2 main db main 948 + B step 50 949 + } {SQLITE_OK} 950 + 951 + do_test backup-10.$tn.4 { 952 + execsql { UPDATE t1 SET b = randomblob(200) WHERE a IN (1, 250) } 953 + } {} 954 + 955 + do_test backup-10.$tn.5 { 956 + B step 50 957 + } $rc 958 + 959 + do_test backup-10.6 { 960 + B finish 961 + } {SQLITE_OK} 962 +} 963 + 964 +db2 close 907 965 finish_test
Changes to test/e_fts3.test.
60 60 # [foreach] loop is testing with OOM errors, disable the lookaside buffer. 61 61 # 62 62 db close 63 63 file delete -force test.db test.db-journal 64 64 sqlite3 db test.db 65 65 if {$DO_MALLOC_TEST} { sqlite3_db_config_lookaside db 0 0 0 } 66 66 db eval "PRAGMA encoding = '$enc'" 67 + 68 +proc mit {blob} { 69 + set scan(littleEndian) i* 70 + set scan(bigEndian) I* 71 + binary scan $blob $scan($::tcl_platform(byteOrder)) r 72 + return $r 73 +} 74 +db func mit mit 67 75 68 76 ########################################################################## 69 77 # Test the example CREATE VIRTUAL TABLE statements in section 1.1 70 78 # of fts3.in. 71 79 # 72 80 ddl_test 1.1.1.1 {CREATE VIRTUAL TABLE data USING fts3()} 73 81 read_test 1.1.1.2 {PRAGMA table_info(data)} {0 content {} 0 {} 0} ................................................................................ 407 415 } {{1 0 5 7 1 0 30 7}} 408 416 read_test 1.7.1.6 { 409 417 SELECT offsets(mail) FROM mail WHERE mail MATCH '"serious mail"' 410 418 } {{1 0 28 7 1 1 36 4}} 411 419 412 420 ddl_test 1.7.2.1 { CREATE VIRTUAL TABLE text USING fts3() } 413 421 414 -write_test 3.2.2 text_content { 422 +write_test 1.7.2.2 text_content { 415 423 INSERT INTO text VALUES(' 416 424 During 30 Nov-1 Dec, 2-3oC drops. Cool in the upper portion, minimum temperature 14-16oC and cool elsewhere, minimum temperature 17-20oC. Cold to very cold on mountaintops, minimum temperature 6-12oC. Northeasterly winds 15-30 km/hr. After that, temperature increases. Northeasterly winds 15-30 km/hr. 417 425 '); 418 426 } 419 427 420 428 read_test 1.7.2.3 { 421 429 SELECT snippet(text) FROM text WHERE text MATCH 'cold' 422 -} {{<b>...</b> elsewhere, minimum temperature 17-20oC. <b>Cold</b> to very <b>cold</b> on mountaintops, minimum <b>...</b>}} 430 +} {{<b>...</b>cool elsewhere, minimum temperature 17-20oC. <b>Cold</b> to very <b>cold</b> on mountaintops, minimum temperature 6<b>...</b>}} 423 431 424 432 read_test 1.7.2.4 { 425 433 SELECT snippet(text, '[', ']', '...') FROM text WHERE text MATCH '"min* tem*"' 426 -} {{... 2-3oC drops. Cool in the upper portion, [minimum] [temperature] 14-16oC and cool elsewhere, [minimum] ...}} 434 +} {{...the upper portion, [minimum] [temperature] 14-16oC and cool elsewhere, [minimum] [temperature] 17-20oC. Cold...}} 435 + 436 +ddl_test 1.7.3.1 { DROP TABLE IF EXISTS t1 } 437 +ddl_test 1.7.3.2 { CREATE VIRTUAL TABLE t1 USING fts3(a, b) } 438 +write_test 1.7.3.3 t1_content { 439 + INSERT INTO t1 VALUES( 440 + 'transaction default models default', 'Non transaction reads'); 441 +} 442 +write_test 1.7.3.4 t1_content { 443 + INSERT INTO t1 VALUES('the default transaction', 'these semantics present'); 444 +} 445 +write_test 1.7.3.5 t1_content { 446 + INSERT INTO t1 VALUES('single request', 'default data'); 447 +} 448 +read_test 1.7.3.6 { 449 + SELECT mit(matchinfo(t1)) FROM t1 450 + WHERE t1 MATCH 'default transaction "these semantics"'; 451 +} {{3 2 1 3 2 0 1 1 1 2 2 0 1 1 0 0 0 1 1 1}} 427 452 428 453 ########################################################################## 429 454 # Test the example in section 5 (custom tokenizers). 430 455 # 431 456 ddl_test 1.8.1.1 { CREATE VIRTUAL TABLE simple USING fts3(tokenize=simple) } 432 457 write_test 1.8.1.2 simple_content { 433 458 INSERT INTO simple VALUES('Right now they''re very frustrated') ................................................................................ 448 473 #------------------------------------------------------------------------- 449 474 450 475 #------------------------------------------------------------------------- 451 476 # Test that errors in the arguments passed to the snippet and offsets 452 477 # functions are handled correctly. 453 478 # 454 479 set DO_MALLOC_TEST 0 480 +ddl_test 2.1.0 { DROP TABLE IF EXISTS t1 } 455 481 ddl_test 2.1.1 { CREATE VIRTUAL TABLE t1 USING fts3(a, b) } 456 482 write_test 2.1.2 t1_content { 457 483 INSERT INTO t1 VALUES('one two three', x'A1B2C3D4E5F6'); 458 484 } 459 485 error_test 2.1.3 { 460 486 SELECT offsets(a) FROM t1 WHERE a MATCH 'one' 461 487 } {illegal first argument to offsets} ................................................................................ 468 494 error_test 2.1.6 { 469 495 SELECT snippet(a) FROM t1 WHERE a MATCH 'one' 470 496 } {illegal first argument to snippet} 471 497 error_test 2.1.7 { 472 498 SELECT snippet() FROM t1 WHERE a MATCH 'one' 473 499 } {unable to use function snippet in the requested context} 474 500 error_test 2.1.8 { 475 - SELECT snippet(a, b, 'A', 'B', 'C') FROM t1 WHERE a MATCH 'one' 501 + SELECT snippet(a, b, 'A', 'B', 'C', 'D', 'E') FROM t1 WHERE a MATCH 'one' 476 502 } {wrong number of arguments to function snippet()} 477 503 #------------------------------------------------------------------------- 478 504 479 505 #------------------------------------------------------------------------- 480 506 # Test the effect of an OOM error while installing the FTS3 module (i.e. 481 507 # opening a database handle). This case was not tested by the OOM testing 482 508 # of the document examples above. ................................................................................ 514 540 set DO_MALLOC_TEST 0 515 541 ddl_test 5.1 { CREATE VIRTUAL TABLE t5 USING fts3(x) } 516 542 write_test 5.2 t5_content { 517 543 INSERT INTO t5 VALUES('In Xanadu did Kubla Khan A stately pleasure-dome decree Where Alph, the sacred river, ran Through caverns measureless to man Down to a sunless sea. So twice five miles of fertile ground With walls and towers were girdled round : And there were gardens bright with sinuous rills, Where blossomed many an incense-bearing tree ; And here were forests ancient as the hills, Enfolding sunny spots of greenery.'); 518 544 } 519 545 read_test 5.3 { 520 546 SELECT snippet(t5) FROM t5 WHERE t5 MATCH 'miles' 521 -} {{<b>...</b> Down to a sunless sea. So twice five <b>miles</b> of fertile ground With walls and towers were <b>...</b>}} 547 +} {{<b>...</b>to a sunless sea. So twice five <b>miles</b> of fertile ground With walls and towers<b>...</b>}} 522 548 read_test 5.4 { 523 549 SELECT snippet(t5, '<i>') FROM t5 WHERE t5 MATCH 'miles' 524 -} {{<b>...</b> Down to a sunless sea. So twice five <i>miles</b> of fertile ground With walls and towers were <b>...</b>}} 550 +} {{<b>...</b>to a sunless sea. So twice five <i>miles</b> of fertile ground With walls and towers<b>...</b>}} 525 551 read_test 5.5 { 526 552 SELECT snippet(t5, '<i>', '</i>') FROM t5 WHERE t5 MATCH 'miles' 527 -} {{<b>...</b> Down to a sunless sea. So twice five <i>miles</i> of fertile ground With walls and towers were <b>...</b>}} 553 +} {{<b>...</b>to a sunless sea. So twice five <i>miles</i> of fertile ground With walls and towers<b>...</b>}} 528 554 read_test 5.6 { 529 555 SELECT snippet(t5, '<i>', '</i>', 'XXX') FROM t5 WHERE t5 MATCH 'miles' 530 -} {{XXX Down to a sunless sea. So twice five <i>miles</i> of fertile ground With walls and towers were XXX}} 556 +} {{XXXto a sunless sea. So twice five <i>miles</i> of fertile ground With walls and towersXXX}} 531 557 #------------------------------------------------------------------------- 532 558 533 559 #------------------------------------------------------------------------- 534 560 # Test that an empty MATCH expression returns an empty result set. As 535 561 # does passing a NULL value as a MATCH expression. 536 562 # 537 563 set DO_MALLOC_TEST 0
Changes to test/fts3_common.tcl.
322 322 } 323 323 324 324 proc doPassiveTest {isRestart name sql catchres} { 325 325 if {![info exists ::DO_MALLOC_TEST]} { set ::DO_MALLOC_TEST 1 } 326 326 327 327 switch $::DO_MALLOC_TEST { 328 328 0 { # No malloc failures. 329 - do_test $name [list catchsql $sql] $catchres 329 + do_test $name [list set {} [uplevel [list catchsql $sql]]] $catchres 330 330 return 331 331 } 332 332 1 { # Simulate transient failures. 333 333 set nRepeat 1 334 334 set zName "transient" 335 335 set nStartLimit 100000 336 336 set nBackup 1
Changes to test/fts3ac.test.
1127 1127 } 1128 1128 } {{Alert Posted 10:00 AM November 20,2000: E-<b>GAS</b> Request <b>Reminder</b>}} 1129 1129 do_test fts3ac-4.2 { 1130 1130 execsql { 1131 1131 SELECT snippet(email) FROM email 1132 1132 WHERE email MATCH 'christmas candlelight' 1133 1133 } 1134 -} {{<b>...</b> place.? What do you think about going here <b>Christmas</b> 1135 -eve?? They have an 11:00 a.m. service and a <b>candlelight</b> service at 5:00 p.m., 1136 -among others. <b>...</b>}} 1134 +} {{<b>...</b>here <b>Christmas</b> 1135 +eve?? They have an 11:00 a.m. service and a <b>candlelight</b> service<b>...</b>}} 1137 1136 1138 1137 do_test fts3ac-4.3 { 1139 1138 execsql { 1140 1139 SELECT snippet(email) FROM email 1141 1140 WHERE email MATCH 'deal sheet potential reuse' 1142 1141 } 1143 -} {{EOL-Accenture <b>Deal</b> <b>Sheet</b> <b>...</b> intent 1144 - Review Enron asset base for <b>potential</b> <b>reuse</b>/ licensing 1145 - Contract negotiations <b>...</b>}} 1142 +} {{EOL-Accenture <b>Deal</b> <b>Sheet</b><b>...</b>asset base for <b>potential</b> <b>reuse</b>/ licensing 1143 + Contract negotiations<b>...</b>}} 1146 1144 do_test fts3ac-4.4 { 1147 1145 execsql { 1148 1146 SELECT snippet(email,'<<<','>>>',' ') FROM email 1149 1147 WHERE email MATCH 'deal sheet potential reuse' 1150 1148 } 1151 -} {{EOL-Accenture <<<Deal>>> <<<Sheet>>> intent 1152 - Review Enron asset base for <<<potential>>> <<<reuse>>>/ licensing 1153 - Contract negotiations }} 1149 +} {{EOL-Accenture <<<Deal>>> <<<Sheet>>> asset base for <<<potential>>> <<<reuse>>>/ licensing 1150 + Contract negotiations }} 1154 1151 do_test fts3ac-4.5 { 1155 1152 execsql { 1156 1153 SELECT snippet(email,'<<<','>>>',' ') FROM email 1157 1154 WHERE email MATCH 'first things' 1158 1155 } 1159 -} {{Re: <<<First>>> Polish Deal! Congrats! <<<Things>>> seem to be building rapidly now on the }} 1156 +} {{Re: <<<First>>> Polish Deal! Congrats! <<<Things>>> seem to be building rapidly now }} 1160 1157 do_test fts3ac-4.6 { 1161 1158 execsql { 1162 1159 SELECT snippet(email) FROM email 1163 1160 WHERE email MATCH 'chris is here' 1164 1161 } 1165 -} {{<b>chris</b>.germany@enron.com <b>...</b> Sounds good to me. I bet this <b>is</b> next to the Warick?? Hotel. <b>...</b> place.? What do you think about going <b>here</b> Christmas 1166 -eve?? They have an 11:00 a.m. <b>...</b>}} 1162 +} {{<b>...</b><b>chris</b>.germany@enron.com'" <<b>chris</b><b>...</b>bet this <b>is</b> next to<b>...</b>about going <b>here</b> Christmas 1163 +eve<b>...</b>}} 1167 1164 do_test fts3ac-4.7 { 1168 1165 execsql { 1169 1166 SELECT snippet(email) FROM email 1170 1167 WHERE email MATCH '"pursuant to"' 1171 1168 } 1172 1169 } {{Erin: 1173 1170 1174 -<b>Pursuant</b> <b>to</b> your request, attached are the Schedule to <b>...</b>}} 1171 +<b>Pursuant</b> <b>to</b> your request, attached are the Schedule to the ISDA Master Agreement, together<b>...</b>}} 1175 1172 do_test fts3ac-4.8 { 1176 1173 execsql { 1177 1174 SELECT snippet(email) FROM email 1178 1175 WHERE email MATCH 'ancillary load davis' 1179 1176 } 1180 -} {{pete.<b>davis</b>@enron.com <b>...</b> Start Date: 4/22/01; HourAhead hour: 3; No <b>ancillary</b> schedules awarded. 1181 -Variances detected. 1182 -Variances detected in <b>Load</b> schedule. 1177 +} {{pete.<b>davis</b>@enron.com<b>...</b>3; No <b>ancillary</b> schedules awarded<b>...</b>detected in <b>Load</b> schedule. 1183 1178 1184 - LOG MESSAGES: 1185 - 1186 -PARSING <b>...</b>}} 1179 + LOG<b>...</b>}} 1187 1180 1188 1181 # Combinations of AND and OR operators: 1189 1182 # 1190 1183 do_test fts3ac-5.1 { 1191 1184 execsql { 1192 1185 SELECT snippet(email) FROM email 1193 1186 WHERE email MATCH 'questar enron OR com' 1194 1187 } 1195 -} {{matt.smith@<b>enron</b>.<b>com</b> <b>...</b> six reports: 1196 - 1197 -31 Keystone Receipts 1188 +} {{matt.smith@<b>enron</b>.<b>com</b><b>...</b>31 Keystone Receipts 1198 1189 15 <b>Questar</b> Pipeline 1199 -40 Rockies Production 1200 -22 West_2 <b>...</b>}} 1190 +40 Rockies<b>...</b>}} 1191 + 1201 1192 do_test fts3ac-5.2 { 1202 1193 execsql { 1203 1194 SELECT snippet(email) FROM email 1204 1195 WHERE email MATCH 'enron OR com questar' 1205 1196 } 1206 -} {{matt.smith@<b>enron</b>.<b>com</b> <b>...</b> six reports: 1207 - 1208 -31 Keystone Receipts 1197 +} {{matt.smith@<b>enron</b>.<b>com</b><b>...</b>31 Keystone Receipts 1209 1198 15 <b>Questar</b> Pipeline 1210 -40 Rockies Production 1211 -22 West_2 <b>...</b>}} 1199 +40 Rockies<b>...</b>}} 1212 1200 1213 1201 finish_test
Changes to test/fts3al.test.
49 49 # exercised. The version with a couple extra spaces should cause the 50 50 # other isspace() call to be exercised. [Both cases have been tested 51 51 # in the debugger, but I'm hoping to continue to catch it if simple 52 52 # constant changes change things slightly. 53 53 # 54 54 # The trailing and leading hi-bit chars help with code which tests for 55 55 # isspace() to coalesce multiple spaces. 56 +# 57 +# UPDATE: The above is no longer true; there is no such code in fts3. 58 +# But leave the test in just the same. 59 +# 56 60 57 61 set word "\x80xxxxx\x80xxxxx\x80xxxxx\x80xxxxx\x80xxxxx\x80xxxxx\x80" 58 62 set phrase1 "$word $word $word target $word $word $word" 59 63 set phrase2 "$word $word $word target $word $word $word" 60 64 61 65 db eval {CREATE VIRTUAL TABLE t4 USING fts3(content)} 62 66 db eval "INSERT INTO t4 (content) VALUES ('$phrase1')" 63 67 db eval "INSERT INTO t4 (content) VALUES ('$phrase2')" 64 68 65 69 do_test fts3al-1.4 { 66 70 execsql {SELECT rowid, length(snippet(t4)) FROM t4 WHERE t4 MATCH 'target'} 67 -} {1 111 2 117} 71 +} {1 241 2 247} 68 72 69 73 finish_test
Changes to test/fts3near.test.
72 72 do_test fts3near-1.14 { 73 73 execsql {SELECT docid FROM t1 WHERE content MATCH 'four NEAR four'} 74 74 } {} 75 75 do_test fts3near-1.15 { 76 76 execsql {SELECT docid FROM t1 WHERE content MATCH 'one NEAR two NEAR one'} 77 77 } {3} 78 78 79 +do_test fts3near-1.16 { 80 + execsql { 81 + SELECT docid FROM t1 WHERE content MATCH '"one three" NEAR/0 "four five"' 82 + } 83 +} {1} 84 +do_test fts3near-1.17 { 85 + execsql { 86 + SELECT docid FROM t1 WHERE content MATCH '"four five" NEAR/0 "one three"' 87 + } 88 +} {1} 89 + 79 90 80 91 # Output format of the offsets() function: 81 92 # 82 93 # <column number> <term number> <starting offset> <number of bytes> 83 94 # 84 95 db eval { 85 96 INSERT INTO t1(content) VALUES('A X B C D A B'); ................................................................................ 150 161 execsql {SELECT offsets(t1) FROM t1 WHERE content MATCH 'two NEAR/2 three'} 151 162 } {{0 0 4 3 0 1 8 5 0 0 14 3 0 1 27 5}} 152 163 do_test fts3near-3.6 { 153 164 execsql { 154 165 SELECT offsets(t1) FROM t1 WHERE content MATCH 'three NEAR/0 "two four"' 155 166 } 156 167 } {{0 0 8 5 0 1 14 3 0 2 18 4}} 168 +breakpoint 157 169 do_test fts3near-3.7 { 158 170 execsql { 159 171 SELECT offsets(t1) FROM t1 WHERE content MATCH '"two four" NEAR/0 three'} 160 172 } {{0 2 8 5 0 0 14 3 0 1 18 4}} 161 173 162 174 db eval { 163 175 INSERT INTO t1(content) VALUES(' ................................................................................ 166 178 CSS2 builds on CSS1 (see [CSS1]) and, with very few exceptions, all valid CSS1 style sheets are valid CSS2 style sheets. CSS2 supports media-specific style sheets so that authors may tailor the presentation of their documents to visual browsers, aural devices, printers, braille devices, handheld devices, etc. This specification also supports content positioning, downloadable fonts, table layout, features for internationalization, automatic counters and numbering, and some properties related to user interface. 167 179 ') 168 180 } 169 181 do_test fts3near-4.1 { 170 182 execsql { 171 183 SELECT snippet(t1) FROM t1 WHERE content MATCH 'specification NEAR supports' 172 184 } 173 -} {{<b>...</b> devices, handheld devices, etc. This <b>specification</b> also <b>supports</b> content positioning, downloadable fonts, <b>...</b>}} 185 +} {{<b>...</b>braille devices, handheld devices, etc. This <b>specification</b> also <b>supports</b> content positioning, downloadable fonts, table layout<b>...</b>}} 174 186 175 187 do_test fts3near-5.1 { 176 188 execsql { 177 189 SELECT docid FROM t1 WHERE content MATCH 'specification attach' 178 190 } 179 191 } {2} 180 192 do_test fts3near-5.2 {
Changes to test/fts3query.test.
95 95 binary scan $blob $scan($::tcl_platform(byteOrder)) r 96 96 return $r 97 97 } 98 98 db func mit mit 99 99 100 100 do_test fts3query-3.3 { 101 101 execsql { SELECT mit(matchinfo(foobar)) FROM foobar WHERE foobar MATCH 'the' } 102 -} {{1 1 3 3}} 102 +} {{1 1 3 3 1}} 103 103 104 104 finish_test 105 105
Changes to test/fts3rnd.test.
155 155 } 156 156 } 157 157 158 158 #lsort -uniq -integer $ret 159 159 set ret 160 160 } 161 161 162 +# This [proc] is used to test the FTS3 matchinfo() function. 163 +# 162 164 proc simple_token_matchinfo {zToken} { 163 - set total(0) 0 164 - set total(1) 0 165 - set total(2) 0 166 165 167 - foreach key [lsort -integer [array names ::t1]] { 166 + set nDoc(0) 0 167 + set nDoc(1) 0 168 + set nDoc(2) 0 169 + set nHit(0) 0 170 + set nHit(1) 0 171 + set nHit(2) 0 172 + 173 + 174 + foreach key [array names ::t1] { 168 175 set value $::t1($key) 169 - set cnt [list] 176 + set a($key) [list] 170 177 foreach i {0 1 2} col $value { 171 - set n [llength [lsearch -all $col $zToken]] 172 - lappend cnt $n 173 - incr total($i) $n 178 + set hit [llength [lsearch -all $col $zToken]] 179 + lappend a($key) $hit 180 + incr nHit($i) $hit 181 + if {$hit>0} { incr nDoc($i) } 174 182 } 175 - if {[lindex [lsort $cnt] end]} { 176 - lappend ret $key [concat 1 3 XXX $cnt] 183 + } 184 + 185 + set ret [list] 186 + foreach docid [lsort -integer [array names a]] { 187 + if { [lindex [lsort -integer $a($docid)] end] } { 188 + set matchinfo [list 1 3] 189 + foreach i {0 1 2} hit $a($docid) { 190 + lappend matchinfo $hit $nHit($i) $nDoc($i) 191 + } 192 + lappend ret $docid $matchinfo 177 193 } 178 194 } 179 - 180 - string map [list XXX "$total(0) $total(1) $total(2)"] $ret 195 + 196 + set ret 181 197 } 182 198 183 199 proc simple_near {termlist nNear} { 184 200 set ret [list] 185 201 186 202 foreach {key value} [array get ::t1] { 187 203 foreach v $value {
Added test/fts3snippet.test.
1 +# 2010 January 07 2 +# 3 +# The author disclaims copyright to this source code. In place of 4 +# a legal notice, here is a blessing: 5 +# 6 +# May you do good and not evil. 7 +# May you find forgiveness for yourself and forgive others. 8 +# May you share freely, never taking more than you give. 9 +# 10 +#************************************************************************* 11 +# 12 +# The tests in this file test the FTS3 auxillary functions offsets(), 13 +# snippet() and matchinfo() work. At time of writing, running this file 14 +# provides full coverage of fts3_snippet.c. 15 +# 16 + 17 +set testdir [file dirname $argv0] 18 +source $testdir/tester.tcl 19 + 20 +# If SQLITE_ENABLE_FTS3 is not defined, omit this file. 21 +ifcapable !fts3 { finish_test ; return } 22 +source $testdir/fts3_common.tcl 23 +source $testdir/malloc_common.tcl 24 + 25 +set sqlite_fts3_enable_parentheses 1 26 +set DO_MALLOC_TEST 0 27 + 28 +# Transform the list $L to its "normal" form. So that it can be compared to 29 +# another list with the same set of elements using [string compare]. 30 +# 31 +proc normalize {L} { 32 + set ret [list] 33 + foreach l $L {lappend ret $l} 34 + return $ret 35 +} 36 + 37 +proc do_offsets_test {name expr args} { 38 + set result [list] 39 + foreach a $args { 40 + lappend result [normalize $a] 41 + } 42 + do_select_test $name { 43 + SELECT offsets(ft) FROM ft WHERE ft MATCH $expr 44 + } $result 45 +} 46 + 47 +# Document text used by a few tests. Contains the English names of all 48 +# integers between 1 and 300. 49 +# 50 +set numbers [normalize { 51 + one two three four five six seven eight nine ten eleven twelve thirteen 52 + fourteen fifteen sixteen seventeen eighteen nineteen twenty twentyone 53 + twentytwo twentythree twentyfour twentyfive twentysix twentyseven 54 + twentyeight twentynine thirty thirtyone thirtytwo thirtythree thirtyfour 55 + thirtyfive thirtysix thirtyseven thirtyeight thirtynine forty fortyone 56 + fortytwo fortythree fortyfour fortyfive fortysix fortyseven fortyeight 57 + fortynine fifty fiftyone fiftytwo fiftythree fiftyfour fiftyfive fiftysix 58 + fiftyseven fiftyeight fiftynine sixty sixtyone sixtytwo sixtythree sixtyfour 59 + sixtyfive sixtysix sixtyseven sixtyeight sixtynine seventy seventyone 60 + seventytwo seventythree seventyfour seventyfive seventysix seventyseven 61 + seventyeight seventynine eighty eightyone eightytwo eightythree eightyfour 62 + eightyfive eightysix eightyseven eightyeight eightynine ninety ninetyone 63 + ninetytwo ninetythree ninetyfour ninetyfive ninetysix ninetyseven 64 + ninetyeight ninetynine onehundred onehundredone onehundredtwo 65 + onehundredthree onehundredfour onehundredfive onehundredsix onehundredseven 66 + onehundredeight onehundrednine onehundredten onehundredeleven 67 + onehundredtwelve onehundredthirteen onehundredfourteen onehundredfifteen 68 + onehundredsixteen onehundredseventeen onehundredeighteen onehundrednineteen 69 + onehundredtwenty onehundredtwentyone onehundredtwentytwo 70 + onehundredtwentythree onehundredtwentyfour onehundredtwentyfive 71 + onehundredtwentysix onehundredtwentyseven onehundredtwentyeight 72 + onehundredtwentynine onehundredthirty onehundredthirtyone 73 + onehundredthirtytwo onehundredthirtythree onehundredthirtyfour 74 + onehundredthirtyfive onehundredthirtysix onehundredthirtyseven 75 + onehundredthirtyeight onehundredthirtynine onehundredforty 76 + onehundredfortyone onehundredfortytwo onehundredfortythree 77 + onehundredfortyfour onehundredfortyfive onehundredfortysix 78 + onehundredfortyseven onehundredfortyeight onehundredfortynine 79 + onehundredfifty onehundredfiftyone onehundredfiftytwo onehundredfiftythree 80 + onehundredfiftyfour onehundredfiftyfive onehundredfiftysix 81 + onehundredfiftyseven onehundredfiftyeight onehundredfiftynine 82 + onehundredsixty onehundredsixtyone onehundredsixtytwo onehundredsixtythree 83 + onehundredsixtyfour onehundredsixtyfive onehundredsixtysix 84 + onehundredsixtyseven onehundredsixtyeight onehundredsixtynine 85 + onehundredseventy onehundredseventyone onehundredseventytwo 86 + onehundredseventythree onehundredseventyfour onehundredseventyfive 87 + onehundredseventysix onehundredseventyseven onehundredseventyeight 88 + onehundredseventynine onehundredeighty onehundredeightyone 89 + onehundredeightytwo onehundredeightythree onehundredeightyfour 90 + onehundredeightyfive onehundredeightysix onehundredeightyseven 91 + onehundredeightyeight onehundredeightynine onehundredninety 92 + onehundredninetyone onehundredninetytwo onehundredninetythree 93 + onehundredninetyfour onehundredninetyfive onehundredninetysix 94 + onehundredninetyseven onehundredninetyeight onehundredninetynine twohundred 95 + twohundredone twohundredtwo twohundredthree twohundredfour twohundredfive 96 + twohundredsix twohundredseven twohundredeight twohundrednine twohundredten 97 + twohundredeleven twohundredtwelve twohundredthirteen twohundredfourteen 98 + twohundredfifteen twohundredsixteen twohundredseventeen twohundredeighteen 99 + twohundrednineteen twohundredtwenty twohundredtwentyone twohundredtwentytwo 100 + twohundredtwentythree twohundredtwentyfour twohundredtwentyfive 101 + twohundredtwentysix twohundredtwentyseven twohundredtwentyeight 102 + twohundredtwentynine twohundredthirty twohundredthirtyone 103 + twohundredthirtytwo twohundredthirtythree twohundredthirtyfour 104 + twohundredthirtyfive twohundredthirtysix twohundredthirtyseven 105 + twohundredthirtyeight twohundredthirtynine twohundredforty 106 + twohundredfortyone twohundredfortytwo twohundredfortythree 107 + twohundredfortyfour twohundredfortyfive twohundredfortysix 108 + twohundredfortyseven twohundredfortyeight twohundredfortynine 109 + twohundredfifty twohundredfiftyone twohundredfiftytwo twohundredfiftythree 110 + twohundredfiftyfour twohundredfiftyfive twohundredfiftysix 111 + twohundredfiftyseven twohundredfiftyeight twohundredfiftynine 112 + twohundredsixty twohundredsixtyone twohundredsixtytwo twohundredsixtythree 113 + twohundredsixtyfour twohundredsixtyfive twohundredsixtysix 114 + twohundredsixtyseven twohundredsixtyeight twohundredsixtynine 115 + twohundredseventy twohundredseventyone twohundredseventytwo 116 + twohundredseventythree twohundredseventyfour twohundredseventyfive 117 + twohundredseventysix twohundredseventyseven twohundredseventyeight 118 + twohundredseventynine twohundredeighty twohundredeightyone 119 + twohundredeightytwo twohundredeightythree twohundredeightyfour 120 + twohundredeightyfive twohundredeightysix twohundredeightyseven 121 + twohundredeightyeight twohundredeightynine twohundredninety 122 + twohundredninetyone twohundredninetytwo twohundredninetythree 123 + twohundredninetyfour twohundredninetyfive twohundredninetysix 124 + twohundredninetyseven twohundredninetyeight twohundredninetynine 125 + threehundred 126 +}] 127 + 128 +foreach {DO_MALLOC_TEST enc} { 129 + 0 utf8 130 + 1 utf8 131 + 1 utf16 132 +} { 133 + 134 + db close 135 + file delete -force test.db 136 + sqlite3 db test.db 137 + db eval "PRAGMA encoding = \"$enc\"" 138 + 139 + # Set variable $T to the test name prefix for this iteration of the loop. 140 + # 141 + set T "fts3snippet-$enc" 142 + 143 + ########################################################################## 144 + # Test the offset function. 145 + # 146 + do_test $T.1.1 { 147 + execsql { 148 + CREATE VIRTUAL TABLE ft USING fts3; 149 + INSERT INTO ft VALUES('xxx xxx xxx xxx'); 150 + } 151 + } {} 152 + do_offsets_test $T.1.2 {xxx} {0 0 0 3 0 0 4 3 0 0 8 3 0 0 12 3} 153 + do_offsets_test $T.1.3 {"xxx xxx"} { 154 + 0 0 0 3 0 0 4 3 0 1 4 3 0 0 8 3 155 + 0 1 8 3 0 1 12 3 156 + } 157 + do_offsets_test $T.1.4 {"xxx xxx" xxx} { 158 + 0 0 0 3 0 2 0 3 0 0 4 3 0 1 4 3 159 + 0 2 4 3 0 0 8 3 0 1 8 3 0 2 8 3 160 + 0 1 12 3 0 2 12 3 161 + } 162 + do_offsets_test $T.1.5 {xxx "xxx xxx"} { 163 + 0 0 0 3 0 1 0 3 0 0 4 3 0 1 4 3 164 + 0 2 4 3 0 0 8 3 0 1 8 3 0 2 8 3 165 + 0 0 12 3 0 2 12 3 166 + } 167 + 168 + do_test $T.2.1 { 169 + set v1 [lrange $numbers 0 99] 170 + execsql { 171 + DROP TABLE IF EXISTS ft; 172 + CREATE VIRTUAL TABLE ft USING fts3(a, b); 173 + INSERT INTO ft VALUES($v1, $numbers); 174 + INSERT INTO ft VALUES($v1, NULL); 175 + } 176 + } {} 177 + 178 + set off [string first "twohundred " $numbers] 179 + do_offsets_test $T.2.1 {twohundred} [list 1 0 $off 10] 180 + 181 + set off [string first "onehundred " $numbers] 182 + do_offsets_test $T.2.2 {onehundred} \ 183 + [list 0 0 $off 10 1 0 $off 10] [list 0 0 $off 10] 184 + 185 + # Test a corruption case: 186 + execsql { UPDATE ft_content SET c1b = 'hello world' WHERE c1b = $numbers } 187 + do_error_test $T.2.3 { 188 + SELECT offsets(ft) FROM ft WHERE ft MATCH 'onehundred' 189 + } {database disk image is malformed} 190 + 191 + ########################################################################## 192 + # Test the snippet function. 193 + # 194 + proc do_snippet_test {name expr iCol nTok args} { 195 + set res [list] 196 + foreach a $args { lappend res [string trim $a] } 197 + do_select_test $name { 198 + SELECT snippet(ft,'{','}','...',$iCol,$nTok) FROM ft WHERE ft MATCH $expr 199 + } $res 200 + } 201 + do_test $T.3.1 { 202 + execsql { 203 + DROP TABLE IF EXISTS ft; 204 + CREATE VIRTUAL TABLE ft USING fts3; 205 + INSERT INTO ft VALUES('one two three four five six seven eight nine ten'); 206 + } 207 + } {} 208 + do_snippet_test $T.3.2 one 0 5 "{one} two three four five..." 209 + do_snippet_test $T.3.3 two 0 5 "one {two} three four five..." 210 + do_snippet_test $T.3.4 three 0 5 "one two {three} four five..." 211 + do_snippet_test $T.3.5 four 0 5 "...two three {four} five six..." 212 + do_snippet_test $T.3.6 five 0 5 "...three four {five} six seven..." 213 + do_snippet_test $T.3.7 six 0 5 "...four five {six} seven eight..." 214 + do_snippet_test $T.3.8 seven 0 5 "...five six {seven} eight nine..." 215 + do_snippet_test $T.3.9 eight 0 5 "...six seven {eight} nine ten" 216 + do_snippet_test $T.3.10 nine 0 5 "...six seven eight {nine} ten" 217 + do_snippet_test $T.3.11 ten 0 5 "...six seven eight nine {ten}" 218 + 219 + do_test $T.4.1 { 220 + execsql { 221 + INSERT INTO ft VALUES( 222 + 'one two three four five ' 223 + || 'six seven eight nine ten ' 224 + || 'eleven twelve thirteen fourteen fifteen ' 225 + || 'sixteen seventeen eighteen nineteen twenty ' 226 + || 'one two three four five ' 227 + || 'six seven eight nine ten ' 228 + || 'eleven twelve thirteen fourteen fifteen ' 229 + || 'sixteen seventeen eighteen nineteen twenty' 230 + ); 231 + } 232 + } {} 233 + 234 + do_snippet_test $T.4.2 {one nine} 0 5 { 235 + {one} two three...eight {nine} ten 236 + } { 237 + {one} two three...eight {nine} ten... 238 + } 239 + 240 + do_snippet_test $T.4.3 {one nine} 0 -5 { 241 + {one} two three four five...six seven eight {nine} ten 242 + } { 243 + {one} two three four five...seven eight {nine} ten eleven... 244 + } 245 + do_snippet_test $T.4.3 {one nineteen} 0 -5 { 246 + ...eighteen {nineteen} twenty {one} two... 247 + } 248 + do_snippet_test $T.4.4 {two nineteen} 0 -5 { 249 + ...eighteen {nineteen} twenty one {two}... 250 + } 251 + do_snippet_test $T.4.5 {three nineteen} 0 -5 { 252 + ...{nineteen} twenty one two {three}... 253 + } 254 + 255 + do_snippet_test $T.4.6 {four nineteen} 0 -5 { 256 + ...two three {four} five six...seventeen eighteen {nineteen} twenty one... 257 + } 258 + do_snippet_test $T.4.7 {four NEAR nineteen} 0 -5 { 259 + ...seventeen eighteen {nineteen} twenty one...two three {four} five six... 260 + } 261 + 262 + do_snippet_test $T.4.8 {four nineteen} 0 5 { 263 + ...three {four} five...eighteen {nineteen} twenty... 264 + } 265 + do_snippet_test $T.4.9 {four NEAR nineteen} 0 5 { 266 + ...eighteen {nineteen} twenty...three {four} five... 267 + } 268 + do_snippet_test $T.4.10 {four NEAR nineteen} 0 -5 { 269 + ...seventeen eighteen {nineteen} twenty one...two three {four} five six... 270 + } 271 + do_snippet_test $T.4.11 {four NOT (nineteen twentyone)} 0 5 { 272 + ...two three {four} five six... 273 + } { 274 + ...two three {four} five six... 275 + } 276 + do_snippet_test $T.4.12 {four OR nineteen NEAR twentyone} 0 5 { 277 + ...two three {four} five six... 278 + } { 279 + ...two three {four} five six... 280 + } 281 + 282 + do_test $T.5.1 { 283 + execsql { 284 + DROP TABLE IF EXISTS ft; 285 + CREATE VIRTUAL TABLE ft USING fts3(a, b, c); 286 + INSERT INTO ft VALUES( 287 + 'one two three four five', 288 + 'four five six seven eight', 289 + 'seven eight nine ten eleven' 290 + ); 291 + } 292 + } {} 293 + 294 + do_snippet_test $T.5.2 {five} -1 3 {...three four {five}} 295 + do_snippet_test $T.5.3 {five} 0 3 {...three four {five}} 296 + do_snippet_test $T.5.4 {five} 1 3 {four {five} six...} 297 + do_snippet_test $T.5.5 {five} 2 3 {seven eight nine...} 298 + 299 + do_test $T.5.6 { 300 + execsql { UPDATE ft SET b = NULL } 301 + } {} 302 + 303 + do_snippet_test $T.5.7 {five} -1 3 {...three four {five}} 304 + do_snippet_test $T.5.8 {five} 0 3 {...three four {five}} 305 + do_snippet_test $T.5.9 {five} 1 3 {} 306 + do_snippet_test $T.5.10 {five} 2 3 {seven eight nine...} 307 + 308 + do_snippet_test $T.5.11 {one "seven eight nine"} -1 -3 { 309 + {one} two three...{seven} {eight} {nine}... 310 + } 311 + 312 + do_test $T.6.1 { 313 + execsql { 314 + DROP TABLE IF EXISTS ft; 315 + CREATE VIRTUAL TABLE ft USING fts3(x); 316 + INSERT INTO ft VALUES($numbers); 317 + } 318 + } {} 319 + do_snippet_test $T.6.2 { 320 + one fifty onehundred onehundredfifty twohundredfifty threehundred 321 + } -1 4 { 322 + {one}...{fifty}...{onehundred}...{onehundredfifty}... 323 + } 324 + do_snippet_test $T.6.3 { 325 + one fifty onehundred onehundredfifty twohundredfifty threehundred 326 + } -1 -4 { 327 + {one} two three four...fortyeight fortynine {fifty} fiftyone...ninetyeight ninetynine {onehundred} onehundredone...onehundredfortyeight onehundredfortynine {onehundredfifty} onehundredfiftyone... 328 + } 329 + 330 + do_test $T.7.1 { 331 + execsql { 332 + BEGIN; 333 + DROP TABLE IF EXISTS ft; 334 + CREATE VIRTUAL TABLE ft USING fts3(x); 335 + } 336 + set testresults [list] 337 + for {set i 1} {$i < 150} {incr i} { 338 + set commas [string repeat , $i] 339 + execsql {INSERT INTO ft VALUES('one' || $commas || 'two')} 340 + lappend testresults "{one}$commas{two}" 341 + } 342 + execsql COMMIT 343 + } {} 344 + do_snippet_test $T.7.2 {one two} -1 3 {*}$testresults 345 + 346 + ########################################################################## 347 + # Test the matchinfo function. 348 + # 349 + proc mit {blob} { 350 + set scan(littleEndian) i* 351 + set scan(bigEndian) I* 352 + binary scan $blob $scan($::tcl_platform(byteOrder)) r 353 + return $r 354 + } 355 + db func mit mit 356 + proc do_matchinfo_test {name expr args} { 357 + set res [list] 358 + foreach a $args { lappend res [normalize $a] } 359 + do_select_test $name { 360 + SELECT mit(matchinfo(ft)) FROM ft WHERE ft MATCH $expr 361 + } $res 362 + } 363 + do_test $T.8.1 { 364 + set ten {one two three four five six seven eight nine ten} 365 + execsql { 366 + DROP TABLE IF EXISTS ft; 367 + CREATE VIRTUAL TABLE ft USING fts3; 368 + INSERT INTO ft VALUES($ten); 369 + INSERT INTO ft VALUES($ten || ' ' || $ten); 370 + } 371 + } {} 372 + 373 + do_matchinfo_test $T.8.2 "one" {1 1 1 3 2} {1 1 2 3 2} 374 + do_matchinfo_test $T.8.3 "one NEAR/3 ten" {2 1 1 1 1 1 1 1} 375 + do_matchinfo_test $T.8.4 "five NEAR/4 ten" \ 376 + {2 1 1 3 2 1 3 2} {2 1 2 3 2 2 3 2} 377 + do_matchinfo_test $T.8.5 "six NEAR/3 ten NEAR/3 two" \ 378 + {3 1 1 1 1 1 1 1 1 1 1} 379 + do_matchinfo_test $T.8.6 "five NEAR/4 ten NEAR/3 two" \ 380 + {3 1 2 2 1 1 1 1 1 1 1} 381 + 382 + do_test $T.9.1 { 383 + execsql { 384 + DROP TABLE IF EXISTS ft; 385 + CREATE VIRTUAL TABLE ft USING fts3(x, y); 386 + } 387 + foreach n {1 2 3} { 388 + set v1 [lrange $numbers 0 [expr $n*100]] 389 + set v2 [string trim [string repeat "$numbers " $n]] 390 + set docid [expr $n * 1000000] 391 + execsql { INSERT INTO ft(docid, x, y) VALUES($docid, $v1, $v2) } 392 + } 393 + } {} 394 + do_matchinfo_test $T.9.2 {two*} \ 395 + { 1 2 1 105 3 101 606 3} \ 396 + { 1 2 3 105 3 202 606 3} \ 397 + { 1 2 101 105 3 303 606 3} 398 + 399 + do_matchinfo_test $T.9.4 {"one* two*"} \ 400 + { 1 2 1 5 3 2 12 3} \ 401 + { 1 2 2 5 3 4 12 3} \ 402 + { 1 2 2 5 3 6 12 3} 403 + 404 + do_matchinfo_test $T.9.5 {twohundredfifty} \ 405 + { 1 2 0 1 1 1 6 3} \ 406 + { 1 2 0 1 1 2 6 3} \ 407 + { 1 2 1 1 1 3 6 3} 408 + 409 + do_matchinfo_test $T.9.6 {"threehundred one"} \ 410 + { 1 2 0 0 0 1 3 2} \ 411 + { 1 2 0 0 0 2 3 2} 412 + 413 + do_matchinfo_test $T.9.7 {one OR fivehundred} \ 414 + { 2 2 1 3 3 1 6 3 0 0 0 0 0 0 } \ 415 + { 2 2 1 3 3 2 6 3 0 0 0 0 0 0 } \ 416 + { 2 2 1 3 3 3 6 3 0 0 0 0 0 0 } 417 + 418 + do_matchinfo_test $T.9.8 {two OR "threehundred one"} \ 419 + { 2 2 1 3 3 1 6 3 0 0 0 0 3 2 } \ 420 + { 2 2 1 3 3 2 6 3 0 0 0 1 3 2 } \ 421 + { 2 2 1 3 3 3 6 3 0 0 0 2 3 2 } 422 + 423 + do_select_test $T.9.9 { 424 + SELECT mit(matchinfo(ft)), mit(matchinfo(ft)) 425 + FROM ft WHERE ft MATCH 'two OR "threehundred one"' 426 + } [normalize { 427 + {2 2 1 3 3 1 6 3 0 0 0 0 3 2} 428 + {2 2 1 3 3 1 6 3 0 0 0 0 3 2} 429 + {2 2 1 3 3 2 6 3 0 0 0 1 3 2} 430 + {2 2 1 3 3 2 6 3 0 0 0 1 3 2} 431 + {2 2 1 3 3 3 6 3 0 0 0 2 3 2} 432 + {2 2 1 3 3 3 6 3 0 0 0 2 3 2} 433 + }] 434 +} 435 + 436 +set sqlite_fts3_enable_parentheses 0 437 +finish_test
Changes to test/insert4.test.
297 297 CREATE INDEX t2_i1 ON t2(x ASC, y COLLATE RTRIM); 298 298 INSERT INTO t2 SELECT * FROM t3; 299 299 } 300 300 set ::sqlite3_xferopt_count 301 301 } {0} 302 302 303 303 304 - 304 +do_test insert4-6.5 { 305 + execsql { 306 + CREATE TABLE t6a(x CHECK( x<>'abc' )); 307 + INSERT INTO t6a VALUES('ABC'); 308 + SELECT * FROM t6a; 309 + } 310 +} {ABC} 311 +do_test insert4-6.6 { 312 + execsql { 313 + CREATE TABLE t6b(x CHECK( x<>'abc' COLLATE nocase )); 314 + } 315 + catchsql { 316 + INSERT INTO t6b SELECT * FROM t6a; 317 + } 318 +} {1 {constraint failed}} 319 +do_test insert4-6.7 { 320 + execsql { 321 + DROP TABLE t6b; 322 + CREATE TABLE t6b(x CHECK( x COLLATE nocase <>'abc' )); 323 + } 324 + catchsql { 325 + INSERT INTO t6b SELECT * FROM t6a; 326 + } 327 +} {1 {constraint failed}} 305 328 306 329 finish_test
Changes to test/minmax3.test.
170 170 do_test minmax3-2.7 { 171 171 execsql { SELECT min(b) FROM t2 WHERE a = 1 AND b<1; } 172 172 } {{}} 173 173 do_test minmax3-2.8 { 174 174 execsql { SELECT min(b) FROM t2 WHERE a = 3 AND b<1; } 175 175 } {{}} 176 176 177 -do_test minmax3-2.1 { 177 +do_test minmax3-3.1 { 178 178 execsql { 179 179 DROP TABLE t2; 180 180 CREATE TABLE t2(a, b); 181 181 CREATE INDEX i3 ON t2(a, b DESC); 182 182 INSERT INTO t2 VALUES(1, NULL); 183 183 INSERT INTO t2 VALUES(1, 1); 184 184 INSERT INTO t2 VALUES(1, 2); ................................................................................ 188 188 INSERT INTO t2 VALUES(2, 2); 189 189 INSERT INTO t2 VALUES(2, 3); 190 190 INSERT INTO t2 VALUES(3, 1); 191 191 INSERT INTO t2 VALUES(3, 2); 192 192 INSERT INTO t2 VALUES(3, 3); 193 193 } 194 194 } {} 195 -do_test minmax3-2.2 { 195 +do_test minmax3-3.2 { 196 196 execsql { SELECT min(b) FROM t2 WHERE a = 1; } 197 197 } {1} 198 -do_test minmax3-2.3 { 198 +do_test minmax3-3.3 { 199 199 execsql { SELECT min(b) FROM t2 WHERE a = 1 AND b>1; } 200 200 } {2} 201 -do_test minmax3-2.4 { 201 +do_test minmax3-3.4 { 202 202 execsql { SELECT min(b) FROM t2 WHERE a = 1 AND b>-1; } 203 203 } {1} 204 -do_test minmax3-2.5 { 204 +do_test minmax3-3.5 { 205 205 execsql { SELECT min(b) FROM t2 WHERE a = 1; } 206 206 } {1} 207 -do_test minmax3-2.6 { 207 +do_test minmax3-3.6 { 208 208 execsql { SELECT min(b) FROM t2 WHERE a = 1 AND b<2; } 209 209 } {1} 210 -do_test minmax3-2.7 { 210 +do_test minmax3-3.7 { 211 211 execsql { SELECT min(b) FROM t2 WHERE a = 1 AND b<1; } 212 212 } {{}} 213 -do_test minmax3-2.8 { 213 +do_test minmax3-3.8 { 214 214 execsql { SELECT min(b) FROM t2 WHERE a = 3 AND b<1; } 215 215 } {{}} 216 + 217 +do_test minmax3-4.1 { 218 + execsql { 219 + CREATE TABLE t4(x); 220 + INSERT INTO t4 VALUES('abc'); 221 + INSERT INTO t4 VALUES('BCD'); 222 + SELECT max(x) FROM t4; 223 + } 224 +} {abc} 225 +do_test minmax3-4.2 { 226 + execsql { 227 + SELECT max(x COLLATE nocase) FROM t4; 228 + } 229 +} {BCD} 230 +do_test minmax3-4.3 { 231 + execsql { 232 + SELECT max(x), max(x COLLATE nocase) FROM t4; 233 + } 234 +} {abc BCD} 235 +do_test minmax3-4.4 { 236 + execsql { 237 + SELECT max(x COLLATE binary), max(x COLLATE nocase) FROM t4; 238 + } 239 +} {abc BCD} 240 +do_test minmax3-4.5 { 241 + execsql { 242 + SELECT max(x COLLATE nocase), max(x COLLATE rtrim) FROM t4; 243 + } 244 +} {BCD abc} 245 +do_test minmax3-4.6 { 246 + execsql { 247 + SELECT max(x COLLATE nocase), max(x) FROM t4; 248 + } 249 +} {BCD abc} 250 +do_test minmax3-4.10 { 251 + execsql { 252 + SELECT min(x) FROM t4; 253 + } 254 +} {BCD} 255 +do_test minmax3-4.11 { 256 + execsql { 257 + SELECT min(x COLLATE nocase) FROM t4; 258 + } 259 +} {abc} 260 +do_test minmax3-4.12 { 261 + execsql { 262 + SELECT min(x), min(x COLLATE nocase) FROM t4; 263 + } 264 +} {BCD abc} 265 +do_test minmax3-4.13 { 266 + execsql { 267 + SELECT min(x COLLATE binary), min(x COLLATE nocase) FROM t4; 268 + } 269 +} {BCD abc} 270 +do_test minmax3-4.14 { 271 + execsql { 272 + SELECT min(x COLLATE nocase), min(x COLLATE rtrim) FROM t4; 273 + } 274 +} {abc BCD} 275 +do_test minmax3-4.15 { 276 + execsql { 277 + SELECT min(x COLLATE nocase), min(x) FROM t4; 278 + } 279 +} {abc BCD} 280 + 216 281 217 282 finish_test
Changes to test/whereA.test.
29 29 db eval { 30 30 PRAGMA reverse_unordered_selects=1; 31 31 SELECT * FROM t1; 32 32 } 33 33 } {3 4.53 {} 2 hello world 1 2 3} 34 34 35 35 do_test whereA-1.3 { 36 + db close 37 + sqlite3 db test.db 38 + db eval { 39 + PRAGMA reverse_unordered_selects=1; 40 + SELECT * FROM t1; 41 + } 42 +} {3 4.53 {} 2 hello world 1 2 3} 43 +do_test whereA-1.4 { 44 + db close 45 + sqlite3 db test.db 36 46 db eval { 37 47 PRAGMA reverse_unordered_selects=1; 38 48 SELECT * FROM t1 ORDER BY rowid; 39 49 } 50 +} {1 2 3 2 hello world 3 4.53 {}} 51 +do_test whereA-1.5 { 52 + db eval { 53 + VACUUM; 54 + SELECT * FROM t1 ORDER BY rowid; 55 + } 40 56 } {1 2 3 2 hello world 3 4.53 {}} 57 +do_test whereA-1.6 { 58 + db eval { 59 + PRAGMA reverse_unordered_selects; 60 + } 61 +} {1} 62 +do_test whereA-1.7 { 63 + db close 64 + sqlite3 db test.db 65 + db eval { 66 + PRAGMA reverse_unordered_selects=1; 67 + VACUUM; 68 + SELECT * FROM t1; 69 + } 70 +} {3 4.53 {} 2 hello world 1 2 3} 41 71 42 72 do_test whereA-2.1 { 43 73 db eval { 44 74 PRAGMA reverse_unordered_selects=0; 45 75 SELECT * FROM t1 WHERE a>0; 46 76 } 47 77 } {1 2 3 2 hello world 3 4.53 {}}
Changes to tool/lemon.c.
405 405 /********************** New code to implement the "acttab" module ***********/ 406 406 /* 407 407 ** This module implements routines use to construct the yy_action[] table. 408 408 */ 409 409 410 410 /* 411 411 ** The state of the yy_action table under construction is an instance of 412 -** the following structure 412 +** the following structure. 413 +** 414 +** The yy_action table maps the pair (state_number, lookahead) into an 415 +** action_number. The table is an array of integers pairs. The state_number 416 +** determines an initial offset into the yy_action array. The lookahead 417 +** value is then added to this initial offset to get an index X into the 418 +** yy_action array. If the aAction[X].lookahead equals the value of the 419 +** of the lookahead input, then the value of the action_number output is 420 +** aAction[X].action. If the lookaheads do not match then the 421 +** default action for the state_number is returned. 422 +** 423 +** All actions associated with a single state_number are first entered 424 +** into aLookahead[] using multiple calls to acttab_action(). Then the 425 +** actions for that single state_number are placed into the aAction[] 426 +** array with a single call to acttab_insert(). The acttab_insert() call 427 +** also resets the aLookahead[] array in preparation for the next 428 +** state number. 413 429 */ 414 430 typedef struct acttab acttab; 415 431 struct acttab { 416 432 int nAction; /* Number of used slots in aAction[] */ 417 433 int nActionAlloc; /* Slots allocated for aAction[] */ 418 434 struct { 419 435 int lookahead; /* Value of the lookahead token */ ................................................................................ 450 466 fprintf(stderr,"Unable to allocate memory for a new acttab."); 451 467 exit(1); 452 468 } 453 469 memset(p, 0, sizeof(*p)); 454 470 return p; 455 471 } 456 472 457 -/* Add a new action to the current transaction set 473 +/* Add a new action to the current transaction set. 474 +** 475 +** This routine is called once for each lookahead for a particular 476 +** state. 458 477 */ 459 478 void acttab_action(acttab *p, int lookahead, int action){ 460 479 if( p->nLookahead>=p->nLookaheadAlloc ){ 461 480 p->nLookaheadAlloc += 25; 462 481 p->aLookahead = realloc( p->aLookahead, 463 482 sizeof(p->aLookahead[0])*p->nLookaheadAlloc ); 464 483 if( p->aLookahead==0 ){ ................................................................................ 487 506 ** into the current action table. Then reset the transaction set back 488 507 ** to an empty set in preparation for a new round of acttab_action() calls. 489 508 ** 490 509 ** Return the offset into the action table of the new transaction. 491 510 */ 492 511 int acttab_insert(acttab *p){ 493 512 int i, j, k, n; 494 - int nActtab; /* Number of slots in the p->aAction[] table */ 495 513 assert( p->nLookahead>0 ); 496 514 497 515 /* Make sure we have enough space to hold the expanded action table 498 516 ** in the worst case. The worst case occurs if the transaction set 499 517 ** must be appended to the current action table 500 518 */ 501 519 n = p->mxLookahead + 1; 502 - nActtab = p->nAction + n; 503 - if( nActtab >= p->nActionAlloc ){ 520 + if( p->nAction + n >= p->nActionAlloc ){ 504 521 int oldAlloc = p->nActionAlloc; 505 522 p->nActionAlloc = p->nAction + n + p->nActionAlloc + 20; 506 523 p->aAction = realloc( p->aAction, 507 524 sizeof(p->aAction[0])*p->nActionAlloc); 508 525 if( p->aAction==0 ){ 509 526 fprintf(stderr,"malloc failed\n"); 510 527 exit(1); ................................................................................ 511 528 } 512 529 for(i=oldAlloc; i<p->nActionAlloc; i++){ 513 530 p->aAction[i].lookahead = -1; 514 531 p->aAction[i].action = -1; 515 532 } 516 533 } 517 534 518 - /* Scan the existing action table looking for an offset where we can 519 - ** insert the current transaction set. Fall out of the loop when that 520 - ** offset is found. In the worst case, we fall out of the loop when 521 - ** i reaches nActtab, which means we append the new transaction set. 535 + /* Scan the existing action table looking for an offset that is a 536 + ** duplicate of the current transaction set. Fall out of the loop 537 + ** if and when the duplicate is found. 522 538 ** 523 539 ** i is the index in p->aAction[] where p->mnLookahead is inserted. 524 540 */ 525 - for(i=nActtab-1; i>=0; i--){ 526 - /* First look for an existing action table entry that can be reused */ 541 + for(i=p->nAction-1; i>=0; i--){ 527 542 if( p->aAction[i].lookahead==p->mnLookahead ){ 543 + /* All lookaheads and actions in the aLookahead[] transaction 544 + ** must match against the candidate aAction[i] entry. */ 528 545 if( p->aAction[i].action!=p->mnAction ) continue; 529 546 for(j=0; j<p->nLookahead; j++){ 530 547 k = p->aLookahead[j].lookahead - p->mnLookahead + i; 531 548 if( k<0 || k>=p->nAction ) break; 532 549 if( p->aLookahead[j].lookahead!=p->aAction[k].lookahead ) break; 533 550 if( p->aLookahead[j].action!=p->aAction[k].action ) break; 534 551 } 535 552 if( j<p->nLookahead ) continue; 553 + 554 + /* No possible lookahead value that is not in the aLookahead[] 555 + ** transaction is allowed to match aAction[i] */ 536 556 n = 0; 537 557 for(j=0; j<p->nAction; j++){ 538 558 if( p->aAction[j].lookahead<0 ) continue; 539 559 if( p->aAction[j].lookahead==j+p->mnLookahead-i ) n++; 540 560 } 541 561 if( n==p->nLookahead ){ 542 - break; /* Same as a prior transaction set */ 562 + break; /* An exact match is found at offset i */ 543 563 } 544 564 } 545 565 } 566 + 567 + /* If no existing offsets exactly match the current transaction, find an 568 + ** an empty offset in the aAction[] table in which we can add the 569 + ** aLookahead[] transaction. 570 + */ 546 571 if( i<0 ){ 547 - /* If no reusable entry is found, look for an empty slot */ 548 - for(i=0; i<nActtab; i++){ 572 + /* Look for holes in the aAction[] table that fit the current 573 + ** aLookahead[] transaction. Leave i set to the offset of the hole. 574 + ** If no holes are found, i is left at p->nAction, which means the 575 + ** transaction will be appended. */ 576 + for(i=0; i<p->nActionAlloc - p->mxLookahead; i++){ 549 577 if( p->aAction[i].lookahead<0 ){ 550 578 for(j=0; j<p->nLookahead; j++){ 551 579 k = p->aLookahead[j].lookahead - p->mnLookahead + i; 552 580 if( k<0 ) break; 553 581 if( p->aAction[k].lookahead>=0 ) break; 554 582 } 555 583 if( j<p->nLookahead ) continue;
Added tool/restore_jrnl.tcl.
1 +# 2010 January 7 2 +# 3 +# The author disclaims copyright to this source code. In place of 4 +# a legal notice, here is a blessing: 5 +# 6 +# May you do good and not evil. 7 +# May you find forgiveness for yourself and forgive others. 8 +# May you share freely, never taking more than you give. 9 +# 10 +#*********************************************************************** 11 +# This file implements utility functions for SQLite library. 12 +# 13 +# This file attempts to restore the header of a journal. 14 +# This may be useful for rolling-back the last committed 15 +# transaction from a recovered journal. 16 +# 17 + 18 +package require sqlite3 19 + 20 +set parm_error 0 21 +set fix_chksums 0 22 +set dump_pages 0 23 +set db_name "" 24 + 25 +for {set i 0} {$i<$argc} {incr i} { 26 + if {[lindex $argv $i] == "-fix_chksums"} { 27 + set fix_chksums -1 28 + } elseif {[lindex $argv $i] == "-dump_pages"} { 29 + set dump_pages -1 30 + } elseif {$db_name == ""} { 31 + set db_name [lindex $argv $i] 32 + set jrnl_name $db_name-journal 33 + } else { 34 + set parm_error -1 35 + } 36 +} 37 +if {$parm_error || $db_name == ""} { 38 + puts "USAGE: restore_jrnl.tcl \[-fix_chksums\] \[-dump_pages\] db_name" 39 + puts "Example: restore_jrnl.tcl foo.sqlite" 40 + return 41 +} 42 + 43 +# is there a way to determine this? 44 +set sectsz 512 45 + 46 +# Copy file $from into $to 47 +# 48 +proc copy_file {from to} { 49 + file copy -force $from $to 50 +} 51 + 52 +# Execute some SQL 53 +# 54 +proc catchsql {sql} { 55 + set rc [catch {uplevel [list db eval $sql]} msg] 56 + list $rc $msg 57 +} 58 + 59 +# Perform a test 60 +# 61 +proc do_test {name cmd expected} { 62 + puts -nonewline "$name ..." 63 + set res [uplevel $cmd] 64 + if {$res eq $expected} { 65 + puts Ok 66 + } else { 67 + puts Error 68 + puts " Got: $res" 69 + puts " Expected: $expected" 70 + } 71 +} 72 + 73 +# Calc checksum nonce from journal page data. 74 +# 75 +proc calc_nonce {jrnl_pgno} { 76 + global sectsz 77 + global db_pgsz 78 + global jrnl_name 79 + set jrnl_pg_offset [expr $sectsz+((4+$db_pgsz+4)*$jrnl_pgno)] 80 + set nonce [hexio_get_int [hexio_read $jrnl_name [expr $jrnl_pg_offset+4+$db_pgsz] 4]] 81 + for {set i [expr $db_pgsz-200]} {$i>0} {set i [expr $i-200]} { 82 + set byte [hexio_get_int [hexio_read $jrnl_name [expr $jrnl_pg_offset+4+$i] 1]] 83 + set nonce [expr $nonce-$byte] 84 + } 85 + return $nonce 86 +} 87 + 88 +# Calc checksum from journal page data. 89 +# 90 +proc calc_chksum {jrnl_pgno} { 91 + global sectsz 92 + global db_pgsz 93 + global jrnl_name 94 + global nonce 95 + set jrnl_pg_offset [expr $sectsz+((4+$db_pgsz+4)*$jrnl_pgno)] 96 + set chksum $nonce 97 + for {set i [expr $db_pgsz-200]} {$i>0} {set i [expr $i-200]} { 98 + set byte [hexio_get_int [hexio_read $jrnl_name [expr $jrnl_pg_offset+4+$i] 1]] 99 + set chksum [expr $chksum+$byte] 100 + } 101 + return $chksum 102 +} 103 + 104 +# Print journal page data in hex dump form 105 +# 106 +proc dump_jrnl_page {jrnl_pgno} { 107 + global sectsz 108 + global db_pgsz 109 + global jrnl_name 110 + 111 + # print a header block for the page 112 + puts [string repeat "-" 79] 113 + set jrnl_pg_offset [expr $sectsz+((4+$db_pgsz+4)*$jrnl_pgno)] 114 + set db_pgno [hexio_get_int [hexio_read $jrnl_name [expr $jrnl_pg_offset] 4]] 115 + set chksum [hexio_get_int [hexio_read $jrnl_name [expr $jrnl_pg_offset+4+$db_pgsz] 4]] 116 + set nonce [calc_nonce $jrnl_pgno] 117 + puts [ format {jrnl_pg_offset: %08x (%d) jrnl_pgno: %d db_pgno: %d} \ 118 + $jrnl_pg_offset $jrnl_pg_offset \ 119 + $jrnl_pgno $db_pgno] 120 + puts [ format {nonce: %08x chksum: %08x} \ 121 + $nonce $chksum] 122 + 123 + # now hex dump the data 124 + # This is derived from the Tcler's WIKI 125 + set fid [open $jrnl_name r] 126 + fconfigure $fid -translation binary -encoding binary 127 + seek $fid [expr $jrnl_pg_offset+4] 128 + set data [read $fid $db_pgsz] 129 + close $fid 130 + for {set addr 0} {$addr<$db_pgsz} {set addr [expr $addr+16]} { 131 + # get 16 bytes of data 132 + set s [string range $data $addr [expr $addr+16]] 133 + 134 + # Convert the data to hex and to characters. 135 + binary scan $s H*@0a* hex ascii 136 + 137 + # Replace non-printing characters in the data. 138 + regsub -all -- {[^[:graph:] ]} $ascii {.} ascii 139 + 140 + # Split the 16 bytes into two 8-byte chunks 141 + regexp -- {(.{16})(.{0,16})} $hex -> hex1 hex2 142 + 143 + # Convert the hex to pairs of hex digits 144 + regsub -all -- {..} $hex1 {& } hex1 145 + regsub -all -- {..} $hex2 {& } hex2 146 + 147 + # Print the hex and ascii data 148 + puts [ format {%08x %-24s %-24s %-16s} \ 149 + $addr $hex1 $hex2 $ascii ] 150 + } 151 +} 152 + 153 +# Setup for the tests. Make a backup copy of the files. 154 +# 155 +if [file exist $db_name.org] { 156 + puts "ERROR: during back-up: $db_name.org exists already." 157 + return; 158 +} 159 +if [file exist $jrnl_name.org] { 160 + puts "ERROR: during back-up: $jrnl_name.org exists already." 161 + return 162 +} 163 +copy_file $db_name $db_name.org 164 +copy_file $jrnl_name $jrnl_name.org 165 + 166 +set db_fsize [file size $db_name] 167 +set db_pgsz [hexio_get_int [hexio_read $db_name 16 2]] 168 +set db_npage [expr {$db_fsize / $db_pgsz}] 169 + 170 +set jrnl_fsize [file size $jrnl_name] 171 +set jrnl_npage [expr {($jrnl_fsize - $sectsz) / (4 + $db_pgsz + 4)}] 172 + 173 +# calculate checksum nonce for first page 174 +set nonce [calc_nonce 0] 175 + 176 +# verify all the pages in the journal use the same nonce 177 +for {set i 1} {$i<$jrnl_npage} {incr i} { 178 + set tnonce [calc_nonce $i] 179 + if {$tnonce != $nonce} { 180 + puts "WARNING: different nonces: 0=$nonce $i=$tnonce" 181 + if {$fix_chksums } { 182 + set jrnl_pg_offset [expr $sectsz+((4+$db_pgsz+4)*$i)] 183 + set tchksum [calc_chksum $i] 184 + hexio_write $jrnl_name [expr $jrnl_pg_offset+4+$db_pgsz] [format %08x $tchksum] 185 + puts "INFO: fixing chksum: $i=$tchksum" 186 + } 187 + } 188 +} 189 + 190 +# verify all the page numbers in the journal 191 +for {set i 0} {$i<$jrnl_npage} {incr i} { 192 + set jrnl_pg_offset [expr $sectsz+((4+$db_pgsz+4)*$i)] 193 + set db_pgno [hexio_get_int [hexio_read $jrnl_name $jrnl_pg_offset 4]] 194 + if {$db_pgno < 1} { 195 + puts "WARNING: page number < 1: $i=$db_pgno" 196 + } 197 + if {$db_pgno >= $db_npage} { 198 + puts "WARNING: page number >= $db_npage: $i=$db_pgno" 199 + } 200 +} 201 + 202 +# dump page data 203 +if {$dump_pages} { 204 + for {set i 0} {$i<$jrnl_npage} {incr i} { 205 + dump_jrnl_page $i 206 + } 207 +} 208 + 209 +# write the 8 byte magic string 210 +hexio_write $jrnl_name 0 d9d505f920a163d7 211 + 212 +# write -1 for number of records 213 +hexio_write $jrnl_name 8 ffffffff 214 + 215 +# write 00 for checksum nonce 216 +hexio_write $jrnl_name 12 [format %08x $nonce] 217 + 218 +# write page count 219 +hexio_write $jrnl_name 16 [format %08x $db_npage] 220 + 221 +# write sector size 222 +hexio_write $jrnl_name 20 [format %08x $sectsz] 223 + 224 +# write page size 225 +hexio_write $jrnl_name 24 [format %08x $db_pgsz] 226 + 227 +# check the integrity of the database with the patched journal 228 +sqlite3 db $db_name 229 +do_test restore_jrnl-1.0 { 230 + catchsql {PRAGMA integrity_check} 231 +} {0 ok} 232 +db close 233 +
Changes to tool/shell2.test.
95 95 END; 96 96 97 97 UPDATE OR REPLACE t5 SET a = 4 WHERE a = 1; 98 98 } 99 99 } {1 {Error: near line 9: too many levels of trigger recursion}} 100 100 101 101 102 + 103 +# Shell not echoing all commands with echo on. 104 +# Ticket [eb620916be]. 105 + 106 +# Test with echo off 107 +# NB. whitespace is important 108 +do_test shell2-1.4.1 { 109 + file delete -force foo.db 110 + catchcmd "foo.db" {CREATE TABLE foo(a); 111 +INSERT INTO foo(a) VALUES(1); 112 +SELECT * FROM foo;} 113 +} {0 1} 114 + 115 +# Test with echo on using command line option 116 +# NB. whitespace is important 117 +do_test shell2-1.4.2 { 118 + file delete -force foo.db 119 + catchcmd "-echo foo.db" {CREATE TABLE foo(a); 120 +INSERT INTO foo(a) VALUES(1); 121 +SELECT * FROM foo;} 122 +} {0 {CREATE TABLE foo(a); 123 +INSERT INTO foo(a) VALUES(1); 124 +SELECT * FROM foo; 125 +1}} 126 + 127 +# Test with echo on using dot command 128 +# NB. whitespace is important 129 +do_test shell2-1.4.3 { 130 + file delete -force foo.db 131 + catchcmd "foo.db" {.echo ON 132 +CREATE TABLE foo(a); 133 +INSERT INTO foo(a) VALUES(1); 134 +SELECT * FROM foo;} 135 +} {0 {CREATE TABLE foo(a); 136 +INSERT INTO foo(a) VALUES(1); 137 +SELECT * FROM foo; 138 +1}} 139 + 140 +# Test with echo on using dot command and 141 +# turning off mid- processing. 142 +# NB. whitespace is important 143 +do_test shell2-1.4.4 { 144 + file delete -force foo.db 145 + catchcmd "foo.db" {.echo ON 146 +CREATE TABLE foo(a); 147 +.echo OFF 148 +INSERT INTO foo(a) VALUES(1); 149 +SELECT * FROM foo;} 150 +} {0 {CREATE TABLE foo(a); 151 +.echo OFF 152 +1}} 153 + 154 +# Test with echo on using dot command and 155 +# multiple commands per line. 156 +# NB. whitespace is important 157 +do_test shell2-1.4.5 { 158 + file delete -force foo.db 159 + catchcmd "foo.db" {.echo ON 160 +CREATE TABLE foo1(a); 161 +INSERT INTO foo1(a) VALUES(1); 162 +CREATE TABLE foo2(b); 163 +INSERT INTO foo2(b) VALUES(1); 164 +SELECT * FROM foo1; SELECT * FROM foo2; 165 +INSERT INTO foo1(a) VALUES(2); INSERT INTO foo2(b) VALUES(2); 166 +SELECT * FROM foo1; SELECT * FROM foo2; 167 +} 168 +} {0 {CREATE TABLE foo1(a); 169 +INSERT INTO foo1(a) VALUES(1); 170 +CREATE TABLE foo2(b); 171 +INSERT INTO foo2(b) VALUES(1); 172 +SELECT * FROM foo1; 173 +1 174 +SELECT * FROM foo2; 175 +1 176 +INSERT INTO foo1(a) VALUES(2); 177 +INSERT INTO foo2(b) VALUES(2); 178 +SELECT * FROM foo1; 179 +1 180 +2 181 +SELECT * FROM foo2; 182 +1 183 +2}} 184 + 185 +# Test with echo on and headers on using dot command and 186 +# multiple commands per line. 187 +# NB. whitespace is important 188 +do_test shell2-1.4.6 { 189 + file delete -force foo.db 190 + catchcmd "foo.db" {.echo ON 191 +.headers ON 192 +CREATE TABLE foo1(a); 193 +INSERT INTO foo1(a) VALUES(1); 194 +CREATE TABLE foo2(b); 195 +INSERT INTO foo2(b) VALUES(1); 196 +SELECT * FROM foo1; SELECT * FROM foo2; 197 +INSERT INTO foo1(a) VALUES(2); INSERT INTO foo2(b) VALUES(2); 198 +SELECT * FROM foo1; SELECT * FROM foo2; 199 +} 200 +} {0 {.headers ON 201 +CREATE TABLE foo1(a); 202 +INSERT INTO foo1(a) VALUES(1); 203 +CREATE TABLE foo2(b); 204 +INSERT INTO foo2(b) VALUES(1); 205 +SELECT * FROM foo1; 206 +a 207 +1 208 +SELECT * FROM foo2; 209 +b 210 +1 211 +INSERT INTO foo1(a) VALUES(2); 212 +INSERT INTO foo2(b) VALUES(2); 213 +SELECT * FROM foo1; 214 +a 215 +1 216 +2 217 +SELECT * FROM foo2; 218 +b 219 +1 220 +2}}