SQLite

Check-in [96499b1dd6]
Login

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

Overview
Comment:Update the Apple OS-X branch to include all of the latest changes in trunk.
Downloads: Tarball | ZIP archive
Timelines: family | ancestors | descendants | both | apple-osx
Files: files | file ages | folders
SHA1: 96499b1dd69a0954e30e4b5b4f2272ab306afcdb
User & Date: drh 2010-01-20 01:26:07.000
Context
2010-01-20
13:20
Align the os_unix.c source file with the version found on trunk. (check-in: fa0f6c140e user: drh tags: apple-osx)
01:26
Update the Apple OS-X branch to include all of the latest changes in trunk. (check-in: 96499b1dd6 user: drh tags: apple-osx)
01:20
Manually copy over the rebustness fixes from the apple-osx branch. (check-in: 095c74ea62 user: drh tags: trunk)
2010-01-19
23:50
robustness fixes for preventing a finalized statement from being reused (check-in: a7a0c8d644 user: adam tags: apple-osx)
Changes
Unified Diff Show Whitespace Changes Patch
Changes to ext/fts3/fts3.c.
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
**     * The FTS3 module is being built as an extension
**       (in which case SQLITE_CORE is not defined), or
**
**     * The FTS3 module is being built into the core of
**       SQLite (in which case SQLITE_ENABLE_FTS3 is defined).
*/

/* TODO(shess) Consider exporting this comment to an HTML file or the
** wiki.
*/
/* The full-text index is stored in a series of b+tree (-like)
** structures called segments which map terms to doclists.  The
** structures are like b+trees in layout, but are constructed from the
** bottom up in optimal fashion and are not updatable.  Since trees
** are built from the bottom up, things will be described from the
** bottom up.
**







<
<
<







19
20
21
22
23
24
25



26
27
28
29
30
31
32
**     * The FTS3 module is being built as an extension
**       (in which case SQLITE_CORE is not defined), or
**
**     * The FTS3 module is being built into the core of
**       SQLite (in which case SQLITE_ENABLE_FTS3 is defined).
*/




/* The full-text index is stored in a series of b+tree (-like)
** structures called segments which map terms to doclists.  The
** structures are like b+trees in layout, but are constructed from the
** bottom up in optimal fashion and are not updatable.  Since trees
** are built from the bottom up, things will be described from the
** bottom up.
**
44
45
46
47
48
49
50
51









52
53
54
55
56
57





58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85














86
87
88
89
90
91
92
**         B = 1xxxxxxx    7 bits of data and one flag bit
**
**  7 bits - A
** 14 bits - BA
** 21 bits - BBA
** and so on.
**
** This is identical to how sqlite encodes varints (see util.c).









**
**
**** Document lists ****
** A doclist (document list) holds a docid-sorted list of hits for a
** given term.  Doclists hold docids, and can optionally associate
** token positions and offsets with docids.





**
** A DL_POSITIONS_OFFSETS doclist is stored like this:
**
** array {
**   varint docid;
**   array {                (position list for column 0)
**     varint position;     (delta from previous position plus POS_BASE)
**     varint startOffset;  (delta from previous startOffset)
**     varint endOffset;    (delta from startOffset)
**   }
**   array {
**     varint POS_COLUMN;   (marks start of position list for new column)
**     varint column;       (index of new column)
**     array {
**       varint position;   (delta from previous position plus POS_BASE)
**       varint startOffset;(delta from previous startOffset)
**       varint endOffset;  (delta from startOffset)
**     }
**   }
**   varint POS_END;        (marks end of positions for this document.
** }
**
** Here, array { X } means zero or more occurrences of X, adjacent in
** memory.  A "position" is an index of a token in the token stream
** generated by the tokenizer, while an "offset" is a byte offset,
** both based at 0.  Note that POS_END and POS_COLUMN occur in the
** same logical place as the position element, and act as sentinals
** ending a position list array.














**
** A DL_POSITIONS doclist omits the startOffset and endOffset
** information.  A DL_DOCIDS doclist omits both the position and
** offset information, becoming an array of varint-encoded docids.
**
** On-disk data is stored as type DL_DEFAULT, so we don't serialize
** the type.  Due to how deletion is implemented in the segmentation







|
>
>
>
>
>
>
>
>
>





|
>
>
>
>
>







<
<






<
<







<
|
|
|
>
>
>
>
>
>
>
>
>
>
>
>
>
>







41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75


76
77
78
79
80
81


82
83
84
85
86
87
88

89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
**         B = 1xxxxxxx    7 bits of data and one flag bit
**
**  7 bits - A
** 14 bits - BA
** 21 bits - BBA
** and so on.
**
** This is similar in concept to how sqlite encodes "varints" but
** the encoding is not the same.  SQLite varints are big-endian
** are are limited to 9 bytes in length whereas FTS3 varints are
** little-endian and can be upt to 10 bytes in length (in theory).
**
** Example encodings:
**
**     1:    0x01
**   127:    0x7f
**   128:    0x81 0x00
**
**
**** Document lists ****
** A doclist (document list) holds a docid-sorted list of hits for a
** given term.  Doclists hold docids, and can optionally associate
** token positions and offsets with docids.  A position is the index
** of a word within the document.  The first word of the document has
** a position of 0.
**
** FTS3 used to optionally store character offsets using a compile-time
** option.  But that functionality is no longer supported.
**
** A DL_POSITIONS_OFFSETS doclist is stored like this:
**
** array {
**   varint docid;
**   array {                (position list for column 0)
**     varint position;     (delta from previous position plus POS_BASE)


**   }
**   array {
**     varint POS_COLUMN;   (marks start of position list for new column)
**     varint column;       (index of new column)
**     array {
**       varint position;   (delta from previous position plus POS_BASE)


**     }
**   }
**   varint POS_END;        (marks end of positions for this document.
** }
**
** Here, array { X } means zero or more occurrences of X, adjacent in
** memory.  A "position" is an index of a token in the token stream

** generated by the tokenizer. Note that POS_END and POS_COLUMN occur 
** in the same logical place as the position element, and act as sentinals
** ending a position list array.  POS_END is 0.  POS_COLUMN is 1.
** The positions numbers are not stored literally but rather as two more
** the difference from the prior position, or the just the position plus
** 2 for the first position.  Example:
**
**   label:       A B C D E  F  G H   I  J K
**   value:     123 5 9 1 1 14 35 0 234 72 0
**
** The 123 value is the first docid.  For column zero in this document
** there are two matches at positions 3 and 10 (5-2 and 9-2+3).  The 1
** at D signals the start of a new column; the 1 at E indicates that the
** new column is column number 1.  There are two positions at 12 and 45
** (14-2 and 35-2+12).  The 0 at H indicate the end-of-document.  The
** 234 at I is the next docid.  It has one position 72 (72-2) and then
** terminates with the 0 at K.
**
** A DL_POSITIONS doclist omits the startOffset and endOffset
** information.  A DL_DOCIDS doclist omits both the position and
** offset information, becoming an array of varint-encoded docids.
**
** On-disk data is stored as type DL_DEFAULT, so we don't serialize
** the type.  Due to how deletion is implemented in the segmentation
384
385
386
387
388
389
390





391
392
393
394
395
396






397
398
399
400
401
402
403
        z[iOut++] = z[iIn++];
      }
    }
    z[iOut] = '\0';
  }
}






static void fts3GetDeltaVarint(char **pp, sqlite3_int64 *pVal){
  sqlite3_int64 iVal;
  *pp += sqlite3Fts3GetVarint(*pp, &iVal);
  *pVal += iVal;
}







static void fts3GetDeltaVarint2(char **pp, char *pEnd, sqlite3_int64 *pVal){
  if( *pp>=pEnd ){
    *pp = 0;
  }else{
    fts3GetDeltaVarint(pp, pVal);
  }
}







>
>
>
>
>






>
>
>
>
>
>







404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
        z[iOut++] = z[iIn++];
      }
    }
    z[iOut] = '\0';
  }
}

/*
** Read a single varint from the doclist at *pp and advance *pp to point
** to the next element of the varlist.  Add the value of the varint
** to *pVal.
*/
static void fts3GetDeltaVarint(char **pp, sqlite3_int64 *pVal){
  sqlite3_int64 iVal;
  *pp += sqlite3Fts3GetVarint(*pp, &iVal);
  *pVal += iVal;
}

/*
** As long as *pp has not reached its end (pEnd), then do the same
** as fts3GetDeltaVarint(): read a single varint and add it to *pVal.
** But if we have reached the end of the varint, just set *pp=0 and
** leave *pVal unchanged.
*/
static void fts3GetDeltaVarint2(char **pp, char *pEnd, sqlite3_int64 *pVal){
  if( *pp>=pEnd ){
    *pp = 0;
  }else{
    fts3GetDeltaVarint(pp, pVal);
  }
}
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
  *ppCsr = pCsr = (sqlite3_vtab_cursor *)sqlite3_malloc(sizeof(Fts3Cursor));
  if( !pCsr ){
    return SQLITE_NOMEM;
  }
  memset(pCsr, 0, sizeof(Fts3Cursor));
  return SQLITE_OK;
}

/****************************************************************/
/****************************************************************/
/****************************************************************/
/****************************************************************/


/*
** Close the cursor.  For additional information see the documentation
** on the xClose method of the virtual table interface.
*/
static int fulltextClose(sqlite3_vtab_cursor *pCursor){
  Fts3Cursor *pCsr = (Fts3Cursor *)pCursor;







<
<
<
<
<
<







808
809
810
811
812
813
814






815
816
817
818
819
820
821
  *ppCsr = pCsr = (sqlite3_vtab_cursor *)sqlite3_malloc(sizeof(Fts3Cursor));
  if( !pCsr ){
    return SQLITE_NOMEM;
  }
  memset(pCsr, 0, sizeof(Fts3Cursor));
  return SQLITE_OK;
}







/*
** Close the cursor.  For additional information see the documentation
** on the xClose method of the virtual table interface.
*/
static int fulltextClose(sqlite3_vtab_cursor *pCursor){
  Fts3Cursor *pCsr = (Fts3Cursor *)pCursor;
1704
1705
1706
1707
1708
1709
1710




































































1711
1712
1713
1714
1715
1716
1717
    *paOut = pOut;
    *pnOut = nOut;
  }else{
    sqlite3_free(pOut);
  }
  return rc;
}





































































/*
** Evaluate the full-text expression pExpr against fts3 table pTab. Store
** the resulting doclist in *paOut and *pnOut.
*/
static int evalFts3Expr(
  Fts3Table *p,                   /* Virtual table handle */







>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>







1729
1730
1731
1732
1733
1734
1735
1736
1737
1738
1739
1740
1741
1742
1743
1744
1745
1746
1747
1748
1749
1750
1751
1752
1753
1754
1755
1756
1757
1758
1759
1760
1761
1762
1763
1764
1765
1766
1767
1768
1769
1770
1771
1772
1773
1774
1775
1776
1777
1778
1779
1780
1781
1782
1783
1784
1785
1786
1787
1788
1789
1790
1791
1792
1793
1794
1795
1796
1797
1798
1799
1800
1801
1802
1803
1804
1805
1806
1807
1808
1809
1810
    *paOut = pOut;
    *pnOut = nOut;
  }else{
    sqlite3_free(pOut);
  }
  return rc;
}

static int fts3NearMerge(
  int mergetype,                  /* MERGE_POS_NEAR or MERGE_NEAR */
  int nNear,                      /* Parameter to NEAR operator */
  int nTokenLeft,                 /* Number of tokens in LHS phrase arg */
  char *aLeft,                    /* Doclist for LHS (incl. positions) */
  int nLeft,                      /* Size of LHS doclist in bytes */
  int nTokenRight,                /* As nTokenLeft */
  char *aRight,                   /* As aLeft */
  int nRight,                     /* As nRight */
  char **paOut,                   /* OUT: Results of merge (malloced) */
  int *pnOut                      /* OUT: Sized of output buffer */
){
  char *aOut;
  int rc;

  assert( mergetype==MERGE_POS_NEAR || MERGE_NEAR );

  aOut = sqlite3_malloc(nLeft+nRight+1);
  if( aOut==0 ){
    rc = SQLITE_NOMEM;
  }else{
    rc = fts3DoclistMerge(mergetype, nNear+nTokenRight, nNear+nTokenLeft, 
      aOut, pnOut, aLeft, nLeft, aRight, nRight
    );
    if( rc!=SQLITE_OK ){
      sqlite3_free(aOut);
      aOut = 0;
    }
  }

  *paOut = aOut;
  return rc;
}

int sqlite3Fts3ExprNearTrim(Fts3Expr *pLeft, Fts3Expr *pRight, int nNear){
  int rc;
  if( pLeft->aDoclist==0 || pRight->aDoclist==0 ){
    sqlite3_free(pLeft->aDoclist);
    sqlite3_free(pRight->aDoclist);
    pRight->aDoclist = 0;
    pLeft->aDoclist = 0;
    rc = SQLITE_OK;
  }else{
    char *aOut;
    int nOut;

    rc = fts3NearMerge(MERGE_POS_NEAR, nNear, 
        pLeft->pPhrase->nToken, pLeft->aDoclist, pLeft->nDoclist,
        pRight->pPhrase->nToken, pRight->aDoclist, pRight->nDoclist,
        &aOut, &nOut
    );
    if( rc!=SQLITE_OK ) return rc;
    sqlite3_free(pRight->aDoclist);
    pRight->aDoclist = aOut;
    pRight->nDoclist = nOut;

    rc = fts3NearMerge(MERGE_POS_NEAR, nNear, 
        pRight->pPhrase->nToken, pRight->aDoclist, pRight->nDoclist,
        pLeft->pPhrase->nToken, pLeft->aDoclist, pLeft->nDoclist,
        &aOut, &nOut
    );
    sqlite3_free(pLeft->aDoclist);
    pLeft->aDoclist = aOut;
    pLeft->nDoclist = nOut;
  }
  return rc;
}

/*
** Evaluate the full-text expression pExpr against fts3 table pTab. Store
** the resulting doclist in *paOut and *pnOut.
*/
static int evalFts3Expr(
  Fts3Table *p,                   /* Virtual table handle */
1749
1750
1751
1752
1753
1754
1755
1756
1757
1758
1759
1760
1761
1762
1763
1764
1765
1766
1767
1768
1769
1770
1771
1772
1773
1774
1775

1776
1777
1778
1779
1780
1781
1782
1783
1784
1785
1786
1787
1788
            || pExpr->eType==FTSQUERY_AND  || pExpr->eType==FTSQUERY_NOT
        );
        switch( pExpr->eType ){
          case FTSQUERY_NEAR: {
            Fts3Expr *pLeft;
            Fts3Expr *pRight;
            int mergetype = isReqPos ? MERGE_POS_NEAR : MERGE_NEAR;
            int nParam1;
            int nParam2;
            char *aBuffer;
           
            if( pExpr->pParent && pExpr->pParent->eType==FTSQUERY_NEAR ){
              mergetype = MERGE_POS_NEAR;
            }
            pLeft = pExpr->pLeft;
            while( pLeft->eType==FTSQUERY_NEAR ){ 
              pLeft=pLeft->pRight;
            }
            pRight = pExpr->pRight;
            assert( pRight->eType==FTSQUERY_PHRASE );
            assert( pLeft->eType==FTSQUERY_PHRASE );

            nParam1 = pExpr->nNear+1;
            nParam2 = nParam1+pLeft->pPhrase->nToken+pRight->pPhrase->nToken-2;
            aBuffer = sqlite3_malloc(nLeft+nRight+1);
            rc = fts3DoclistMerge(mergetype, nParam1, nParam2, aBuffer,
                pnOut, aLeft, nLeft, aRight, nRight

            );
            if( rc!=SQLITE_OK ){
              sqlite3_free(aBuffer);
            }else{
              *paOut = aBuffer;
            }
            sqlite3_free(aLeft);
            break;
          }

          case FTSQUERY_OR: {
            /* Allocate a buffer for the output. The maximum size is the
            ** sum of the sizes of the two input buffers. The +1 term is







<
<
<












|
|
<
<
|
>

<
<
<
<
<







1842
1843
1844
1845
1846
1847
1848



1849
1850
1851
1852
1853
1854
1855
1856
1857
1858
1859
1860
1861
1862


1863
1864
1865





1866
1867
1868
1869
1870
1871
1872
            || pExpr->eType==FTSQUERY_AND  || pExpr->eType==FTSQUERY_NOT
        );
        switch( pExpr->eType ){
          case FTSQUERY_NEAR: {
            Fts3Expr *pLeft;
            Fts3Expr *pRight;
            int mergetype = isReqPos ? MERGE_POS_NEAR : MERGE_NEAR;



           
            if( pExpr->pParent && pExpr->pParent->eType==FTSQUERY_NEAR ){
              mergetype = MERGE_POS_NEAR;
            }
            pLeft = pExpr->pLeft;
            while( pLeft->eType==FTSQUERY_NEAR ){ 
              pLeft=pLeft->pRight;
            }
            pRight = pExpr->pRight;
            assert( pRight->eType==FTSQUERY_PHRASE );
            assert( pLeft->eType==FTSQUERY_PHRASE );

            rc = fts3NearMerge(mergetype, pExpr->nNear, 
                pLeft->pPhrase->nToken, aLeft, nLeft,


                pRight->pPhrase->nToken, aRight, nRight,
                paOut, pnOut
            );





            sqlite3_free(aLeft);
            break;
          }

          case FTSQUERY_OR: {
            /* Allocate a buffer for the output. The maximum size is the
            ** sum of the sizes of the two input buffers. The +1 term is
2025
2026
2027
2028
2029
2030
2031
2032
2033
2034
2035
2036
2037
2038
2039
*/
int sqlite3Fts3ExprLoadDoclist(Fts3Table *pTab, Fts3Expr *pExpr){
  return evalFts3Expr(pTab, pExpr, &pExpr->aDoclist, &pExpr->nDoclist, 1);
}

/*
** After ExprLoadDoclist() (see above) has been called, this function is
** used to iterate through the position lists that make up the doclist
** stored in pExpr->aDoclist.
*/
char *sqlite3Fts3FindPositions(
  Fts3Expr *pExpr,                /* Access this expressions doclist */
  sqlite3_int64 iDocid,           /* Docid associated with requested pos-list */
  int iCol                        /* Column of requested pos-list */
){







|







2109
2110
2111
2112
2113
2114
2115
2116
2117
2118
2119
2120
2121
2122
2123
*/
int sqlite3Fts3ExprLoadDoclist(Fts3Table *pTab, Fts3Expr *pExpr){
  return evalFts3Expr(pTab, pExpr, &pExpr->aDoclist, &pExpr->nDoclist, 1);
}

/*
** After ExprLoadDoclist() (see above) has been called, this function is
** used to iterate/search through the position lists that make up the doclist
** stored in pExpr->aDoclist.
*/
char *sqlite3Fts3FindPositions(
  Fts3Expr *pExpr,                /* Access this expressions doclist */
  sqlite3_int64 iDocid,           /* Docid associated with requested pos-list */
  int iCol                        /* Column of requested pos-list */
){
2060
2061
2062
2063
2064
2065
2066
2067
2068
2069
2070
2071
2072
2073
2074
          }
          while( iThis<iCol ){
            fts3ColumnlistCopy(0, &pCsr);
            if( *pCsr==0x00 ) return 0;
            pCsr++;
            pCsr += sqlite3Fts3GetVarint32(pCsr, &iThis);
          }
          if( iCol==iThis ) return pCsr;
        }
        return 0;
      }
    }
  }

  return 0;







|







2144
2145
2146
2147
2148
2149
2150
2151
2152
2153
2154
2155
2156
2157
2158
          }
          while( iThis<iCol ){
            fts3ColumnlistCopy(0, &pCsr);
            if( *pCsr==0x00 ) return 0;
            pCsr++;
            pCsr += sqlite3Fts3GetVarint32(pCsr, &iThis);
          }
          if( iCol==iThis && (*pCsr&0xFE) ) return pCsr;
        }
        return 0;
      }
    }
  }

  return 0;
2112
2113
2114
2115
2116
2117
2118
2119
2120
2121
2122
2123
2124
2125
2126
2127
2128
2129
2130
2131
2132
2133
2134
2135
2136
2137
2138
2139
2140
2141
2142
2143
2144
2145
2146
2147
2148
2149
2150
2151
2152
2153
2154
2155
2156
2157
2158
2159
2160
2161
2162
2163
2164
  int nVal,                       /* Size of apVal[] array */
  sqlite3_value **apVal           /* Array of arguments */
){
  Fts3Cursor *pCsr;               /* Cursor handle passed through apVal[0] */
  const char *zStart = "<b>";
  const char *zEnd = "</b>";
  const char *zEllipsis = "<b>...</b>";

  /* There must be at least one argument passed to this function (otherwise
  ** the non-overloaded version would have been called instead of this one).
  */
  assert( nVal>=1 );

  if( nVal>4 ){
    sqlite3_result_error(pContext, 
        "wrong number of arguments to function snippet()", -1);
    return;
  }
  if( fts3FunctionArg(pContext, "snippet", apVal[0], &pCsr) ) return;

  switch( nVal ){
    case 4: zEllipsis = (const char*)sqlite3_value_text(apVal[3]);
    case 3: zEnd = (const char*)sqlite3_value_text(apVal[2]);
    case 2: zStart = (const char*)sqlite3_value_text(apVal[1]);
  }
  if( !zEllipsis || !zEnd || !zStart ){
    sqlite3_result_error_nomem(pContext);
  }else if( SQLITE_OK==fts3CursorSeek(pContext, pCsr) ){
    sqlite3Fts3Snippet(pContext, pCsr, zStart, zEnd, zEllipsis);
  }
}

/*
** Implementation of the snippet2() function for FTS3
*/
static void fts3Snippet2Func(
  sqlite3_context *pContext,      /* SQLite function call context */
  int nVal,                       /* Size of apVal[] array */
  sqlite3_value **apVal           /* Array of arguments */
){
  Fts3Cursor *pCsr;               /* Cursor handle passed through apVal[0] */
  const char *zStart = "<b>";
  const char *zEnd = "</b>";
  const char *zEllipsis = "<b>...</b>";
  int iCol = -1;
  int nToken = 10;

  /* There must be at least one argument passed to this function (otherwise
  ** the non-overloaded version would have been called instead of this one).
  */
  assert( nVal>=1 );

  if( nVal>6 ){







<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<

|







2196
2197
2198
2199
2200
2201
2202





































2203
2204
2205
2206
2207
2208
2209
2210
2211
  int nVal,                       /* Size of apVal[] array */
  sqlite3_value **apVal           /* Array of arguments */
){
  Fts3Cursor *pCsr;               /* Cursor handle passed through apVal[0] */
  const char *zStart = "<b>";
  const char *zEnd = "</b>";
  const char *zEllipsis = "<b>...</b>";





































  int iCol = -1;
  int nToken = 15;                /* Default number of tokens in snippet */

  /* There must be at least one argument passed to this function (otherwise
  ** the non-overloaded version would have been called instead of this one).
  */
  assert( nVal>=1 );

  if( nVal>6 ){
2174
2175
2176
2177
2178
2179
2180
2181
2182
2183
2184
2185
2186
2187
2188
    case 4: zEllipsis = (const char*)sqlite3_value_text(apVal[3]);
    case 3: zEnd = (const char*)sqlite3_value_text(apVal[2]);
    case 2: zStart = (const char*)sqlite3_value_text(apVal[1]);
  }
  if( !zEllipsis || !zEnd || !zStart ){
    sqlite3_result_error_nomem(pContext);
  }else if( SQLITE_OK==fts3CursorSeek(pContext, pCsr) ){
    sqlite3Fts3Snippet2(pContext, pCsr, zStart, zEnd, zEllipsis, iCol, nToken);
  }
}

/*
** Implementation of the offsets() function for FTS3
*/
static void fts3OffsetsFunc(







|







2221
2222
2223
2224
2225
2226
2227
2228
2229
2230
2231
2232
2233
2234
2235
    case 4: zEllipsis = (const char*)sqlite3_value_text(apVal[3]);
    case 3: zEnd = (const char*)sqlite3_value_text(apVal[2]);
    case 2: zStart = (const char*)sqlite3_value_text(apVal[1]);
  }
  if( !zEllipsis || !zEnd || !zStart ){
    sqlite3_result_error_nomem(pContext);
  }else if( SQLITE_OK==fts3CursorSeek(pContext, pCsr) ){
    sqlite3Fts3Snippet(pContext, pCsr, zStart, zEnd, zEllipsis, iCol, nToken);
  }
}

/*
** Implementation of the offsets() function for FTS3
*/
static void fts3OffsetsFunc(
2275
2276
2277
2278
2279
2280
2281
2282
2283
2284
2285
2286
2287
2288
2289
  void **ppArg                    /* Unused */
){
  struct Overloaded {
    const char *zName;
    void (*xFunc)(sqlite3_context*,int,sqlite3_value**);
  } aOverload[] = {
    { "snippet", fts3SnippetFunc },
    { "snippet2", fts3Snippet2Func },
    { "offsets", fts3OffsetsFunc },
    { "optimize", fts3OptimizeFunc },
    { "matchinfo", fts3MatchinfoFunc },
  };
  int i;                          /* Iterator variable */

  UNUSED_PARAMETER(pVtab);







<







2322
2323
2324
2325
2326
2327
2328

2329
2330
2331
2332
2333
2334
2335
  void **ppArg                    /* Unused */
){
  struct Overloaded {
    const char *zName;
    void (*xFunc)(sqlite3_context*,int,sqlite3_value**);
  } aOverload[] = {
    { "snippet", fts3SnippetFunc },

    { "offsets", fts3OffsetsFunc },
    { "optimize", fts3OptimizeFunc },
    { "matchinfo", fts3MatchinfoFunc },
  };
  int i;                          /* Iterator variable */

  UNUSED_PARAMETER(pVtab);
2425
2426
2427
2428
2429
2430
2431
2432
2433
2434
2435
2436
2437
2438
2439
  /* Create the virtual table wrapper around the hash-table and overload 
  ** the two scalar functions. If this is successful, register the
  ** module with sqlite.
  */
  if( SQLITE_OK==rc 
   && SQLITE_OK==(rc = sqlite3Fts3InitHashTable(db, pHash, "fts3_tokenizer"))
   && SQLITE_OK==(rc = sqlite3_overload_function(db, "snippet", -1))
   && SQLITE_OK==(rc = sqlite3_overload_function(db, "snippet2", -1))
   && SQLITE_OK==(rc = sqlite3_overload_function(db, "offsets", 1))
   && SQLITE_OK==(rc = sqlite3_overload_function(db, "matchinfo", -1))
   && SQLITE_OK==(rc = sqlite3_overload_function(db, "optimize", 1))
  ){
    return sqlite3_create_module_v2(
        db, "fts3", &fts3Module, (void *)pHash, hashDestroy
    );







<







2471
2472
2473
2474
2475
2476
2477

2478
2479
2480
2481
2482
2483
2484
  /* Create the virtual table wrapper around the hash-table and overload 
  ** the two scalar functions. If this is successful, register the
  ** module with sqlite.
  */
  if( SQLITE_OK==rc 
   && SQLITE_OK==(rc = sqlite3Fts3InitHashTable(db, pHash, "fts3_tokenizer"))
   && SQLITE_OK==(rc = sqlite3_overload_function(db, "snippet", -1))

   && SQLITE_OK==(rc = sqlite3_overload_function(db, "offsets", 1))
   && SQLITE_OK==(rc = sqlite3_overload_function(db, "matchinfo", -1))
   && SQLITE_OK==(rc = sqlite3_overload_function(db, "optimize", 1))
  ){
    return sqlite3_create_module_v2(
        db, "fts3", &fts3Module, (void *)pHash, hashDestroy
    );
Changes to ext/fts3/fts3Int.h.
275
276
277
278
279
280
281

282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
int sqlite3Fts3GetVarint(const char *, sqlite_int64 *);
int sqlite3Fts3GetVarint32(const char *, int *);
int sqlite3Fts3VarintLen(sqlite3_uint64);
void sqlite3Fts3Dequote(char *);

char *sqlite3Fts3FindPositions(Fts3Expr *, sqlite3_int64, int);
int sqlite3Fts3ExprLoadDoclist(Fts3Table *, Fts3Expr *);


/* fts3_tokenizer.c */
const char *sqlite3Fts3NextToken(const char *, int *);
int sqlite3Fts3InitHashTable(sqlite3 *, Fts3Hash *, const char *);
int sqlite3Fts3InitTokenizer(Fts3Hash *pHash, 
  const char *, sqlite3_tokenizer **, const char **, char **
);

/* fts3_snippet.c */
void sqlite3Fts3Offsets(sqlite3_context*, Fts3Cursor*);
void sqlite3Fts3Snippet(sqlite3_context*, Fts3Cursor*, 
  const char *, const char *, const char *
);
void sqlite3Fts3Snippet2(sqlite3_context *, Fts3Cursor *, const char *,
  const char *, const char *, int, int
);
void sqlite3Fts3Matchinfo(sqlite3_context *, Fts3Cursor *);

/* fts3_expr.c */
int sqlite3Fts3ExprParse(sqlite3_tokenizer *, 
  char **, int, int, const char *, int, Fts3Expr **







>










|
<
<
<







275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293



294
295
296
297
298
299
300
int sqlite3Fts3GetVarint(const char *, sqlite_int64 *);
int sqlite3Fts3GetVarint32(const char *, int *);
int sqlite3Fts3VarintLen(sqlite3_uint64);
void sqlite3Fts3Dequote(char *);

char *sqlite3Fts3FindPositions(Fts3Expr *, sqlite3_int64, int);
int sqlite3Fts3ExprLoadDoclist(Fts3Table *, Fts3Expr *);
int sqlite3Fts3ExprNearTrim(Fts3Expr *, Fts3Expr *, int);

/* fts3_tokenizer.c */
const char *sqlite3Fts3NextToken(const char *, int *);
int sqlite3Fts3InitHashTable(sqlite3 *, Fts3Hash *, const char *);
int sqlite3Fts3InitTokenizer(Fts3Hash *pHash, 
  const char *, sqlite3_tokenizer **, const char **, char **
);

/* fts3_snippet.c */
void sqlite3Fts3Offsets(sqlite3_context*, Fts3Cursor*);
void sqlite3Fts3Snippet(sqlite3_context *, Fts3Cursor *, const char *,



  const char *, const char *, int, int
);
void sqlite3Fts3Matchinfo(sqlite3_context *, Fts3Cursor *);

/* fts3_expr.c */
int sqlite3Fts3ExprParse(sqlite3_tokenizer *, 
  char **, int, int, const char *, int, Fts3Expr **
Changes to ext/fts3/fts3_snippet.c.
14
15
16
17
18
19
20
















21


22

























23









24
25





26

27
28

29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746

























747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762



















763

764
765
766

767
768





769
770



771
772
773

774


775

776
777
778


779
780
781
782
783
784
785
786
787

788


789
790
791
792














793

794
795
796
797
798
799










800




801
802
803
804



805

806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850




851





852






853






854











855
856





























857

858



859
860
861








862
863
864


865





866





867





868
869
870
871
872
873
874
875

876
877

878
879
880




881

882
883
884
885

886

887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928

929
930

931
932
933
934
935
936

937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958

959
960
961
962
963
964
965
966
967
968
969
970
971
972
973


974
975
976
977
978
979

980
981
982
983

984
985


986
987
988
989
990
991
992
993
994





995
996
997
998
999
1000
1001
1002
1003
1004
1005
1006
1007
1008
1009
1010
1011


1012
1013
1014
1015
1016
1017
1018
1019
1020
1021
1022
1023

1024
1025

1026
1027
1028
1029
1030
1031
1032
1033
1034
1035
1036
1037
1038




1039
1040
1041
1042
1043
1044
1045
1046
1047
1048

1049
1050
1051
1052
1053
1054
1055




















1056
1057

1058
1059
































































1060
1061
1062
1063
1064
1065
1066

1067
1068
1069


1070
1071
1072
1073



1074
1075
1076

1077




1078
1079

1080


1081
1082
1083
1084
1085
1086
1087
1088

1089
1090
1091
1092
1093
1094
1095
1096
1097
1098


1099
1100
1101
1102
1103
1104
1105
1106
1107
1108
1109
1110
1111
1112
1113
1114
1115
1116
1117
1118
1119
1120




1121




1122




1123




1124
1125



1126


1127
1128

1129
1130


1131
1132


1133

1134
1135
1136
1137


1138
1139
1140
1141
1142
1143
1144
1145
1146
1147
1148
1149
1150
1151

1152
1153
1154
1155
1156
1157
1158
1159
1160
1161
1162
1163
1164
1165
1166
1167
1168
1169
1170
1171
1172
1173
1174
1175
1176
1177
1178
1179
1180
1181
1182
1183
1184
1185
1186
1187
1188
1189
1190
1191
1192
1193
1194

1195
1196
1197
1198
1199
1200





1201
1202
1203
1204
1205
1206
1207
1208

1209
1210
1211

1212
1213
1214
1215
1216
1217
1218
1219
1220
1221
1222
1223
1224
1225
1226
1227
1228
1229
1230
1231
1232





1233
1234

1235
1236
1237
1238
1239
1240
1241

1242
1243

1244
1245

1246

1247
1248
1249
1250
1251
1252
1253
1254
1255
1256
1257
1258




1259





1260
1261
1262
1263
1264
1265
1266
1267
1268
1269
1270
1271
1272
1273
1274
1275
1276
1277
1278
1279
1280


1281
1282
1283
1284
1285
1286
1287
1288
1289
1290
1291
1292
1293
1294
1295
1296
1297
1298
1299
1300
1301



1302
1303
1304
1305
1306
1307
1308
1309
1310

1311
1312

1313











1314

























1315




























































































































1316

1317


1318




1319







1320







1321




1322



























1323




1324






1325


1326




1327
1328

1329
1330
1331

1332
1333



1334
1335
1336
1337
1338
1339
1340
1341
1342
1343
1344
#if !defined(SQLITE_CORE) || defined(SQLITE_ENABLE_FTS3)

#include "fts3Int.h"
#include <string.h>
#include <assert.h>
#include <ctype.h>

















typedef struct Snippet Snippet;




























/*









** An instance of the following structure keeps track of generated
** matching-word offset information and snippets.





*/

struct Snippet {
  int nMatch;                     /* Total number of matches */

  int nAlloc;                     /* Space allocated for aMatch[] */
  struct snippetMatch {  /* One entry for each matching term */
    char snStatus;       /* Status flag for use while constructing snippets */
    short int nByte;     /* Number of bytes in the term */
    short int iCol;      /* The column that contains the match */
    short int iTerm;     /* The index in Query.pTerms[] of the matching term */
    int iToken;          /* The index of the matching document token */
    int iStart;          /* The offset to the first character of the term */
  } *aMatch;                      /* Points to space obtained from malloc */
  char *zOffset;                  /* Text rendering of aMatch[] */
  int nOffset;                    /* strlen(zOffset) */
  char *zSnippet;                 /* Snippet text */
  int nSnippet;                   /* strlen(zSnippet) */
};


/* It is not safe to call isspace(), tolower(), or isalnum() on
** hi-bit-set characters.  This is the same solution used in the
** tokenizer.
*/
static int fts3snippetIsspace(char c){
  return (c&0x80)==0 ? isspace(c) : 0;
}


/*
** A StringBuffer object holds a zero-terminated string that grows
** arbitrarily by appending.  Space to hold the string is obtained
** from sqlite3_malloc().  After any memory allocation failure, 
** StringBuffer.z is set to NULL and no further allocation is attempted.
*/
typedef struct StringBuffer {
  char *z;         /* Text of the string.  Space from malloc. */
  int nUsed;       /* Number bytes of z[] used, not counting \000 terminator */
  int nAlloc;      /* Bytes allocated for z[] */
} StringBuffer;


/*
** Initialize a new StringBuffer.
*/
static void fts3SnippetSbInit(StringBuffer *p){
  p->nAlloc = 100;
  p->nUsed = 0;
  p->z = sqlite3_malloc( p->nAlloc );
}

/*
** Append text to the string buffer.
*/
static void fts3SnippetAppend(StringBuffer *p, const char *zNew, int nNew){
  if( p->z==0 ) return;
  if( nNew<0 ) nNew = (int)strlen(zNew);
  if( p->nUsed + nNew >= p->nAlloc ){
    int nAlloc;
    char *zNew;

    nAlloc = p->nUsed + nNew + p->nAlloc;
    zNew = sqlite3_realloc(p->z, nAlloc);
    if( zNew==0 ){
      sqlite3_free(p->z);
      p->z = 0;
      return;
    }
    p->z = zNew;
    p->nAlloc = nAlloc;
  }
  memcpy(&p->z[p->nUsed], zNew, nNew);
  p->nUsed += nNew;
  p->z[p->nUsed] = 0;
}

/* If the StringBuffer ends in something other than white space, add a
** single space character to the end.
*/
static void fts3SnippetAppendWhiteSpace(StringBuffer *p){
  if( p->z && p->nUsed && !fts3snippetIsspace(p->z[p->nUsed-1]) ){
    fts3SnippetAppend(p, " ", 1);
  }
}

/* Remove white space from the end of the StringBuffer */
static void fts3SnippetTrimWhiteSpace(StringBuffer *p){
  if( p->z ){
    while( p->nUsed && fts3snippetIsspace(p->z[p->nUsed-1]) ){
      p->nUsed--;
    }
    p->z[p->nUsed] = 0;
  }
}

/* 
** Release all memory associated with the Snippet structure passed as
** an argument.
*/
static void fts3SnippetFree(Snippet *p){
  if( p ){
    sqlite3_free(p->aMatch);
    sqlite3_free(p->zOffset);
    sqlite3_free(p->zSnippet);
    sqlite3_free(p);
  }
}

/*
** Append a single entry to the p->aMatch[] log.
*/
static int snippetAppendMatch(
  Snippet *p,               /* Append the entry to this snippet */
  int iCol, int iTerm,      /* The column and query term */
  int iToken,               /* Matching token in document */
  int iStart, int nByte     /* Offset and size of the match */
){
  int i;
  struct snippetMatch *pMatch;
  if( p->nMatch+1>=p->nAlloc ){
    struct snippetMatch *pNew;
    p->nAlloc = p->nAlloc*2 + 10;
    pNew = sqlite3_realloc(p->aMatch, p->nAlloc*sizeof(p->aMatch[0]) );
    if( pNew==0 ){
      p->aMatch = 0;
      p->nMatch = 0;
      p->nAlloc = 0;
      return SQLITE_NOMEM;
    }
    p->aMatch = pNew;
  }
  i = p->nMatch++;
  pMatch = &p->aMatch[i];
  pMatch->iCol = (short)iCol;
  pMatch->iTerm = (short)iTerm;
  pMatch->iToken = iToken;
  pMatch->iStart = iStart;
  pMatch->nByte = (short)nByte;
  return SQLITE_OK;
}

/*
** Sizing information for the circular buffer used in snippetOffsetsOfColumn()
*/
#define FTS3_ROTOR_SZ   (32)
#define FTS3_ROTOR_MASK (FTS3_ROTOR_SZ-1)

/*
** Function to iterate through the tokens of a compiled expression.
**
** Except, skip all tokens on the right-hand side of a NOT operator.
** This function is used to find tokens as part of snippet and offset
** generation and we do nt want snippets and offsets to report matches
** for tokens on the RHS of a NOT.
*/
static int fts3NextExprToken(Fts3Expr **ppExpr, int *piToken){
  Fts3Expr *p = *ppExpr;
  int iToken = *piToken;
  if( iToken<0 ){
    /* In this case the expression p is the root of an expression tree.
    ** Move to the first token in the expression tree.
    */
    while( p->pLeft ){
      p = p->pLeft;
    }
    iToken = 0;
  }else{
    assert(p && p->eType==FTSQUERY_PHRASE );
    if( iToken<(p->pPhrase->nToken-1) ){
      iToken++;
    }else{
      iToken = 0;
      while( p->pParent && p->pParent->pLeft!=p ){
        assert( p->pParent->pRight==p );
        p = p->pParent;
      }
      p = p->pParent;
      if( p ){
        assert( p->pRight!=0 );
        p = p->pRight;
        while( p->pLeft ){
          p = p->pLeft;
        }
      }
    }
  }

  *ppExpr = p;
  *piToken = iToken;
  return p?1:0;
}

/*
** Return TRUE if the expression node pExpr is located beneath the
** RHS of a NOT operator.
*/
static int fts3ExprBeneathNot(Fts3Expr *p){
  Fts3Expr *pParent;
  while( p ){
    pParent = p->pParent;
    if( pParent && pParent->eType==FTSQUERY_NOT && pParent->pRight==p ){
      return 1;
    }
    p = pParent;
  }
  return 0;
}

/*
** Add entries to pSnippet->aMatch[] for every match that occurs against
** document zDoc[0..nDoc-1] which is stored in column iColumn.
*/
static int snippetOffsetsOfColumn(
  Fts3Cursor *pCur,         /* The fulltest search cursor */
  Snippet *pSnippet,             /* The Snippet object to be filled in */
  int iColumn,                   /* Index of fulltext table column */
  const char *zDoc,              /* Text of the fulltext table column */
  int nDoc                       /* Length of zDoc in bytes */
){
  const sqlite3_tokenizer_module *pTModule;  /* The tokenizer module */
  sqlite3_tokenizer *pTokenizer;             /* The specific tokenizer */
  sqlite3_tokenizer_cursor *pTCursor;        /* Tokenizer cursor */
  Fts3Table *pVtab;                /* The full text index */
  int nColumn;                         /* Number of columns in the index */
  int i, j;                            /* Loop counters */
  int rc;                              /* Return code */
  unsigned int match, prevMatch;       /* Phrase search bitmasks */
  const char *zToken;                  /* Next token from the tokenizer */
  int nToken;                          /* Size of zToken */
  int iBegin, iEnd, iPos;              /* Offsets of beginning and end */

  /* The following variables keep a circular buffer of the last
  ** few tokens */
  unsigned int iRotor = 0;             /* Index of current token */
  int iRotorBegin[FTS3_ROTOR_SZ];      /* Beginning offset of token */
  int iRotorLen[FTS3_ROTOR_SZ];        /* Length of token */

  pVtab =  (Fts3Table *)pCur->base.pVtab;
  nColumn = pVtab->nColumn;
  pTokenizer = pVtab->pTokenizer;
  pTModule = pTokenizer->pModule;
  rc = pTModule->xOpen(pTokenizer, zDoc, nDoc, &pTCursor);
  if( rc ) return rc;
  pTCursor->pTokenizer = pTokenizer;

  prevMatch = 0;
  while( (rc = pTModule->xNext(pTCursor, &zToken, &nToken,
                               &iBegin, &iEnd, &iPos))==SQLITE_OK ){
    Fts3Expr *pIter = pCur->pExpr;
    int iIter = -1;
    iRotorBegin[iRotor&FTS3_ROTOR_MASK] = iBegin;
    iRotorLen[iRotor&FTS3_ROTOR_MASK] = iEnd-iBegin;
    match = 0;
    for(i=0; i<(FTS3_ROTOR_SZ-1) && fts3NextExprToken(&pIter, &iIter); i++){
      int nPhrase;                    /* Number of tokens in current phrase */
      struct PhraseToken *pToken;     /* Current token */
      int iCol;                       /* Column index */

      if( fts3ExprBeneathNot(pIter) ) continue;
      nPhrase = pIter->pPhrase->nToken;
      pToken = &pIter->pPhrase->aToken[iIter];
      iCol = pIter->pPhrase->iColumn;
      if( iCol>=0 && iCol<nColumn && iCol!=iColumn ) continue;
      if( pToken->n>nToken ) continue;
      if( !pToken->isPrefix && pToken->n<nToken ) continue;
      assert( pToken->n<=nToken );
      if( memcmp(pToken->z, zToken, pToken->n) ) continue;
      if( iIter>0 && (prevMatch & (1<<i))==0 ) continue;
      match |= 1<<i;
      if( i==(FTS3_ROTOR_SZ-2) || nPhrase==iIter+1 ){
        for(j=nPhrase-1; j>=0; j--){
          int k = (iRotor-j) & FTS3_ROTOR_MASK;
          rc = snippetAppendMatch(pSnippet, iColumn, i-j, iPos-j,
                                  iRotorBegin[k], iRotorLen[k]);
          if( rc ) goto end_offsets_of_column;
        }
      }
    }
    prevMatch = match<<1;
    iRotor++;
  }
end_offsets_of_column:
  pTModule->xClose(pTCursor);  
  return rc==SQLITE_DONE ? SQLITE_OK : rc;
}

/*
** Remove entries from the pSnippet structure to account for the NEAR
** operator. When this is called, pSnippet contains the list of token 
** offsets produced by treating all NEAR operators as AND operators.
** This function removes any entries that should not be present after
** accounting for the NEAR restriction. For example, if the queried
** document is:
**
**     "A B C D E A"
**
** and the query is:
** 
**     A NEAR/0 E
**
** then when this function is called the Snippet contains token offsets
** 0, 4 and 5. This function removes the "0" entry (because the first A
** is not near enough to an E).
**
** When this function is called, the value pointed to by parameter piLeft is
** the integer id of the left-most token in the expression tree headed by
** pExpr. This function increments *piLeft by the total number of tokens
** in the expression tree headed by pExpr.
**
** Return 1 if any trimming occurs.  Return 0 if no trimming is required.
*/
static int trimSnippetOffsets(
  Fts3Expr *pExpr,      /* The search expression */
  Snippet *pSnippet,    /* The set of snippet offsets to be trimmed */
  int *piLeft           /* Index of left-most token in pExpr */
){
  if( pExpr ){
    if( trimSnippetOffsets(pExpr->pLeft, pSnippet, piLeft) ){
      return 1;
    }

    switch( pExpr->eType ){
      case FTSQUERY_PHRASE:
        *piLeft += pExpr->pPhrase->nToken;
        break;
      case FTSQUERY_NEAR: {
        /* The right-hand-side of a NEAR operator is always a phrase. The
        ** left-hand-side is either a phrase or an expression tree that is 
        ** itself headed by a NEAR operator. The following initializations
        ** set local variable iLeft to the token number of the left-most
        ** token in the right-hand phrase, and iRight to the right most
        ** token in the same phrase. For example, if we had:
        **
        **     <col> MATCH '"abc def" NEAR/2 "ghi jkl"'
        **
        ** then iLeft will be set to 2 (token number of ghi) and nToken will
        ** be set to 4.
        */
        Fts3Expr *pLeft = pExpr->pLeft;
        Fts3Expr *pRight = pExpr->pRight;
        int iLeft = *piLeft;
        int nNear = pExpr->nNear;
        int nToken = pRight->pPhrase->nToken;
        int jj, ii;
        if( pLeft->eType==FTSQUERY_NEAR ){
          pLeft = pLeft->pRight;
        }
        assert( pRight->eType==FTSQUERY_PHRASE );
        assert( pLeft->eType==FTSQUERY_PHRASE );
        nToken += pLeft->pPhrase->nToken;

        for(ii=0; ii<pSnippet->nMatch; ii++){
          struct snippetMatch *p = &pSnippet->aMatch[ii];
          if( p->iTerm==iLeft ){
            int isOk = 0;
            /* Snippet ii is an occurence of query term iLeft in the document.
            ** It occurs at position (p->iToken) of the document. We now
            ** search for an instance of token (iLeft-1) somewhere in the 
            ** range (p->iToken - nNear)...(p->iToken + nNear + nToken) within 
            ** the set of snippetMatch structures. If one is found, proceed. 
            ** If one cannot be found, then remove snippets ii..(ii+N-1) 
            ** from the matching snippets, where N is the number of tokens 
            ** in phrase pRight->pPhrase.
            */
            for(jj=0; isOk==0 && jj<pSnippet->nMatch; jj++){
              struct snippetMatch *p2 = &pSnippet->aMatch[jj];
              if( p2->iTerm==(iLeft-1) ){
                if( p2->iToken>=(p->iToken-nNear-1) 
                 && p2->iToken<(p->iToken+nNear+nToken) 
                ){
                  isOk = 1;
                }
              }
            }
            if( !isOk ){
              int kk;
              for(kk=0; kk<pRight->pPhrase->nToken; kk++){
                pSnippet->aMatch[kk+ii].iTerm = -2;
              }
              return 1;
            }
          }
          if( p->iTerm==(iLeft-1) ){
            int isOk = 0;
            for(jj=0; isOk==0 && jj<pSnippet->nMatch; jj++){
              struct snippetMatch *p2 = &pSnippet->aMatch[jj];
              if( p2->iTerm==iLeft ){
                if( p2->iToken<=(p->iToken+nNear+1) 
                 && p2->iToken>(p->iToken-nNear-nToken) 
                ){
                  isOk = 1;
                }
              }
            }
            if( !isOk ){
              int kk;
              for(kk=0; kk<pLeft->pPhrase->nToken; kk++){
                pSnippet->aMatch[ii-kk].iTerm = -2;
              }
              return 1;
            }
          }
        }
        break;
      }
    }

    if( trimSnippetOffsets(pExpr->pRight, pSnippet, piLeft) ){
      return 1;
    }
  }
  return 0;
}

/*
** Compute all offsets for the current row of the query.  
** If the offsets have already been computed, this routine is a no-op.
*/
static int snippetAllOffsets(Fts3Cursor *pCsr, Snippet **ppSnippet){
  Fts3Table *p = (Fts3Table *)pCsr->base.pVtab;  /* The FTS3 virtual table */
  int nColumn;           /* Number of columns.  Docid does count */
  int iColumn;           /* Index of of a column */
  int i;                 /* Loop index */
  int iFirst;            /* First column to search */
  int iLast;             /* Last coumn to search */
  int iTerm = 0;
  Snippet *pSnippet;
  int rc = SQLITE_OK;

  if( pCsr->pExpr==0 ){
    return SQLITE_OK;
  }

  pSnippet = (Snippet *)sqlite3_malloc(sizeof(Snippet));
  *ppSnippet = pSnippet;
  if( !pSnippet ){
    return SQLITE_NOMEM;
  }
  memset(pSnippet, 0, sizeof(Snippet));

  nColumn = p->nColumn;
  iColumn = (pCsr->eSearch - 2);
  if( iColumn<0 || iColumn>=nColumn ){
    /* Look for matches over all columns of the full-text index */
    iFirst = 0;
    iLast = nColumn-1;
  }else{
    /* Look for matches in the iColumn-th column of the index only */
    iFirst = iColumn;
    iLast = iColumn;
  }
  for(i=iFirst; rc==SQLITE_OK && i<=iLast; i++){
    const char *zDoc;
    int nDoc;
    zDoc = (const char*)sqlite3_column_text(pCsr->pStmt, i+1);
    nDoc = sqlite3_column_bytes(pCsr->pStmt, i+1);
    if( zDoc==0 && sqlite3_column_type(pCsr->pStmt, i+1)!=SQLITE_NULL ){
      rc = SQLITE_NOMEM;
    }else{
      rc = snippetOffsetsOfColumn(pCsr, pSnippet, i, zDoc, nDoc);
    }
  }

  while( trimSnippetOffsets(pCsr->pExpr, pSnippet, &iTerm) ){
    iTerm = 0;
  }

  return rc;
}

/*
** Convert the information in the aMatch[] array of the snippet
** into the string zOffset[0..nOffset-1]. This string is used as
** the return of the SQL offsets() function.
*/
static void snippetOffsetText(Snippet *p){
  int i;
  int cnt = 0;
  StringBuffer sb;
  char zBuf[200];
  if( p->zOffset ) return;
  fts3SnippetSbInit(&sb);
  for(i=0; i<p->nMatch; i++){
    struct snippetMatch *pMatch = &p->aMatch[i];
    if( pMatch->iTerm>=0 ){
      /* If snippetMatch.iTerm is less than 0, then the match was 
      ** discarded as part of processing the NEAR operator (see the 
      ** trimSnippetOffsetsForNear() function for details). Ignore 
      ** it in this case
      */
      zBuf[0] = ' ';
      sqlite3_snprintf(sizeof(zBuf)-1, &zBuf[cnt>0], "%d %d %d %d",
          pMatch->iCol, pMatch->iTerm, pMatch->iStart, pMatch->nByte);
      fts3SnippetAppend(&sb, zBuf, -1);
      cnt++;
    }
  }
  p->zOffset = sb.z;
  p->nOffset = sb.z ? sb.nUsed : 0;
}

/*
** zDoc[0..nDoc-1] is phrase of text.  aMatch[0..nMatch-1] are a set
** of matching words some of which might be in zDoc.  zDoc is column
** number iCol.
**
** iBreak is suggested spot in zDoc where we could begin or end an
** excerpt.  Return a value similar to iBreak but possibly adjusted
** to be a little left or right so that the break point is better.
*/
static int wordBoundary(
  int iBreak,                   /* The suggested break point */
  const char *zDoc,             /* Document text */
  int nDoc,                     /* Number of bytes in zDoc[] */
  struct snippetMatch *aMatch,  /* Matching words */
  int nMatch,                   /* Number of entries in aMatch[] */
  int iCol                      /* The column number for zDoc[] */
){
  int i;
  if( iBreak<=10 ){
    return 0;
  }
  if( iBreak>=nDoc-10 ){
    return nDoc;
  }
  for(i=0; ALWAYS(i<nMatch) && aMatch[i].iCol<iCol; i++){}
  while( i<nMatch && aMatch[i].iStart+aMatch[i].nByte<iBreak ){ i++; }
  if( i<nMatch ){
    if( aMatch[i].iStart<iBreak+10 ){
      return aMatch[i].iStart;
    }
    if( i>0 && aMatch[i-1].iStart+aMatch[i-1].nByte>=iBreak ){
      return aMatch[i-1].iStart;
    }
  }
  for(i=1; i<=10; i++){
    if( fts3snippetIsspace(zDoc[iBreak-i]) ){
      return iBreak - i + 1;
    }
    if( fts3snippetIsspace(zDoc[iBreak+i]) ){
      return iBreak + i + 1;
    }
  }
  return iBreak;
}



/*
** Allowed values for Snippet.aMatch[].snStatus
*/
#define SNIPPET_IGNORE  0   /* It is ok to omit this match from the snippet */
#define SNIPPET_DESIRED 1   /* We want to include this match in the snippet */

/*
** Generate the text of a snippet.
*/
static void snippetText(
  Fts3Cursor *pCursor,   /* The cursor we need the snippet for */
  Snippet *pSnippet,
  const char *zStartMark,     /* Markup to appear before each match */
  const char *zEndMark,       /* Markup to appear after each match */
  const char *zEllipsis       /* Ellipsis mark */
){
  int i, j;
  struct snippetMatch *aMatch;
  int nMatch;
  int nDesired;
  StringBuffer sb;
  int tailCol;
  int tailOffset;
  int iCol;
  int nDoc;
  const char *zDoc;
  int iStart, iEnd;
  int tailEllipsis = 0;
  int iMatch;
  

  sqlite3_free(pSnippet->zSnippet);
  pSnippet->zSnippet = 0;
  aMatch = pSnippet->aMatch;
  nMatch = pSnippet->nMatch;
  fts3SnippetSbInit(&sb);

  for(i=0; i<nMatch; i++){
    aMatch[i].snStatus = SNIPPET_IGNORE;
  }
  nDesired = 0;
  for(i=0; i<FTS3_ROTOR_SZ; i++){
    for(j=0; j<nMatch; j++){
      if( aMatch[j].iTerm==i ){
        aMatch[j].snStatus = SNIPPET_DESIRED;
        nDesired++;
        break;
      }
    }
  }

  iMatch = 0;
  tailCol = -1;
  tailOffset = 0;
  for(i=0; i<nMatch && nDesired>0; i++){
    if( aMatch[i].snStatus!=SNIPPET_DESIRED ) continue;
    nDesired--;
    iCol = aMatch[i].iCol;
    zDoc = (const char*)sqlite3_column_text(pCursor->pStmt, iCol+1);
    nDoc = sqlite3_column_bytes(pCursor->pStmt, iCol+1);
    iStart = aMatch[i].iStart - 40;
    iStart = wordBoundary(iStart, zDoc, nDoc, aMatch, nMatch, iCol);
    if( iStart<=10 ){
      iStart = 0;
    }
    if( iCol==tailCol && iStart<=tailOffset+20 ){
      iStart = tailOffset;
    }
    if( (iCol!=tailCol && tailCol>=0) || iStart!=tailOffset ){
      fts3SnippetTrimWhiteSpace(&sb);
      fts3SnippetAppendWhiteSpace(&sb);
      fts3SnippetAppend(&sb, zEllipsis, -1);
      fts3SnippetAppendWhiteSpace(&sb);
    }
    iEnd = aMatch[i].iStart + aMatch[i].nByte + 40;
    iEnd = wordBoundary(iEnd, zDoc, nDoc, aMatch, nMatch, iCol);
    if( iEnd>=nDoc-10 ){
      iEnd = nDoc;
      tailEllipsis = 0;
    }else{
      tailEllipsis = 1;
    }
    while( iMatch<nMatch && aMatch[iMatch].iCol<iCol ){ iMatch++; }
    while( iStart<iEnd ){
      while( iMatch<nMatch && aMatch[iMatch].iStart<iStart
             && aMatch[iMatch].iCol<=iCol ){
        iMatch++;
      }
      if( iMatch<nMatch && aMatch[iMatch].iStart<iEnd
             && aMatch[iMatch].iCol==iCol ){
        fts3SnippetAppend(&sb, &zDoc[iStart], aMatch[iMatch].iStart - iStart);
        iStart = aMatch[iMatch].iStart;
        fts3SnippetAppend(&sb, zStartMark, -1);
        fts3SnippetAppend(&sb, &zDoc[iStart], aMatch[iMatch].nByte);
        fts3SnippetAppend(&sb, zEndMark, -1);
        iStart += aMatch[iMatch].nByte;
        for(j=iMatch+1; j<nMatch; j++){
          if( aMatch[j].iTerm==aMatch[iMatch].iTerm
              && aMatch[j].snStatus==SNIPPET_DESIRED ){
            nDesired--;
            aMatch[j].snStatus = SNIPPET_IGNORE;
          }
        }
      }else{
        fts3SnippetAppend(&sb, &zDoc[iStart], iEnd - iStart);
        iStart = iEnd;
      }
    }
    tailCol = iCol;
    tailOffset = iEnd;
  }
  fts3SnippetTrimWhiteSpace(&sb);
  if( tailEllipsis ){
    fts3SnippetAppendWhiteSpace(&sb);
    fts3SnippetAppend(&sb, zEllipsis, -1);
  }
  pSnippet->zSnippet = sb.z;
  pSnippet->nSnippet = sb.z ? sb.nUsed : 0;
}

void sqlite3Fts3Offsets(
  sqlite3_context *pCtx,          /* SQLite function call context */
  Fts3Cursor *pCsr                /* Cursor object */
){
  Snippet *p;                     /* Snippet structure */
  int rc = snippetAllOffsets(pCsr, &p);
  if( rc==SQLITE_OK ){
    snippetOffsetText(p);
    if( p->zOffset ){
      sqlite3_result_text(pCtx, p->zOffset, p->nOffset, SQLITE_TRANSIENT);
    }else{
      sqlite3_result_error_nomem(pCtx);
    }
  }else{
    sqlite3_result_error_nomem(pCtx);
  }
  fts3SnippetFree(p);
}

void sqlite3Fts3Snippet(
  sqlite3_context *pCtx,          /* SQLite function call context */
  Fts3Cursor *pCsr,               /* Cursor object */
  const char *zStart,             /* Snippet start text - "<b>" */
  const char *zEnd,               /* Snippet end text - "</b>" */
  const char *zEllipsis           /* Snippet ellipsis text - "<b>...</b>" */
){
  Snippet *p;                     /* Snippet structure */
  int rc = snippetAllOffsets(pCsr, &p);
  if( rc==SQLITE_OK ){
    snippetText(pCsr, p, zStart, zEnd, zEllipsis);
    if( p->zSnippet ){
      sqlite3_result_text(pCtx, p->zSnippet, p->nSnippet, SQLITE_TRANSIENT);
    }else{
      sqlite3_result_error_nomem(pCtx);
    }
  }else{
    sqlite3_result_error_nomem(pCtx);
  }
  fts3SnippetFree(p);
}

/*************************************************************************
** Below this point is the alternative, experimental snippet() implementation.
*/

#define SNIPPET_BUFFER_CHUNK  64
#define SNIPPET_BUFFER_SIZE   SNIPPET_BUFFER_CHUNK*4
#define SNIPPET_BUFFER_MASK   (SNIPPET_BUFFER_SIZE-1)

static void fts3GetDeltaPosition(char **pp, int *piPos){
  int iVal;
  *pp += sqlite3Fts3GetVarint32(*pp, &iVal);
  *piPos += (iVal-2);
}


























/*
** Iterate through all phrase nodes in an FTS3 query, except those that
** are part of a sub-tree that is the right-hand-side of a NOT operator.
** For each phrase node found, the supplied callback function is invoked.
**
** If the callback function returns anything other than SQLITE_OK, 
** the iteration is abandoned and the error code returned immediately.
** Otherwise, SQLITE_OK is returned after a callback has been made for
** all eligible phrase nodes.
*/
static int fts3ExprIterate(
  Fts3Expr *pExpr,                /* Expression to iterate phrases of */
  int (*x)(Fts3Expr *, void *),   /* Callback function to invoke for phrases */
  void *pCtx                      /* Second argument to pass to callback */
){



















  int rc;

  int eType = pExpr->eType;
  if( eType==FTSQUERY_NOT ){
    rc = SQLITE_OK;

  }else if( eType!=FTSQUERY_PHRASE ){
    assert( pExpr->pLeft && pExpr->pRight );





    rc = fts3ExprIterate(pExpr->pLeft, x, pCtx);
    if( rc==SQLITE_OK ){



      rc = fts3ExprIterate(pExpr->pRight, x, pCtx);
    }
  }else{

    rc = x(pExpr, pCtx);


  }

  return rc;
}



typedef struct LoadDoclistCtx LoadDoclistCtx;
struct LoadDoclistCtx {
  Fts3Table *pTab;                /* FTS3 Table */
  int nPhrase;                    /* Number of phrases so far */
};

static int fts3ExprLoadDoclistsCb(Fts3Expr *pExpr, void *ctx){
  int rc = SQLITE_OK;
  LoadDoclistCtx *p = (LoadDoclistCtx *)ctx;

  p->nPhrase++;


  if( pExpr->isLoaded==0 ){
    rc = sqlite3Fts3ExprLoadDoclist(p->pTab, pExpr);
    pExpr->isLoaded = 1;
    if( rc==SQLITE_OK && pExpr->aDoclist ){














      pExpr->pCurrent = pExpr->aDoclist;

      pExpr->pCurrent += sqlite3Fts3GetVarint(pExpr->pCurrent,&pExpr->iCurrent);
    }
  }
  return rc;
}











static int fts3ExprLoadDoclists(Fts3Cursor *pCsr, int *pnPhrase){




  int rc;
  LoadDoclistCtx sCtx = {0, 0};
  sCtx.pTab = (Fts3Table *)pCsr->base.pVtab;
  rc = fts3ExprIterate(pCsr->pExpr, fts3ExprLoadDoclistsCb, (void *)&sCtx);



  *pnPhrase = sCtx.nPhrase;

  return rc;
}

/*
** Each call to this function populates a chunk of a snippet-buffer 
** SNIPPET_BUFFER_CHUNK bytes in size.
**
** Return true if the end of the data has been reached (and all subsequent
** calls to fts3LoadSnippetBuffer() with the same arguments will be no-ops), 
** or false otherwise.
*/
static int fts3LoadSnippetBuffer(
  int iPos,                       /* Document token offset to load data for */
  u8 *aBuffer,                    /* Circular snippet buffer to populate */
  int nList,                      /* Number of position lists in appList */
  char **apList,                  /* IN/OUT: nList position list pointers */
  int *aiPrev                     /* IN/OUT: Previous positions read */
){
  int i;
  int nFin = 0;

  assert( (iPos&(SNIPPET_BUFFER_CHUNK-1))==0 );

  memset(&aBuffer[iPos&SNIPPET_BUFFER_MASK], 0, SNIPPET_BUFFER_CHUNK);

  for(i=0; i<nList; i++){
    int iPrev = aiPrev[i];
    char *pList = apList[i];

    if( !pList ){
      nFin++;
      continue;
    }

    while( iPrev<(iPos+SNIPPET_BUFFER_CHUNK) ){
      if( iPrev>=iPos ){
        aBuffer[iPrev&SNIPPET_BUFFER_MASK] = (u8)(i+1);
      }
      if( 0==((*pList)&0xFE) ){
        nFin++;
        break;
      }
      fts3GetDeltaPosition(&pList, &iPrev); 
    }





    aiPrev[i] = iPrev;





    apList[i] = pList;






  }


















  return (nFin==nList);
}































typedef struct SnippetCtx SnippetCtx;



struct SnippetCtx {
  Fts3Cursor *pCsr;
  int iCol;








  int iPhrase;
  int *aiPrev;
  int *anToken;


  char **apList;





};











static int fts3SnippetFindPositions(Fts3Expr *pExpr, void *ctx){
  SnippetCtx *p = (SnippetCtx *)ctx;
  int iPhrase = p->iPhrase++;
  char *pCsr;

  p->anToken[iPhrase] = pExpr->pPhrase->nToken;
  pCsr = sqlite3Fts3FindPositions(pExpr, p->pCsr->iPrevId, p->iCol);


  if( pCsr ){
    int iVal;

    pCsr += sqlite3Fts3GetVarint32(pCsr, &iVal);
    p->apList[iPhrase] = pCsr;
    p->aiPrev[iPhrase] = iVal-2;




  }

  return SQLITE_OK;
}

static void fts3SnippetCnt(

  int iIdx, 

  int nSnippet, 
  int *anCnt, 
  u8 *aBuffer,
  int *anToken,
  u64 *pHlmask
){
  int iSub =  (iIdx-1)&SNIPPET_BUFFER_MASK;
  int iAdd =  (iIdx+nSnippet-1)&SNIPPET_BUFFER_MASK;
  int iSub2 = (iIdx+(nSnippet/3)-1)&SNIPPET_BUFFER_MASK;
  int iAdd2 = (iIdx+(nSnippet*2/3)-1)&SNIPPET_BUFFER_MASK;

  u64 h = *pHlmask;

  anCnt[ aBuffer[iSub]  ]--;
  anCnt[ aBuffer[iSub2] ]--;
  anCnt[ aBuffer[iAdd]  ]++;
  anCnt[ aBuffer[iAdd2] ]++;

  h = h >> 1;
  if( aBuffer[iAdd] ){
    int j;
    for(j=anToken[aBuffer[iAdd]-1]; j>=1; j--){
      h |= (u64)1 << (nSnippet-j);
    }
  }
  *pHlmask = h;
}

static int fts3SnippetScore(int n, int *anCnt){
  int j;
  int iScore = 0;
  for(j=1; j<=n; j++){
    int nCnt = anCnt[j];
    iScore += nCnt + (nCnt ? 1000 : 0);
  }
  return iScore;
}

static int fts3BestSnippet(
  int nSnippet,                   /* Desired snippet length */
  Fts3Cursor *pCsr,               /* Cursor to create snippet for */
  int iCol,                       /* Index of column to create snippet from */

  int *piPos,                     /* OUT: Starting token for best snippet */
  u64 *pHlmask                    /* OUT: Highlight mask for best snippet */

){
  int rc;                         /* Return Code */
  u8 aBuffer[SNIPPET_BUFFER_SIZE];/* Circular snippet buffer */
  int *aiPrev;                    /* Used by fts3LoadSnippetBuffer() */
  int *anToken;                   /* Number of tokens in each phrase */
  char **apList;                  /* Array of position lists */

  int *anCnt;                     /* Running totals of phrase occurences */
  int nList;

  int i;

  u64 hlmask = 0;                 /* Current mask of highlighted terms */
  u64 besthlmask = 0;             /* Mask of highlighted terms for iBestPos */
  int iBestPos = 0;               /* Starting position of 'best' snippet */
  int iBestScore = 0;             /* Score of best snippet higher->better */
  SnippetCtx sCtx;

  /* Iterate through the phrases in the expression to count them. The same
  ** callback makes sure the doclists are loaded for each phrase.
  */
  rc = fts3ExprLoadDoclists(pCsr, &nList);
  if( rc!=SQLITE_OK ){
    return rc;
  }

  /* Now that it is known how many phrases there are, allocate and zero
  ** the required arrays using malloc().
  */

  apList = sqlite3_malloc(
      sizeof(u8*)*nList +         /* apList */
      sizeof(int)*(nList) +       /* anToken */
      sizeof(int)*nList +         /* aiPrev */
      sizeof(int)*(nList+1)       /* anCnt */
  );
  if( !apList ){
    return SQLITE_NOMEM;
  }
  memset(apList, 0, sizeof(u8*)*nList+sizeof(int)*nList+sizeof(int)*nList);
  anToken = (int *)&apList[nList];
  aiPrev = &anToken[nList];
  anCnt = &aiPrev[nList];

  /* Initialize the contents of the aiPrev and aiList arrays. */


  sCtx.pCsr = pCsr;
  sCtx.iCol = iCol;
  sCtx.apList = apList;
  sCtx.aiPrev = aiPrev;
  sCtx.anToken = anToken;
  sCtx.iPhrase = 0;

  (void)fts3ExprIterate(pCsr->pExpr, fts3SnippetFindPositions, (void *)&sCtx);

  /* Load the first two chunks of data into the buffer. */
  memset(aBuffer, 0, SNIPPET_BUFFER_SIZE);

  fts3LoadSnippetBuffer(0, aBuffer, nList, apList, aiPrev);
  fts3LoadSnippetBuffer(SNIPPET_BUFFER_CHUNK, aBuffer, nList, apList, aiPrev);



  /* Set the initial contents of the highlight-mask and anCnt[] array. */
  for(i=1-nSnippet; i<=0; i++){
    fts3SnippetCnt(i, nSnippet, anCnt, aBuffer, anToken, &hlmask);
  }
  iBestScore = fts3SnippetScore(nList, anCnt);
  besthlmask = hlmask;
  iBestPos = 0;






  for(i=1; 1; i++){
    int iScore;

    if( 0==(i&(SNIPPET_BUFFER_CHUNK-1)) ){
      int iLoad = i + SNIPPET_BUFFER_CHUNK;
      if( fts3LoadSnippetBuffer(iLoad, aBuffer, nList, apList, aiPrev) ) break;
    }

    /* Figure out how highly a snippet starting at token offset i scores
    ** according to fts3SnippetScore(). If it is higher than any previously
    ** considered position, save the current position, score and hlmask as 
    ** the best snippet candidate found so far.
    */
    fts3SnippetCnt(i, nSnippet, anCnt, aBuffer, anToken, &hlmask);
    iScore = fts3SnippetScore(nList, anCnt);
    if( iScore>iBestScore ){
      iBestPos = i;


      iBestScore = iScore;
      besthlmask = hlmask;
    }
  }

  sqlite3_free(apList);
  *piPos = iBestPos;
  *pHlmask = besthlmask;
  return SQLITE_OK;
}

typedef struct StrBuffer StrBuffer;

struct StrBuffer {
  char *z;

  int n;
  int nAlloc;
};

static int fts3StringAppend(
  StrBuffer *pStr, 
  const char *zAppend, 
  int nAppend
){
  if( nAppend<0 ){
    nAppend = (int)strlen(zAppend);
  }





  if( pStr->n+nAppend+1>=pStr->nAlloc ){
    int nAlloc = pStr->nAlloc+nAppend+100;
    char *zNew = sqlite3_realloc(pStr->z, nAlloc);
    if( !zNew ){
      return SQLITE_NOMEM;
    }
    pStr->z = zNew;
    pStr->nAlloc = nAlloc;
  }


  memcpy(&pStr->z[pStr->n], zAppend, nAppend);
  pStr->n += nAppend;
  pStr->z[pStr->n] = '\0';

  return SQLITE_OK;
}





















static int fts3SnippetText(
  Fts3Cursor *pCsr,               /* FTS3 Cursor */

  const char *zDoc,               /* Document to extract snippet from */
  int nDoc,                       /* Size of zDoc in bytes */
































































  int nSnippet,                   /* Number of tokens in extracted snippet */
  int iPos,                       /* Index of first document token in snippet */
  u64 hlmask,                     /* Bitmask of terms to highlight in snippet */
  const char *zOpen,              /* String inserted before highlighted term */
  const char *zClose,             /* String inserted after highlighted term */
  const char *zEllipsis,
  char **pzSnippet                /* OUT: Snippet text */

){
  Fts3Table *pTab = (Fts3Table *)pCsr->base.pVtab;
  int rc;                         /* Return code */


  int iCurrent = 0;
  int iStart = 0;
  int iEnd;




  sqlite3_tokenizer_module *pMod; /* Tokenizer module methods object */
  sqlite3_tokenizer_cursor *pC;   /* Tokenizer cursor open on zDoc/nDoc */
  const char *ZDUMMY;             /* Dummy arguments used with tokenizer */

  int DUMMY1, DUMMY2, DUMMY3;     /* Dummy arguments used with tokenizer */





  StrBuffer res = {0, 0, 0};   /* Result string */




  /* Open a token cursor on the document. Read all tokens up to and 
  ** including token iPos (the first token of the snippet). Set variable
  ** iStart to the byte offset in zDoc of the start of token iPos.
  */
  pMod = (sqlite3_tokenizer_module *)pTab->pTokenizer->pModule;
  rc = pMod->xOpen(pTab->pTokenizer, zDoc, nDoc, &pC);
  while( rc==SQLITE_OK && iCurrent<iPos ){
    rc = pMod->xNext(pC, &ZDUMMY, &DUMMY1, &iStart, &DUMMY2, &iCurrent);

  }
  iEnd = iStart;

  if( rc==SQLITE_OK && iStart>0 ){
    rc = fts3StringAppend(&res, zEllipsis, -1);
  }

  while( rc==SQLITE_OK ){
    int iBegin;
    int iFin;


    rc = pMod->xNext(pC, &ZDUMMY, &DUMMY1, &iBegin, &iFin, &iCurrent);

    if( rc==SQLITE_OK ){
      if( iCurrent>=(iPos+nSnippet) ){
        rc = SQLITE_DONE;
      }else{
        iEnd = iFin;
        if( hlmask & ((u64)1 << (iCurrent-iPos)) ){
          if( fts3StringAppend(&res, &zDoc[iStart], iBegin-iStart)
           || fts3StringAppend(&res, zOpen, -1)
           || fts3StringAppend(&res, &zDoc[iBegin], iEnd-iBegin)
           || fts3StringAppend(&res, zClose, -1)
          ){
            rc = SQLITE_NOMEM;
          }
          iStart = iEnd;
        }
      }
    }
  }
  assert( rc!=SQLITE_OK );
  if( rc==SQLITE_DONE ){




    rc = fts3StringAppend(&res, &zDoc[iStart], iEnd-iStart);




    if( rc==SQLITE_OK ){




      rc = pMod->xNext(pC, &ZDUMMY, &DUMMY1, &DUMMY2, &DUMMY3, &iCurrent);




      if( rc==SQLITE_OK ){
        rc = fts3StringAppend(&res, zEllipsis, -1);



      }else if( rc==SQLITE_DONE ){


        rc = fts3StringAppend(&res, &zDoc[iEnd], -1);
      }

    }
  }



  pMod->xClose(pC);


  if( rc!=SQLITE_OK ){

    sqlite3_free(res.z);
  }else{
    *pzSnippet = res.z;
  }


  return rc;
}


/*
** An instance of this structure is used to collect the 'global' part of
** the matchinfo statistics. The 'global' part consists of the following:
**
**   1. The number of phrases in the query (nPhrase).
**
**   2. The number of columns in the FTS3 table (nCol).
**
**   3. A matrix of (nPhrase*nCol) integers containing the sum of the
**      number of hits for each phrase in each column across all rows

**      of the table.
**
** The total size of the global matchinfo array, assuming the number of
** columns is N and the number of phrases is P is:
**
**   2 + P*(N+1)
**
** The number of hits for the 3rd phrase in the second column is found
** using the expression:
**
**   aGlobal[2 + P*(1+2) + 1]
*/
typedef struct MatchInfo MatchInfo;
struct MatchInfo {
  Fts3Table *pTab;                /* FTS3 Table */
  Fts3Cursor *pCursor;            /* FTS3 Cursor */
  int iPhrase;                    /* Number of phrases so far */
  int nCol;                       /* Number of columns in table */
  u32 *aGlobal;                   /* Pre-allocated buffer */
};

/*
** This function is used to count the entries in a column-list (delta-encoded
** list of term offsets within a single column of a single row).
*/
static int fts3ColumnlistCount(char **ppCollist){
  char *pEnd = *ppCollist;
  char c = 0;
  int nEntry = 0;

  /* A column-list is terminated by either a 0x01 or 0x00. */
  while( 0xFE & (*pEnd | c) ){
    c = *pEnd++ & 0x80;
    if( !c ) nEntry++;
  }

  *ppCollist = pEnd;
  return nEntry;
}

static void fts3LoadColumnlistCounts(char **pp, u32 *aOut){
  char *pCsr = *pp;
  while( *pCsr ){

    sqlite3_int64 iCol = 0;
    if( *pCsr==0x01 ){
      pCsr++;
      pCsr += sqlite3Fts3GetVarint(pCsr, &iCol);
    }
    aOut[iCol] += fts3ColumnlistCount(&pCsr);





  }
  pCsr++;
  *pp = pCsr;
}

/*
** fts3ExprIterate() callback used to collect the "global" matchinfo stats
** for a single query.

*/
static int fts3ExprGlobalMatchinfoCb(
  Fts3Expr *pExpr,                /* Phrase expression node */

  void *pCtx                      /* Pointer to MatchInfo structure */
){
  MatchInfo *p = (MatchInfo *)pCtx;
  char *pCsr;
  char *pEnd;
  const int iStart = 2 + p->nCol*p->iPhrase;

  assert( pExpr->isLoaded );

  /* Fill in the global hit count matrix row for this phrase. */
  pCsr = pExpr->aDoclist;
  pEnd = &pExpr->aDoclist[pExpr->nDoclist];
  while( pCsr<pEnd ){
    while( *pCsr++ & 0x80 );
    fts3LoadColumnlistCounts(&pCsr, &p->aGlobal[iStart]);
  }

  p->iPhrase++;
  return SQLITE_OK;
}






static int fts3ExprLocalMatchinfoCb(
  Fts3Expr *pExpr,                /* Phrase expression node */

  void *pCtx                      /* Pointer to MatchInfo structure */
){
  MatchInfo *p = (MatchInfo *)pCtx;
  int iPhrase = p->iPhrase++;

  if( pExpr->aDoclist ){
    char *pCsr;

    int iOffset = 2 + p->nCol*(p->aGlobal[0]+iPhrase);


    memset(&p->aGlobal[iOffset], 0, p->nCol*sizeof(u32));
    pCsr = sqlite3Fts3FindPositions(pExpr, p->pCursor->iPrevId, -1);

    if( pCsr ) fts3LoadColumnlistCounts(&pCsr, &p->aGlobal[iOffset]);

  }

  return SQLITE_OK;
}

/*
** Populate pCsr->aMatchinfo[] with data for the current row. The 'matchinfo'
** data is an array of 32-bit unsigned integers (C type u32).
*/
static int fts3GetMatchinfo(Fts3Cursor *pCsr){
  MatchInfo g;
  Fts3Table *pTab = (Fts3Table *)pCsr->base.pVtab;




  if( pCsr->aMatchinfo==0 ){





    int rc;
    int nPhrase;
    int nMatchinfo;

    g.pTab = pTab;
    g.nCol = pTab->nColumn;
    g.iPhrase = 0;
    rc = fts3ExprLoadDoclists(pCsr, &nPhrase);
    if( rc!=SQLITE_OK ){
      return rc;
    }

    nMatchinfo = 2 + 2*g.nCol*nPhrase;

    g.iPhrase = 0;
    g.aGlobal = (u32 *)sqlite3_malloc(sizeof(u32)*nMatchinfo);
    if( !g.aGlobal ){ 
      return SQLITE_NOMEM;
    }
    memset(g.aGlobal, 0, sizeof(u32)*nMatchinfo);



    g.aGlobal[0] = nPhrase;
    g.aGlobal[1] = g.nCol;
    (void)fts3ExprIterate(pCsr->pExpr, fts3ExprGlobalMatchinfoCb, (void *)&g);

    pCsr->aMatchinfo = g.aGlobal;
  }

  g.pTab = pTab;
  g.pCursor = pCsr;
  g.nCol = pTab->nColumn;
  g.iPhrase = 0;
  g.aGlobal = pCsr->aMatchinfo;

  if( pCsr->isMatchinfoOk ){
    (void)fts3ExprIterate(pCsr->pExpr, fts3ExprLocalMatchinfoCb, (void *)&g);
    pCsr->isMatchinfoOk = 0;
  }

  return SQLITE_OK;
}




void sqlite3Fts3Snippet2(
  sqlite3_context *pCtx,          /* SQLite function call context */
  Fts3Cursor *pCsr,               /* Cursor object */
  const char *zStart,             /* Snippet start text - "<b>" */
  const char *zEnd,               /* Snippet end text - "</b>" */
  const char *zEllipsis,          /* Snippet ellipsis text - "<b>...</b>" */
  int iCol,                       /* Extract snippet from this column */
  int nToken                      /* Approximate number of tokens in snippet */
){

  int rc;
  int iPos = 0;

  u64 hlmask = 0;











  char *z = 0;

























  int nDoc;




























































































































  const char *zDoc;




  rc = fts3BestSnippet(nToken, pCsr, iCol, &iPos, &hlmask);












  nDoc = sqlite3_column_bytes(pCsr->pStmt, iCol+1);







  zDoc = (const char *)sqlite3_column_text(pCsr->pStmt, iCol+1);
































  if( rc==SQLITE_OK ){




    rc = fts3SnippetText(






        pCsr, zDoc, nDoc, nToken, iPos, hlmask, zStart, zEnd, zEllipsis, &z);


  }




  if( rc!=SQLITE_OK ){
    sqlite3_result_error_code(pCtx, rc);

  }else{
    sqlite3_result_text(pCtx, z, -1, sqlite3_free);
  }

}




void sqlite3Fts3Matchinfo(sqlite3_context *pContext, Fts3Cursor *pCsr){
  int rc = fts3GetMatchinfo(pCsr);
  if( rc!=SQLITE_OK ){
    sqlite3_result_error_code(pContext, rc);
  }else{
    int n = sizeof(u32)*(2+pCsr->aMatchinfo[0]*pCsr->aMatchinfo[1]*2);
    sqlite3_result_blob(pContext, pCsr->aMatchinfo, n, SQLITE_TRANSIENT);
  }
}

#endif







>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
|
>
>

>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>

>
>
>
>
>
>
>
>
>
|
|
>
>
>
>
>

>
|
|
>
|
<
<
<
<
<
<
<
<
<
<
<
<



<
<
<
<
<
<
<
<
<

<
<
<
<
<
<
<
<
<
<
|
|
<
<
<
<
<
<
<
<
|
<
<
<
<
<
<
<
<
<
|
<
<
<
<
<
<
<
<
<
<
<
<
<
<
|
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<

<
<
<
<
|

<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
|
<
<
<
<
<
<
<
|
<
<
<
<
<
<
<
<
<
<
<
<
<
|
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<

<
<
<
<
<
<
<
<
<
<
<
<
<
<
|
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
|
<
<
<
<
|
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
|
|
<
<
<
<
<
|
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<

<
<
<
<
<





>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>













|


>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
|
>
|
|
|
>
|
|
>
>
>
>
>
|
|
>
>
>
|

|
>
|
>
>

>



>
>
|
|
<
<
<
|
|


>

>
>



|
>
>
>
>
>
>
>
>
>
>
>
>
>
>

>


<
|


>
>
>
>
>
>
>
>
>
>
|
>
>
>
>
|
|

|
>
>
>
|
>




|
<
<
<
|
|

|
<
<
<
|
<
|
|
<

<
|
<
|
<
<
<
|
<
<
<
<
|
<
<
<
<
<
<


|


>
>
>
>
|
>
>
>
>
>
|
>
>
>
>
>
>
|
>
>
>
>
>
>
|
>
>
>
>
>
>
>
>
>
>
>
|

>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>

>
|
>
>
>
|
|
|
>
>
>
>
>
>
>
>
|
|
|
>
>
|
>
>
>
>
>
|
>
>
>
>
>
|
>
>
>
>
>
|
|
|


|
<

>

|
>
|
|
|
>
>
>
>

>



<
>
|
>
|
|
|
<
<
<
<
<
<
<
|
<
|
<
<
<
<
|
<
<
<
<
|
<
<
|
<
|
<
<
|
<
<
<
<
|
<
|




>
|
|
>


<
|
|
|
>
|
<

<
|
<
<
<
<
<




|





|

>
|
<
<
<
<
<
|


|
<
<
<

|
>
>
|
|
|
<
<
|
>
|

<
<
>
|
<
>
>
|
<
<
<

<
<
<

>
>
>
>
>
|

|
<
<
<
<
|
<
<
<
<
<
|
|

|
>
>

<



|
|
<



|
>
|
|
>
|
<
<
|

|
|
|


|


>
>
>
>










>







>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
|
|
>
|
|
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>

<
<


|
<
>



>
>
|
<
|
|
>
>
>


|
>
|
>
>
>
>
|
<
>
|
>
>
|
<
<
<


|
<
>

<
|
<
<
|
<

|
|
>
>

<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
|

>
>
>
>
|
>
>
>
>
|
>
>
>
>
|
>
>
>
>
|
|
>
>
>
|
>
>
|

>

|
>
>

<
>
>
|
>
|
<
|

>
>





|
|
|
<
<
|
<
|
<
>
|
<
|
<
<
<
<
<
|

<
<
<
<
<
<
<
<
<
<
|
<
<
<
















|


>





|
>
>
>
>
>







|
>



>





|







|
|


<



>
>
>
>
>


>



<



>
|

>
|

>
|
>






|
|


|

>
>
>
>

>
>
>
>
>
|
|
|

<
<
|
|



<
|

<
|
|


|

>
>
|
|
|

|


<
<
<
<
|
<

|






>
>
>
|








>
|
|
>
|
>
>
>
>
>
>
>
>
>
>
>
|
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
|
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>

>

>
>
|
>
>
>
>

>
>
>
>
>
>
>

>
>
>
>
>
>
>
|
>
>
>
>

>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>

>
>
>
>
|
>
>
>
>
>
>
|
>
>

>
>
>
>


>

|

>


>
>
>





|





14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88












89
90
91









92










93
94








95









96














97

























































































































































































































98




99
100










































































































101







102













103












































































104














105



























106




107























108
109





110






























































































































111





112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210



211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240

241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272



273
274
275
276



277

278
279

280

281

282



283




284






285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402

403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419

420
421
422
423
424
425







426

427




428




429


430

431


432




433

434
435
436
437
438
439
440
441
442
443
444

445
446
447
448
449

450

451





452
453
454
455
456
457
458
459
460
461
462
463
464
465





466
467
468
469



470
471
472
473
474
475
476


477
478
479
480


481
482

483
484
485



486



487
488
489
490
491
492
493
494
495




496





497
498
499
500
501
502
503

504
505
506
507
508

509
510
511
512
513
514
515
516
517


518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639


640
641
642

643
644
645
646
647
648
649

650
651
652
653
654
655
656
657
658
659
660
661
662
663
664

665
666
667
668
669



670
671
672

673
674

675


676

677
678
679
680
681
682



















683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719

720
721
722
723
724

725
726
727
728
729
730
731
732
733
734
735
736


737

738

739
740

741





742
743










744



745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805

806
807
808
809
810
811
812
813
814
815
816
817
818
819

820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857


858
859
860
861
862

863
864

865
866
867
868
869
870
871
872
873
874
875
876
877
878
879




880

881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
1001
1002
1003
1004
1005
1006
1007
1008
1009
1010
1011
1012
1013
1014
1015
1016
1017
1018
1019
1020
1021
1022
1023
1024
1025
1026
1027
1028
1029
1030
1031
1032
1033
1034
1035
1036
1037
1038
1039
1040
1041
1042
1043
1044
1045
1046
1047
1048
1049
1050
1051
1052
1053
1054
1055
1056
1057
1058
1059
1060
1061
1062
1063
1064
1065
1066
1067
1068
1069
1070
1071
1072
1073
1074
1075
1076
1077
1078
1079
1080
1081
1082
1083
1084
1085
1086
1087
1088
1089
1090
1091
1092
1093
1094
1095
1096
1097
1098
1099
1100
1101
1102
1103
1104
1105
1106
1107
1108
1109
1110
1111
1112
1113
1114
1115
1116
1117
1118
1119
1120
1121
1122
1123
1124
1125
1126
1127
1128
1129
1130
1131
1132
1133
1134
1135
1136
1137
1138
1139
1140
1141
1142
1143
1144
1145
1146
1147
1148
1149
1150
1151
1152
1153
1154
1155
1156
1157
1158
1159
1160
1161
1162
1163
1164
1165
1166
1167
1168
1169
#if !defined(SQLITE_CORE) || defined(SQLITE_ENABLE_FTS3)

#include "fts3Int.h"
#include <string.h>
#include <assert.h>
#include <ctype.h>


/*
** Used as an fts3ExprIterate() context when loading phrase doclists to
** Fts3Expr.aDoclist[]/nDoclist.
*/
typedef struct LoadDoclistCtx LoadDoclistCtx;
struct LoadDoclistCtx {
  Fts3Table *pTab;                /* FTS3 Table */
  int nPhrase;                    /* Number of phrases seen so far */
  int nToken;                     /* Number of tokens seen so far */
};

/*
** The following types are used as part of the implementation of the 
** fts3BestSnippet() routine.
*/
typedef struct SnippetIter SnippetIter;
typedef struct SnippetPhrase SnippetPhrase;
typedef struct SnippetFragment SnippetFragment;

struct SnippetIter {
  Fts3Cursor *pCsr;               /* Cursor snippet is being generated from */
  int iCol;                       /* Extract snippet from this column */
  int nSnippet;                   /* Requested snippet length (in tokens) */
  int nPhrase;                    /* Number of phrases in query */
  SnippetPhrase *aPhrase;         /* Array of size nPhrase */
  int iCurrent;                   /* First token of current snippet */
};

struct SnippetPhrase {
  int nToken;                     /* Number of tokens in phrase */
  char *pList;                    /* Pointer to start of phrase position list */
  int iHead;                      /* Next value in position list */
  char *pHead;                    /* Position list data following iHead */
  int iTail;                      /* Next value in trailing position list */
  char *pTail;                    /* Position list data following iTail */
};

struct SnippetFragment {
  int iCol;                       /* Column snippet is extracted from */
  int iPos;                       /* Index of first token in snippet */
  u64 covered;                    /* Mask of query phrases covered */
  u64 hlmask;                     /* Mask of snippet terms to highlight */
};

/*
** This type is used as an fts3ExprIterate() context object while 
** accumulating the data returned by the matchinfo() function.
*/
typedef struct MatchInfo MatchInfo;
struct MatchInfo {
  Fts3Cursor *pCursor;            /* FTS3 Cursor */
  int nCol;                       /* Number of columns in table */
  u32 *aMatchinfo;                /* Pre-allocated buffer */
};



/*
** The snippet() and offsets() functions both return text values. An instance
** of the following structure is used to accumulate those values while the
** functions are running. See fts3StringAppend() for details.
*/
typedef struct StrBuffer StrBuffer;
struct StrBuffer {
  char *z;                        /* Pointer to buffer containing string */
  int n;                          /* Length of z in bytes (excl. nul-term) */
  int nAlloc;                     /* Allocated size of buffer z in bytes */












};











/*










** This function is used to help iterate through a position-list. A position
** list is a list of unique integers, sorted from smallest to largest. Each








** element of the list is represented by an FTS3 varint that takes the value









** of the difference between the current element and the previous one plus














** two. For example, to store the position-list:

























































































































































































































**




**     4 9 113
**










































































































** the three varints:







**













**     6 7 106












































































**














** are encoded.



























**




** When this function is called, *pp points to the start of an element of























** the list. *piPos contains the value of the previous entry in the list.
** After it returns, *piPos contains the value of the next element of the





** list and *pp is advanced to the following varint.






























































































































*/





static void fts3GetDeltaPosition(char **pp, int *piPos){
  int iVal;
  *pp += sqlite3Fts3GetVarint32(*pp, &iVal);
  *piPos += (iVal-2);
}

/*
** Helper function for fts3ExprIterate() (see below).
*/
static int fts3ExprIterate2(
  Fts3Expr *pExpr,                /* Expression to iterate phrases of */
  int *piPhrase,                  /* Pointer to phrase counter */
  int (*x)(Fts3Expr*,int,void*),  /* Callback function to invoke for phrases */
  void *pCtx                      /* Second argument to pass to callback */
){
  int rc;                         /* Return code */
  int eType = pExpr->eType;       /* Type of expression node pExpr */

  if( eType!=FTSQUERY_PHRASE ){
    assert( pExpr->pLeft && pExpr->pRight );
    rc = fts3ExprIterate2(pExpr->pLeft, piPhrase, x, pCtx);
    if( rc==SQLITE_OK && eType!=FTSQUERY_NOT ){
      rc = fts3ExprIterate2(pExpr->pRight, piPhrase, x, pCtx);
    }
  }else{
    rc = x(pExpr, *piPhrase, pCtx);
    (*piPhrase)++;
  }
  return rc;
}

/*
** Iterate through all phrase nodes in an FTS3 query, except those that
** are part of a sub-tree that is the right-hand-side of a NOT operator.
** For each phrase node found, the supplied callback function is invoked.
**
** If the callback function returns anything other than SQLITE_OK, 
** the iteration is abandoned and the error code returned immediately.
** Otherwise, SQLITE_OK is returned after a callback has been made for
** all eligible phrase nodes.
*/
static int fts3ExprIterate(
  Fts3Expr *pExpr,                /* Expression to iterate phrases of */
  int (*x)(Fts3Expr*,int,void*),  /* Callback function to invoke for phrases */
  void *pCtx                      /* Second argument to pass to callback */
){
  int iPhrase = 0;                /* Variable used as the phrase counter */
  return fts3ExprIterate2(pExpr, &iPhrase, x, pCtx);
}

/*
** The argument to this function is always a phrase node. Its doclist 
** (Fts3Expr.aDoclist[]) and the doclists associated with all phrase nodes
** to the left of this one in the query tree have already been loaded.
**
** If this phrase node is part of a series of phrase nodes joined by 
** NEAR operators (and is not the left-most of said series), then elements are
** removed from the phrases doclist consistent with the NEAR restriction. If
** required, elements may be removed from the doclists of phrases to the
** left of this one that are part of the same series of NEAR operator 
** connected phrases.
**
** If an OOM error occurs, SQLITE_NOMEM is returned. Otherwise, SQLITE_OK.
*/
static int fts3ExprNearTrim(Fts3Expr *pExpr){
  int rc = SQLITE_OK;
  Fts3Expr *pParent = pExpr->pParent;

  assert( pExpr->eType==FTSQUERY_PHRASE );
  while( rc==SQLITE_OK
   && pParent 
   && pParent->eType==FTSQUERY_NEAR 
   && pParent->pRight==pExpr 
  ){
    /* This expression (pExpr) is the right-hand-side of a NEAR operator. 
    ** Find the expression to the left of the same operator.
    */
    int nNear = pParent->nNear;
    Fts3Expr *pLeft = pParent->pLeft;

    if( pLeft->eType!=FTSQUERY_PHRASE ){
      assert( pLeft->eType==FTSQUERY_NEAR );
      assert( pLeft->pRight->eType==FTSQUERY_PHRASE );
      pLeft = pLeft->pRight;
    }

    rc = sqlite3Fts3ExprNearTrim(pLeft, pExpr, nNear);

    pExpr = pLeft;
    pParent = pExpr->pParent;
  }

  return rc;
}

/*
** This is an fts3ExprIterate() callback used while loading the doclists
** for each phrase into Fts3Expr.aDoclist[]/nDoclist. See also
** fts3ExprLoadDoclists().



*/
static int fts3ExprLoadDoclistsCb1(Fts3Expr *pExpr, int iPhrase, void *ctx){
  int rc = SQLITE_OK;
  LoadDoclistCtx *p = (LoadDoclistCtx *)ctx;

  p->nPhrase++;
  p->nToken += pExpr->pPhrase->nToken;

  if( pExpr->isLoaded==0 ){
    rc = sqlite3Fts3ExprLoadDoclist(p->pTab, pExpr);
    pExpr->isLoaded = 1;
    if( rc==SQLITE_OK ){
      rc = fts3ExprNearTrim(pExpr);
    }
  }

  return rc;
}

/*
** This is an fts3ExprIterate() callback used while loading the doclists
** for each phrase into Fts3Expr.aDoclist[]/nDoclist. See also
** fts3ExprLoadDoclists().
*/
static int fts3ExprLoadDoclistsCb2(Fts3Expr *pExpr, int iPhrase, void *ctx){
  if( pExpr->aDoclist ){
      pExpr->pCurrent = pExpr->aDoclist;
    pExpr->iCurrent = 0;
      pExpr->pCurrent += sqlite3Fts3GetVarint(pExpr->pCurrent,&pExpr->iCurrent);
    }

  return SQLITE_OK;
}

/*
** Load the doclists for each phrase in the query associated with FTS3 cursor
** pCsr. 
**
** If pnPhrase is not NULL, then *pnPhrase is set to the number of matchable 
** phrases in the expression (all phrases except those directly or 
** indirectly descended from the right-hand-side of a NOT operator). If 
** pnToken is not NULL, then it is set to the number of tokens in all
** matchable phrases of the expression.
*/
static int fts3ExprLoadDoclists(
  Fts3Cursor *pCsr,               /* Fts3 cursor for current query */
  int *pnPhrase,                  /* OUT: Number of phrases in query */
  int *pnToken                    /* OUT: Number of tokens in query */
){
  int rc;                         /* Return Code */
  LoadDoclistCtx sCtx = {0,0,0};  /* Context for fts3ExprIterate() */
  sCtx.pTab = (Fts3Table *)pCsr->base.pVtab;
  rc = fts3ExprIterate(pCsr->pExpr, fts3ExprLoadDoclistsCb1, (void *)&sCtx);
  if( rc==SQLITE_OK ){
    (void)fts3ExprIterate(pCsr->pExpr, fts3ExprLoadDoclistsCb2, 0);
  }
  if( pnPhrase ) *pnPhrase = sCtx.nPhrase;
  if( pnToken ) *pnToken = sCtx.nToken;
  return rc;
}

/*
** Advance the position list iterator specified by the first two 



** arguments so that it points to the first element with a value greater
** than or equal to parameter iNext.
*/
static void fts3SnippetAdvance(char **ppIter, int *piIter, int iNext){



  char *pIter = *ppIter;

  if( pIter ){
    int iIter = *piIter;



    while( iIter<iNext ){

      if( 0==(*pIter & 0xFE) ){



        iIter = -1;




        pIter = 0;






        break;
      }
      fts3GetDeltaPosition(&pIter, &iIter);
    }

    *piIter = iIter;
    *ppIter = pIter;
  }
}

/*
** Advance the snippet iterator to the next candidate snippet.
*/
static int fts3SnippetNextCandidate(SnippetIter *pIter){
  int i;                          /* Loop counter */

  if( pIter->iCurrent<0 ){
    /* The SnippetIter object has just been initialized. The first snippet
    ** candidate always starts at offset 0 (even if this candidate has a
    ** score of 0.0).
    */
    pIter->iCurrent = 0;

    /* Advance the 'head' iterator of each phrase to the first offset that
    ** is greater than or equal to (iNext+nSnippet).
    */
    for(i=0; i<pIter->nPhrase; i++){
      SnippetPhrase *pPhrase = &pIter->aPhrase[i];
      fts3SnippetAdvance(&pPhrase->pHead, &pPhrase->iHead, pIter->nSnippet);
    }
  }else{
    int iStart;
    int iEnd = 0x7FFFFFFF;

    for(i=0; i<pIter->nPhrase; i++){
      SnippetPhrase *pPhrase = &pIter->aPhrase[i];
      if( pPhrase->pHead && pPhrase->iHead<iEnd ){
        iEnd = pPhrase->iHead;
      }
    }
    if( iEnd==0x7FFFFFFF ){
      return 1;
}

    pIter->iCurrent = iStart = iEnd - pIter->nSnippet + 1;
    for(i=0; i<pIter->nPhrase; i++){
      SnippetPhrase *pPhrase = &pIter->aPhrase[i];
      fts3SnippetAdvance(&pPhrase->pHead, &pPhrase->iHead, iEnd+1);
      fts3SnippetAdvance(&pPhrase->pTail, &pPhrase->iTail, iStart);
    }
  }

  return 0;
}

/*
** Retrieve information about the current candidate snippet of snippet 
** iterator pIter.
*/
static void fts3SnippetDetails(
  SnippetIter *pIter,             /* Snippet iterator */
  u64 mCovered,                   /* Bitmask of phrases already covered */
  int *piToken,                   /* OUT: First token of proposed snippet */
  int *piScore,                   /* OUT: "Score" for this snippet */
  u64 *pmCover,                   /* OUT: Bitmask of phrases covered */
  u64 *pmHighlight                /* OUT: Bitmask of terms to highlight */
){
  int iStart = pIter->iCurrent;   /* First token of snippet */
  int iScore = 0;                 /* Score of this snippet */
  int i;                          /* Loop counter */
  u64 mCover = 0;                 /* Mask of phrases covered by this snippet */
  u64 mHighlight = 0;             /* Mask of tokens to highlight in snippet */

  for(i=0; i<pIter->nPhrase; i++){
    SnippetPhrase *pPhrase = &pIter->aPhrase[i];
    if( pPhrase->pTail ){
      char *pCsr = pPhrase->pTail;
      int iCsr = pPhrase->iTail;

      while( iCsr<(iStart+pIter->nSnippet) ){
        int j;
        u64 mPhrase = (u64)1 << i;
        u64 mPos = (u64)1 << (iCsr - iStart);
        assert( iCsr>=iStart );
        if( (mCover|mCovered)&mPhrase ){
          iScore++;
        }else{
          iScore += 1000;
        }
        mCover |= mPhrase;

        for(j=0; j<pPhrase->nToken; j++){
          mHighlight |= (mPos>>j);
        }

        if( 0==(*pCsr & 0x0FE) ) break;
        fts3GetDeltaPosition(&pCsr, &iCsr);
      }
    }
  }

  /* Set the output variables before returning. */
  *piToken = iStart;
  *piScore = iScore;
  *pmCover = mCover;
  *pmHighlight = mHighlight;
}

/*
** This function is an fts3ExprIterate() callback used by fts3BestSnippet().
** Each invocation populates an element of the SnippetIter.aPhrase[] array.
*/
static int fts3SnippetFindPositions(Fts3Expr *pExpr, int iPhrase, void *ctx){
  SnippetIter *p = (SnippetIter *)ctx;
  SnippetPhrase *pPhrase = &p->aPhrase[iPhrase];
  char *pCsr;

  pPhrase->nToken = pExpr->pPhrase->nToken;


  pCsr = sqlite3Fts3FindPositions(pExpr, p->pCsr->iPrevId, p->iCol);
  if( pCsr ){
    int iFirst = 0;
    pPhrase->pList = pCsr;
    fts3GetDeltaPosition(&pCsr, &iFirst);
    pPhrase->pHead = pCsr;
    pPhrase->pTail = pCsr;
    pPhrase->iHead = iFirst;
    pPhrase->iTail = iFirst;
  }else{
    assert( pPhrase->pList==0 && pPhrase->pHead==0 && pPhrase->pTail==0 );
  }

  return SQLITE_OK;
}


/*
** Select the fragment of text consisting of nFragment contiguous tokens 
** from column iCol that represent the "best" snippet. The best snippet
** is the snippet with the highest score, where scores are calculated
** by adding:
**







**   (a) +1 point for each occurence of a matchable phrase in the snippet.

**




**   (b) +1000 points for the first occurence of each matchable phrase in 




**       the snippet for which the corresponding mCovered bit is not set.


**

** The selected snippet parameters are stored in structure *pFragment before


** returning. The score of the selected snippet is stored in *piScore




** before returning.

*/
static int fts3BestSnippet(
  int nSnippet,                   /* Desired snippet length */
  Fts3Cursor *pCsr,               /* Cursor to create snippet for */
  int iCol,                       /* Index of column to create snippet from */
  u64 mCovered,                   /* Mask of phrases already covered */
  u64 *pmSeen,                    /* IN/OUT: Mask of phrases seen */
  SnippetFragment *pFragment,     /* OUT: Best snippet found */
  int *piScore                    /* OUT: Score of snippet pFragment */
){
  int rc;                         /* Return Code */

  int nList;                      /* Number of phrases in expression */
  SnippetIter sIter;              /* Iterates through snippet candidates */
  int nByte;                      /* Number of bytes of space to allocate */
  int iBestScore = -1;            /* Best snippet score found so far */
  int i;                          /* Loop counter */



  memset(&sIter, 0, sizeof(sIter));






  /* Iterate through the phrases in the expression to count them. The same
  ** callback makes sure the doclists are loaded for each phrase.
  */
  rc = fts3ExprLoadDoclists(pCsr, &nList, 0);
  if( rc!=SQLITE_OK ){
    return rc;
  }

  /* Now that it is known how many phrases there are, allocate and zero
  ** the required space using malloc().
  */
  nByte = sizeof(SnippetPhrase) * nList;
  sIter.aPhrase = (SnippetPhrase *)sqlite3_malloc(nByte);





  if( !sIter.aPhrase ){
    return SQLITE_NOMEM;
  }
  memset(sIter.aPhrase, 0, nByte);




  /* Initialize the contents of the SnippetIter object. Then iterate through
  ** the set of phrases in the expression to populate the aPhrase[] array.
  */
  sIter.pCsr = pCsr;
  sIter.iCol = iCol;
  sIter.nSnippet = nSnippet;


  sIter.nPhrase = nList;
  sIter.iCurrent = -1;
  (void)fts3ExprIterate(pCsr->pExpr, fts3SnippetFindPositions, (void *)&sIter);



  /* Set the *pmSeen output variable. */
  for(i=0; i<nList; i++){

    if( sIter.aPhrase[i].pHead ){
      *pmSeen |= (u64)1 << i;
    }



  }




  /* Loop through all candidate snippets. Store the best snippet in 
  ** *pFragment. Store its associated 'score' in iBestScore.
  */
  pFragment->iCol = iCol;
  while( !fts3SnippetNextCandidate(&sIter) ){
    int iPos;
    int iScore;
    u64 mCover;




    u64 mHighlight;





    fts3SnippetDetails(&sIter, mCovered, &iPos, &iScore, &mCover, &mHighlight);
    assert( iScore>=0 );
    if( iScore>iBestScore ){
      pFragment->iPos = iPos;
      pFragment->hlmask = mHighlight;
      pFragment->covered = mCover;
      iBestScore = iScore;

    }
  }

  sqlite3_free(sIter.aPhrase);
  *piScore = iBestScore;

  return SQLITE_OK;
}


/*
** Append a string to the string-buffer passed as the first argument.
**
** If nAppend is negative, then the length of the string zAppend is
** determined using strlen().


*/
static int fts3StringAppend(
  StrBuffer *pStr,                /* Buffer to append to */
  const char *zAppend,            /* Pointer to data to append to buffer */
  int nAppend                     /* Size of zAppend in bytes (or -1) */
){
  if( nAppend<0 ){
    nAppend = strlen(zAppend);
  }

  /* If there is insufficient space allocated at StrBuffer.z, use realloc()
  ** to grow the buffer until so that it is big enough to accomadate the
  ** appended data.
  */
  if( pStr->n+nAppend+1>=pStr->nAlloc ){
    int nAlloc = pStr->nAlloc+nAppend+100;
    char *zNew = sqlite3_realloc(pStr->z, nAlloc);
    if( !zNew ){
      return SQLITE_NOMEM;
    }
    pStr->z = zNew;
    pStr->nAlloc = nAlloc;
  }

  /* Append the data to the string buffer. */
  memcpy(&pStr->z[pStr->n], zAppend, nAppend);
  pStr->n += nAppend;
  pStr->z[pStr->n] = '\0';

  return SQLITE_OK;
}

/*
** The fts3BestSnippet() function often selects snippets that end with a
** query term. That is, the final term of the snippet is always a term
** that requires highlighting. For example, if 'X' is a highlighted term
** and '.' is a non-highlighted term, BestSnippet() may select:
**
**     ........X.....X
**
** This function "shifts" the beginning of the snippet forward in the 
** document so that there are approximately the same number of 
** non-highlighted terms to the right of the final highlighted term as there
** are to the left of the first highlighted term. For example, to this:
**
**     ....X.....X....
**
** This is done as part of extracting the snippet text, not when selecting
** the snippet. Snippet selection is done based on doclists only, so there
** is no way for fts3BestSnippet() to know whether or not the document 
** actually contains terms that follow the final highlighted term. 
*/
int fts3SnippetShift(
  Fts3Table *pTab,                /* FTS3 table snippet comes from */
  int nSnippet,                   /* Number of tokens desired for snippet */
  const char *zDoc,               /* Document text to extract snippet from */
  int nDoc,                       /* Size of buffer zDoc in bytes */
  int *piPos,                     /* IN/OUT: First token of snippet */
  u64 *pHlmask                    /* IN/OUT: Mask of tokens to highlight */
){
  u64 hlmask = *pHlmask;          /* Local copy of initial highlight-mask */

  if( hlmask ){
    int nLeft;                    /* Tokens to the left of first highlight */
    int nRight;                   /* Tokens to the right of last highlight */
    int nDesired;                 /* Ideal number of tokens to shift forward */

    for(nLeft=0; !(hlmask & ((u64)1 << nLeft)); nLeft++);
    for(nRight=0; !(hlmask & ((u64)1 << (nSnippet-1-nRight))); nRight++);
    nDesired = (nLeft-nRight)/2;

    /* Ideally, the start of the snippet should be pushed forward in the
    ** document nDesired tokens. This block checks if there are actually
    ** nDesired tokens to the right of the snippet. If so, *piPos and
    ** *pHlMask are updated to shift the snippet nDesired tokens to the
    ** right. Otherwise, the snippet is shifted by the number of tokens
    ** available.
    */
    if( nDesired>0 ){
      int nShift;                 /* Number of tokens to shift snippet by */
      int iCurrent = 0;           /* Token counter */
      int rc;                     /* Return Code */
      sqlite3_tokenizer_module *pMod;
      sqlite3_tokenizer_cursor *pC;
      pMod = (sqlite3_tokenizer_module *)pTab->pTokenizer->pModule;

      /* Open a cursor on zDoc/nDoc. Check if there are (nSnippet+nDesired)
      ** or more tokens in zDoc/nDoc.
      */
      rc = pMod->xOpen(pTab->pTokenizer, zDoc, nDoc, &pC);
      if( rc!=SQLITE_OK ){
        return rc;
      }
      pC->pTokenizer = pTab->pTokenizer;
      while( rc==SQLITE_OK && iCurrent<(nSnippet+nDesired) ){
        const char *ZDUMMY; int DUMMY1, DUMMY2, DUMMY3;
        rc = pMod->xNext(pC, &ZDUMMY, &DUMMY1, &DUMMY2, &DUMMY3, &iCurrent);
      }
      pMod->xClose(pC);
      if( rc!=SQLITE_OK && rc!=SQLITE_DONE ){ return rc; }

      nShift = (rc==SQLITE_DONE)+iCurrent-nSnippet;
      assert( nShift<=nDesired );
      if( nShift>0 ){
        *piPos += nShift;
        *pHlmask = hlmask >> nShift;
      }
    }
  }
  return SQLITE_OK;
}

/*
** Extract the snippet text for fragment pFragment from cursor pCsr and
** append it to string buffer pOut.
*/
static int fts3SnippetText(
  Fts3Cursor *pCsr,               /* FTS3 Cursor */
  SnippetFragment *pFragment,     /* Snippet to extract */
  int iFragment,                  /* Fragment number */
  int isLast,                     /* True for final fragment in snippet */
  int nSnippet,                   /* Number of tokens in extracted snippet */


  const char *zOpen,              /* String inserted before highlighted term */
  const char *zClose,             /* String inserted after highlighted term */
  const char *zEllipsis,          /* String inserted between snippets */

  StrBuffer *pOut                 /* Write output here */
){
  Fts3Table *pTab = (Fts3Table *)pCsr->base.pVtab;
  int rc;                         /* Return code */
  const char *zDoc;               /* Document text to extract snippet from */
  int nDoc;                       /* Size of zDoc in bytes */
  int iCurrent = 0;               /* Current token number of document */

  int iEnd = 0;                   /* Byte offset of end of current token */
  int isShiftDone = 0;            /* True after snippet is shifted */
  int iPos = pFragment->iPos;     /* First token of snippet */
  u64 hlmask = pFragment->hlmask; /* Highlight-mask for snippet */
  int iCol = pFragment->iCol+1;   /* Query column to extract text from */
  sqlite3_tokenizer_module *pMod; /* Tokenizer module methods object */
  sqlite3_tokenizer_cursor *pC;   /* Tokenizer cursor open on zDoc/nDoc */
  const char *ZDUMMY;             /* Dummy argument used with tokenizer */
  int DUMMY1;                     /* Dummy argument used with tokenizer */
  
  zDoc = (const char *)sqlite3_column_text(pCsr->pStmt, iCol);
  if( zDoc==0 ){
    if( sqlite3_column_type(pCsr->pStmt, iCol)!=SQLITE_NULL ){
      return SQLITE_NOMEM;
    }

    return SQLITE_OK;
  }
  nDoc = sqlite3_column_bytes(pCsr->pStmt, iCol);

  /* Open a token cursor on the document. */



  pMod = (sqlite3_tokenizer_module *)pTab->pTokenizer->pModule;
  rc = pMod->xOpen(pTab->pTokenizer, zDoc, nDoc, &pC);
  if( rc!=SQLITE_OK ){

    return rc;
  }

  pC->pTokenizer = pTab->pTokenizer;




  while( rc==SQLITE_OK ){
    int iBegin;                   /* Offset in zDoc of start of token */
    int iFin;                     /* Offset in zDoc of end of token */
    int isHighlight;              /* True for highlighted terms */

    rc = pMod->xNext(pC, &ZDUMMY, &DUMMY1, &iBegin, &iFin, &iCurrent);



















    if( rc!=SQLITE_OK ){
  if( rc==SQLITE_DONE ){
        /* Special case - the last token of the snippet is also the last token
        ** of the column. Append any punctuation that occurred between the end
        ** of the previous token and the end of the document to the output. 
        ** Then break out of the loop. */
        rc = fts3StringAppend(pOut, &zDoc[iEnd], -1);
      }
      break;
    }
    if( iCurrent<iPos ){ continue; }

    if( !isShiftDone ){
      int n = nDoc - iBegin;
      rc = fts3SnippetShift(pTab, nSnippet, &zDoc[iBegin], n, &iPos, &hlmask);
      isShiftDone = 1;

      /* Now that the shift has been done, check if the initial "..." are
      ** required. They are required if (a) this is not the first fragment,
      ** or (b) this fragment does not begin at position 0 of its column. 
      */
      if( rc==SQLITE_OK && (iPos>0 || iFragment>0) ){
        rc = fts3StringAppend(pOut, zEllipsis, -1);
      }
      if( rc!=SQLITE_OK || iCurrent<iPos ) continue;
    }

    if( iCurrent>=(iPos+nSnippet) ){
      if( isLast ){
        rc = fts3StringAppend(pOut, zEllipsis, -1);
      }
      break;
    }

    /* Set isHighlight to true if this term should be highlighted. */
    isHighlight = (hlmask & ((u64)1 << (iCurrent-iPos)))!=0;


    if( iCurrent>iPos ) rc = fts3StringAppend(pOut, &zDoc[iEnd], iBegin-iEnd);
    if( rc==SQLITE_OK && isHighlight ) rc = fts3StringAppend(pOut, zOpen, -1);
    if( rc==SQLITE_OK ) rc = fts3StringAppend(pOut, &zDoc[iBegin], iFin-iBegin);
    if( rc==SQLITE_OK && isHighlight ) rc = fts3StringAppend(pOut, zClose, -1);


    iEnd = iFin;
  }

  pMod->xClose(pC);
  return rc;
}


/*
** This function is used to count the entries in a column-list (a 
** delta-encoded list of term offsets within a single column of a single 
** row). When this function is called, *ppCollist should point to the


** beginning of the first varint in the column-list (the varint that

** contains the position of the first matching term in the column data).

** Before returning, *ppCollist is set to point to the first byte after
** the last varint in the column-list (either the 0x00 signifying the end

** of the position-list, or the 0x01 that precedes the column number of





** the next column in the position-list).
**










** The number of elements in the column-list is returned.



*/
static int fts3ColumnlistCount(char **ppCollist){
  char *pEnd = *ppCollist;
  char c = 0;
  int nEntry = 0;

  /* A column-list is terminated by either a 0x01 or 0x00. */
  while( 0xFE & (*pEnd | c) ){
    c = *pEnd++ & 0x80;
    if( !c ) nEntry++;
  }

  *ppCollist = pEnd;
  return nEntry;
}

static void fts3LoadColumnlistCounts(char **pp, u32 *aOut, int isGlobal){
  char *pCsr = *pp;
  while( *pCsr ){
    int nHit;
    sqlite3_int64 iCol = 0;
    if( *pCsr==0x01 ){
      pCsr++;
      pCsr += sqlite3Fts3GetVarint(pCsr, &iCol);
    }
    nHit = fts3ColumnlistCount(&pCsr);
    assert( nHit>0 );
    if( isGlobal ){
      aOut[iCol*3+1]++;
    }
    aOut[iCol*3] += nHit;
  }
  pCsr++;
  *pp = pCsr;
}

/*
** fts3ExprIterate() callback used to collect the "global" matchinfo stats
** for a single query. The "global" stats are those elements of the matchinfo
** array that are constant for all rows returned by the current query.
*/
static int fts3ExprGlobalMatchinfoCb(
  Fts3Expr *pExpr,                /* Phrase expression node */
  int iPhrase,                    /* Phrase number (numbered from zero) */
  void *pCtx                      /* Pointer to MatchInfo structure */
){
  MatchInfo *p = (MatchInfo *)pCtx;
  char *pCsr;
  char *pEnd;
  const int iStart = 2 + (iPhrase * p->nCol * 3) + 1;

  assert( pExpr->isLoaded );

  /* Fill in the global hit count matrix row for this phrase. */
  pCsr = pExpr->aDoclist;
  pEnd = &pExpr->aDoclist[pExpr->nDoclist];
  while( pCsr<pEnd ){
    while( *pCsr++ & 0x80 );      /* Skip past docid. */
    fts3LoadColumnlistCounts(&pCsr, &p->aMatchinfo[iStart], 1);
  }


  return SQLITE_OK;
}

/*
** fts3ExprIterate() callback used to collect the "local" matchinfo stats
** for a single query. The "local" stats are those elements of the matchinfo
** array that are different for each row returned by the query.
*/
static int fts3ExprLocalMatchinfoCb(
  Fts3Expr *pExpr,                /* Phrase expression node */
  int iPhrase,                    /* Phrase number */
  void *pCtx                      /* Pointer to MatchInfo structure */
){
  MatchInfo *p = (MatchInfo *)pCtx;


  if( pExpr->aDoclist ){
    char *pCsr;
    int iStart = 2 + (iPhrase * p->nCol * 3);
    int i;

    for(i=0; i<p->nCol; i++) p->aMatchinfo[iStart+i*3] = 0;

    pCsr = sqlite3Fts3FindPositions(pExpr, p->pCursor->iPrevId, -1);
    if( pCsr ){
      fts3LoadColumnlistCounts(&pCsr, &p->aMatchinfo[iStart], 0);
    }
  }

  return SQLITE_OK;
}

/*
** Populate pCsr->aMatchinfo[] with data for the current row. The 
** 'matchinfo' data is an array of 32-bit unsigned integers (C type u32).
*/
static int fts3GetMatchinfo(Fts3Cursor *pCsr){
  MatchInfo sInfo;
  Fts3Table *pTab = (Fts3Table *)pCsr->base.pVtab;

  sInfo.pCursor = pCsr;
  sInfo.nCol = pTab->nColumn;

  if( pCsr->aMatchinfo==0 ){
    /* If Fts3Cursor.aMatchinfo[] is NULL, then this is the first time the
    ** matchinfo function has been called for this query. In this case 
    ** allocate the array used to accumulate the matchinfo data and
    ** initialize those elements that are constant for every row.
    */
    int rc;                       /* Return Code */
    int nPhrase;                  /* Number of phrases */
    int nMatchinfo;               /* Number of u32 elements in match-info */



    /* Load doclists for each phrase in the query. */
    rc = fts3ExprLoadDoclists(pCsr, &nPhrase, 0);
    if( rc!=SQLITE_OK ){
      return rc;
    }

    nMatchinfo = 2 + 3*sInfo.nCol*nPhrase;


    sInfo.aMatchinfo = (u32 *)sqlite3_malloc(sizeof(u32)*nMatchinfo);
    if( !sInfo.aMatchinfo ){ 
      return SQLITE_NOMEM;
    }
    memset(sInfo.aMatchinfo, 0, sizeof(u32)*nMatchinfo);


    /* First element of match-info is the number of phrases in the query */
    sInfo.aMatchinfo[0] = nPhrase;
    sInfo.aMatchinfo[1] = sInfo.nCol;
    (void)fts3ExprIterate(pCsr->pExpr, fts3ExprGlobalMatchinfoCb,(void*)&sInfo);

    pCsr->aMatchinfo = sInfo.aMatchinfo;
  }





  sInfo.aMatchinfo = pCsr->aMatchinfo;

  if( pCsr->isMatchinfoOk ){
    (void)fts3ExprIterate(pCsr->pExpr, fts3ExprLocalMatchinfoCb, (void*)&sInfo);
    pCsr->isMatchinfoOk = 0;
  }

  return SQLITE_OK;
}

/*
** Implementation of snippet() function.
*/
void sqlite3Fts3Snippet(
  sqlite3_context *pCtx,          /* SQLite function call context */
  Fts3Cursor *pCsr,               /* Cursor object */
  const char *zStart,             /* Snippet start text - "<b>" */
  const char *zEnd,               /* Snippet end text - "</b>" */
  const char *zEllipsis,          /* Snippet ellipsis text - "<b>...</b>" */
  int iCol,                       /* Extract snippet from this column */
  int nToken                      /* Approximate number of tokens in snippet */
){
  Fts3Table *pTab = (Fts3Table *)pCsr->base.pVtab;
  int rc = SQLITE_OK;
  int i;
  StrBuffer res = {0, 0, 0};

  /* The returned text includes up to four fragments of text extracted from
  ** the data in the current row. The first iteration of the for(...) loop
  ** below attempts to locate a single fragment of text nToken tokens in 
  ** size that contains at least one instance of all phrases in the query
  ** expression that appear in the current row. If such a fragment of text
  ** cannot be found, the second iteration of the loop attempts to locate
  ** a pair of fragments, and so on.
  */
  int nSnippet = 0;               /* Number of fragments in this snippet */
  SnippetFragment aSnippet[4];    /* Maximum of 4 fragments per snippet */
  int nFToken = -1;               /* Number of tokens in each fragment */

  for(nSnippet=1; 1; nSnippet++){

    int iSnip;                    /* Loop counter 0..nSnippet-1 */
    u64 mCovered = 0;             /* Bitmask of phrases covered by snippet */
    u64 mSeen = 0;                /* Bitmask of phrases seen by BestSnippet() */

    if( nToken>=0 ){
      nFToken = (nToken+nSnippet-1) / nSnippet;
    }else{
      nFToken = -1 * nToken;
    }

    for(iSnip=0; iSnip<nSnippet; iSnip++){
      int iBestScore = -1;        /* Best score of columns checked so far */
      int iRead;                  /* Used to iterate through columns */
      SnippetFragment *pFragment = &aSnippet[iSnip];

      memset(pFragment, 0, sizeof(*pFragment));

      /* Loop through all columns of the table being considered for snippets.
      ** If the iCol argument to this function was negative, this means all
      ** columns of the FTS3 table. Otherwise, only column iCol is considered.
      */
      for(iRead=0; iRead<pTab->nColumn; iRead++){
        SnippetFragment sF;
        int iS;
        if( iCol>=0 && iRead!=iCol ) continue;

        /* Find the best snippet of nFToken tokens in column iRead. */
        rc = fts3BestSnippet(nFToken, pCsr, iRead, mCovered, &mSeen, &sF, &iS);
        if( rc!=SQLITE_OK ){
          goto snippet_out;
        }
        if( iS>iBestScore ){
          *pFragment = sF;
          iBestScore = iS;
        }
      }

      mCovered |= pFragment->covered;
    }

    /* If all query phrases seen by fts3BestSnippet() are present in at least
    ** one of the nSnippet snippet fragments, break out of the loop.
    */
    assert( (mCovered&mSeen)==mCovered );
    if( mSeen==mCovered || nSnippet==SizeofArray(aSnippet) ) break;
  }

  assert( nFToken>0 );

  for(i=0; i<nSnippet && rc==SQLITE_OK; i++){
    rc = fts3SnippetText(pCsr, &aSnippet[i], 
        i, (i==nSnippet-1), nFToken, zStart, zEnd, zEllipsis, &res
    );
  }

 snippet_out:
  if( rc!=SQLITE_OK ){
    sqlite3_result_error_code(pCtx, rc);
    sqlite3_free(res.z);
  }else{
    sqlite3_result_text(pCtx, res.z, -1, sqlite3_free);
  }
}


typedef struct TermOffset TermOffset;
typedef struct TermOffsetCtx TermOffsetCtx;

struct TermOffset {
  char *pList;                    /* Position-list */
  int iPos;                       /* Position just read from pList */
  int iOff;                       /* Offset of this term from read positions */
};

struct TermOffsetCtx {
  int iCol;                       /* Column of table to populate aTerm for */
  int iTerm;
  sqlite3_int64 iDocid;
  TermOffset *aTerm;
};

/*
** This function is an fts3ExprIterate() callback used by sqlite3Fts3Offsets().
*/
static int fts3ExprTermOffsetInit(Fts3Expr *pExpr, int iPhrase, void *ctx){
  TermOffsetCtx *p = (TermOffsetCtx *)ctx;
  int nTerm;                      /* Number of tokens in phrase */
  int iTerm;                      /* For looping through nTerm phrase terms */
  char *pList;                    /* Pointer to position list for phrase */
  int iPos = 0;                   /* First position in position-list */

  pList = sqlite3Fts3FindPositions(pExpr, p->iDocid, p->iCol);
  nTerm = pExpr->pPhrase->nToken;
  if( pList ){
    fts3GetDeltaPosition(&pList, &iPos);
    assert( iPos>=0 );
  }

  for(iTerm=0; iTerm<nTerm; iTerm++){
    TermOffset *pT = &p->aTerm[p->iTerm++];
    pT->iOff = nTerm-iTerm-1;
    pT->pList = pList;
    pT->iPos = iPos;
  }

  return SQLITE_OK;
}

/*
** Implementation of offsets() function.
*/
void sqlite3Fts3Offsets(
  sqlite3_context *pCtx,          /* SQLite function call context */
  Fts3Cursor *pCsr                /* Cursor object */
){
  Fts3Table *pTab = (Fts3Table *)pCsr->base.pVtab;
  sqlite3_tokenizer_module const *pMod = pTab->pTokenizer->pModule;
  const char *ZDUMMY;             /* Dummy argument used with xNext() */
  int NDUMMY;                     /* Dummy argument used with xNext() */
  int rc;                         /* Return Code */
  int nToken;                     /* Number of tokens in query */
  int iCol;                       /* Column currently being processed */
  StrBuffer res = {0, 0, 0};      /* Result string */
  TermOffsetCtx sCtx;             /* Context for fts3ExprTermOffsetInit() */

  memset(&sCtx, 0, sizeof(sCtx));
  assert( pCsr->isRequireSeek==0 );

  /* Count the number of terms in the query */
  rc = fts3ExprLoadDoclists(pCsr, 0, &nToken);
  if( rc!=SQLITE_OK ) goto offsets_out;

  /* Allocate the array of TermOffset iterators. */
  sCtx.aTerm = (TermOffset *)sqlite3_malloc(sizeof(TermOffset)*nToken);
  if( 0==sCtx.aTerm ){
    rc = SQLITE_NOMEM;
    goto offsets_out;
  }
  sCtx.iDocid = pCsr->iPrevId;

  /* Loop through the table columns, appending offset information to 
  ** string-buffer res for each column.
  */
  for(iCol=0; iCol<pTab->nColumn; iCol++){
    sqlite3_tokenizer_cursor *pC; /* Tokenizer cursor */
    int iStart;
    int iEnd;
    int iCurrent;
  const char *zDoc;
    int nDoc;

    /* Initialize the contents of sCtx.aTerm[] for column iCol. There is 
    ** no way that this operation can fail, so the return code from
    ** fts3ExprIterate() can be discarded.
    */
    sCtx.iCol = iCol;
    sCtx.iTerm = 0;
    (void)fts3ExprIterate(pCsr->pExpr, fts3ExprTermOffsetInit, (void *)&sCtx);

    /* Retreive the text stored in column iCol. If an SQL NULL is stored 
    ** in column iCol, jump immediately to the next iteration of the loop.
    ** If an OOM occurs while retrieving the data (this can happen if SQLite
    ** needs to transform the data from utf-16 to utf-8), return SQLITE_NOMEM 
    ** to the caller. 
    */
    zDoc = (const char *)sqlite3_column_text(pCsr->pStmt, iCol+1);
  nDoc = sqlite3_column_bytes(pCsr->pStmt, iCol+1);
    if( zDoc==0 ){
      if( sqlite3_column_type(pCsr->pStmt, iCol+1)==SQLITE_NULL ){
        continue;
      }
      rc = SQLITE_NOMEM;
      goto offsets_out;
    }

    /* Initialize a tokenizer iterator to iterate through column iCol. */
    rc = pMod->xOpen(pTab->pTokenizer, zDoc, nDoc, &pC);
    if( rc!=SQLITE_OK ) goto offsets_out;
    pC->pTokenizer = pTab->pTokenizer;

    rc = pMod->xNext(pC, &ZDUMMY, &NDUMMY, &iStart, &iEnd, &iCurrent);
    while( rc==SQLITE_OK ){
      int i;                      /* Used to loop through terms */
      int iMinPos = 0x7FFFFFFF;   /* Position of next token */
      TermOffset *pTerm = 0;      /* TermOffset associated with next token */

      for(i=0; i<nToken; i++){
        TermOffset *pT = &sCtx.aTerm[i];
        if( pT->pList && (pT->iPos-pT->iOff)<iMinPos ){
          iMinPos = pT->iPos-pT->iOff;
          pTerm = pT;
        }
      }

      if( !pTerm ){
        /* All offsets for this column have been gathered. */
        break;
      }else{
        assert( iCurrent<=iMinPos );
        if( 0==(0xFE&*pTerm->pList) ){
          pTerm->pList = 0;
        }else{
          fts3GetDeltaPosition(&pTerm->pList, &pTerm->iPos);
        }
        while( rc==SQLITE_OK && iCurrent<iMinPos ){
          rc = pMod->xNext(pC, &ZDUMMY, &NDUMMY, &iStart, &iEnd, &iCurrent);
        }
  if( rc==SQLITE_OK ){
          char aBuffer[64];
          sqlite3_snprintf(sizeof(aBuffer), aBuffer, 
              "%d %d %d %d ", iCol, pTerm-sCtx.aTerm, iStart, iEnd-iStart
          );
          rc = fts3StringAppend(&res, aBuffer, -1);
        }
      }
    }
    if( rc==SQLITE_DONE ){
      rc = SQLITE_CORRUPT;
    }

    pMod->xClose(pC);
    if( rc!=SQLITE_OK ) goto offsets_out;
  }

 offsets_out:
  sqlite3_free(sCtx.aTerm);
  assert( rc!=SQLITE_DONE );
  if( rc!=SQLITE_OK ){
    sqlite3_result_error_code(pCtx, rc);
    sqlite3_free(res.z);
  }else{
    sqlite3_result_text(pCtx, res.z, res.n-1, sqlite3_free);
  }
  return;
}

/*
** Implementation of matchinfo() function.
*/
void sqlite3Fts3Matchinfo(sqlite3_context *pContext, Fts3Cursor *pCsr){
  int rc = fts3GetMatchinfo(pCsr);
  if( rc!=SQLITE_OK ){
    sqlite3_result_error_code(pContext, rc);
  }else{
    int n = sizeof(u32)*(2+pCsr->aMatchinfo[0]*pCsr->aMatchinfo[1]*3);
    sqlite3_result_blob(pContext, pCsr->aMatchinfo, n, SQLITE_TRANSIENT);
  }
}

#endif
Changes to src/date.c.
1088
1089
1090
1091
1092
1093
1094
1095
1096
1097
1098
1099
1100
1101
1102
1103
    FUNCTION(datetime,         -1, 0, 0, datetimeFunc  ),
    FUNCTION(strftime,         -1, 0, 0, strftimeFunc  ),
    FUNCTION(current_time,      0, 0, 0, ctimeFunc     ),
    FUNCTION(current_timestamp, 0, 0, 0, ctimestampFunc),
    FUNCTION(current_date,      0, 0, 0, cdateFunc     ),
#else
    STR_FUNCTION(current_time,      0, "%H:%M:%S",          0, currentTimeFunc),
    STR_FUNCTION(current_timestamp, 0, "%Y-%m-%d",          0, currentTimeFunc),
    STR_FUNCTION(current_date,      0, "%Y-%m-%d %H:%M:%S", 0, currentTimeFunc),
#endif
  };
  int i;
  FuncDefHash *pHash = &GLOBAL(FuncDefHash, sqlite3GlobalFunctions);
  FuncDef *aFunc = (FuncDef*)&GLOBAL(FuncDef, aDateTimeFuncs);

  for(i=0; i<ArraySize(aDateTimeFuncs); i++){







|
|







1088
1089
1090
1091
1092
1093
1094
1095
1096
1097
1098
1099
1100
1101
1102
1103
    FUNCTION(datetime,         -1, 0, 0, datetimeFunc  ),
    FUNCTION(strftime,         -1, 0, 0, strftimeFunc  ),
    FUNCTION(current_time,      0, 0, 0, ctimeFunc     ),
    FUNCTION(current_timestamp, 0, 0, 0, ctimestampFunc),
    FUNCTION(current_date,      0, 0, 0, cdateFunc     ),
#else
    STR_FUNCTION(current_time,      0, "%H:%M:%S",          0, currentTimeFunc),
    STR_FUNCTION(current_date,      0, "%Y-%m-%d",          0, currentTimeFunc),
    STR_FUNCTION(current_timestamp, 0, "%Y-%m-%d %H:%M:%S", 0, currentTimeFunc),
#endif
  };
  int i;
  FuncDefHash *pHash = &GLOBAL(FuncDefHash, sqlite3GlobalFunctions);
  FuncDef *aFunc = (FuncDef*)&GLOBAL(FuncDef, aDateTimeFuncs);

  for(i=0; i<ArraySize(aDateTimeFuncs); i++){
Changes to src/expr.c.
1878
1879
1880
1881
1882
1883
1884

1885
1886
1887
1888
1889
1890
1891
1892
1893
1894
1895
1896
1897
1898
1899
1900
1901
1902
1903

1904
1905
1906
1907
1908
1909
1910
1911
1912
1913
1914

1915
1916
1917
1918
1919
1920
1921
1922
1923
1924
1925
1926
1927
1928
1929



1930

1931
1932
1933
1934
1935
1936
1937
  char *out = sqlite3DbMallocRaw(sqlite3VdbeDb(v), 8);
  if( out ){
    memcpy(out, in, 8);
  }
  return out;
}


/*
** Generate an instruction that will put the floating point
** value described by z[0..n-1] into register iMem.
**
** The z[] string will probably not be zero-terminated.  But the 
** z[n] character is guaranteed to be something that does not look
** like the continuation of the number.
*/
static void codeReal(Vdbe *v, const char *z, int negateFlag, int iMem){
  if( ALWAYS(z!=0) ){
    double value;
    char *zV;
    sqlite3AtoF(z, &value);
    assert( !sqlite3IsNaN(value) ); /* The new AtoF never returns NaN */
    if( negateFlag ) value = -value;
    zV = dup8bytes(v, (char*)&value);
    sqlite3VdbeAddOp4(v, OP_Real, 0, iMem, 0, zV, P4_REAL);
  }
}



/*
** Generate an instruction that will put the integer describe by
** text z[0..n-1] into register iMem.
**
** The z[] string will probably not be zero-terminated.  But the 
** z[n] character is guaranteed to be something that does not look
** like the continuation of the number.
*/
static void codeInteger(Vdbe *v, Expr *pExpr, int negFlag, int iMem){

  if( pExpr->flags & EP_IntValue ){
    int i = pExpr->u.iValue;
    if( negFlag ) i = -i;
    sqlite3VdbeAddOp2(v, OP_Integer, i, iMem);
  }else{
    const char *z = pExpr->u.zToken;
    assert( z!=0 );
    if( sqlite3FitsIn64Bits(z, negFlag) ){
      i64 value;
      char *zV;
      sqlite3Atoi64(z, &value);
      if( negFlag ) value = -value;
      zV = dup8bytes(v, (char*)&value);
      sqlite3VdbeAddOp4(v, OP_Int64, 0, iMem, 0, zV, P4_INT64);
    }else{



      codeReal(v, z, negFlag, iMem);

    }
  }
}

/*
** Clear a cache entry.
*/







>



















>










|
>















>
>
>

>







1878
1879
1880
1881
1882
1883
1884
1885
1886
1887
1888
1889
1890
1891
1892
1893
1894
1895
1896
1897
1898
1899
1900
1901
1902
1903
1904
1905
1906
1907
1908
1909
1910
1911
1912
1913
1914
1915
1916
1917
1918
1919
1920
1921
1922
1923
1924
1925
1926
1927
1928
1929
1930
1931
1932
1933
1934
1935
1936
1937
1938
1939
1940
1941
1942
1943
1944
  char *out = sqlite3DbMallocRaw(sqlite3VdbeDb(v), 8);
  if( out ){
    memcpy(out, in, 8);
  }
  return out;
}

#ifndef SQLITE_OMIT_FLOATING_POINT
/*
** Generate an instruction that will put the floating point
** value described by z[0..n-1] into register iMem.
**
** The z[] string will probably not be zero-terminated.  But the 
** z[n] character is guaranteed to be something that does not look
** like the continuation of the number.
*/
static void codeReal(Vdbe *v, const char *z, int negateFlag, int iMem){
  if( ALWAYS(z!=0) ){
    double value;
    char *zV;
    sqlite3AtoF(z, &value);
    assert( !sqlite3IsNaN(value) ); /* The new AtoF never returns NaN */
    if( negateFlag ) value = -value;
    zV = dup8bytes(v, (char*)&value);
    sqlite3VdbeAddOp4(v, OP_Real, 0, iMem, 0, zV, P4_REAL);
  }
}
#endif


/*
** Generate an instruction that will put the integer describe by
** text z[0..n-1] into register iMem.
**
** The z[] string will probably not be zero-terminated.  But the 
** z[n] character is guaranteed to be something that does not look
** like the continuation of the number.
*/
static void codeInteger(Parse *pParse, Expr *pExpr, int negFlag, int iMem){
  Vdbe *v = pParse->pVdbe;
  if( pExpr->flags & EP_IntValue ){
    int i = pExpr->u.iValue;
    if( negFlag ) i = -i;
    sqlite3VdbeAddOp2(v, OP_Integer, i, iMem);
  }else{
    const char *z = pExpr->u.zToken;
    assert( z!=0 );
    if( sqlite3FitsIn64Bits(z, negFlag) ){
      i64 value;
      char *zV;
      sqlite3Atoi64(z, &value);
      if( negFlag ) value = -value;
      zV = dup8bytes(v, (char*)&value);
      sqlite3VdbeAddOp4(v, OP_Int64, 0, iMem, 0, zV, P4_INT64);
    }else{
#ifdef SQLITE_OMIT_FLOATING_POINT
      sqlite3ErrorMsg(pParse, "oversized integer: %s%s", negFlag ? "-" : "", z);
#else
      codeReal(v, z, negFlag, iMem);
#endif
    }
  }
}

/*
** Clear a cache entry.
*/
2310
2311
2312
2313
2314
2315
2316
2317
2318
2319

2320
2321
2322
2323
2324

2325
2326
2327
2328
2329
2330
2331
      }else{
        inReg = sqlite3ExprCodeGetColumn(pParse, pExpr->pTab,
                                 pExpr->iColumn, pExpr->iTable, target);
      }
      break;
    }
    case TK_INTEGER: {
      codeInteger(v, pExpr, 0, target);
      break;
    }

    case TK_FLOAT: {
      assert( !ExprHasProperty(pExpr, EP_IntValue) );
      codeReal(v, pExpr->u.zToken, 0, target);
      break;
    }

    case TK_STRING: {
      assert( !ExprHasProperty(pExpr, EP_IntValue) );
      sqlite3VdbeAddOp4(v, OP_String8, 0, target, 0, pExpr->u.zToken, 0);
      break;
    }
    case TK_NULL: {
      sqlite3VdbeAddOp2(v, OP_Null, 0, target);







|


>





>







2317
2318
2319
2320
2321
2322
2323
2324
2325
2326
2327
2328
2329
2330
2331
2332
2333
2334
2335
2336
2337
2338
2339
2340
      }else{
        inReg = sqlite3ExprCodeGetColumn(pParse, pExpr->pTab,
                                 pExpr->iColumn, pExpr->iTable, target);
      }
      break;
    }
    case TK_INTEGER: {
      codeInteger(pParse, pExpr, 0, target);
      break;
    }
#ifndef SQLITE_OMIT_FLOATING_POINT
    case TK_FLOAT: {
      assert( !ExprHasProperty(pExpr, EP_IntValue) );
      codeReal(v, pExpr->u.zToken, 0, target);
      break;
    }
#endif
    case TK_STRING: {
      assert( !ExprHasProperty(pExpr, EP_IntValue) );
      sqlite3VdbeAddOp4(v, OP_String8, 0, target, 0, pExpr->u.zToken, 0);
      break;
    }
    case TK_NULL: {
      sqlite3VdbeAddOp2(v, OP_Null, 0, target);
2487
2488
2489
2490
2491
2492
2493
2494



2495
2496
2497
2498

2499
2500
2501
2502
2503
2504
2505
      testcase( regFree1==0 );
      testcase( regFree2==0 );
      break;
    }
    case TK_UMINUS: {
      Expr *pLeft = pExpr->pLeft;
      assert( pLeft );
      if( pLeft->op==TK_FLOAT ){



        assert( !ExprHasProperty(pExpr, EP_IntValue) );
        codeReal(v, pLeft->u.zToken, 1, target);
      }else if( pLeft->op==TK_INTEGER ){
        codeInteger(v, pLeft, 1, target);

      }else{
        regFree1 = r1 = sqlite3GetTempReg(pParse);
        sqlite3VdbeAddOp2(v, OP_Integer, 0, r1);
        r2 = sqlite3ExprCodeTemp(pParse, pExpr->pLeft, &regFree2);
        sqlite3VdbeAddOp3(v, OP_Subtract, r2, r1, target);
        testcase( regFree2==0 );
      }







|
>
>
>


<
<
>







2496
2497
2498
2499
2500
2501
2502
2503
2504
2505
2506
2507
2508


2509
2510
2511
2512
2513
2514
2515
2516
      testcase( regFree1==0 );
      testcase( regFree2==0 );
      break;
    }
    case TK_UMINUS: {
      Expr *pLeft = pExpr->pLeft;
      assert( pLeft );
      if( pLeft->op==TK_INTEGER ){
        codeInteger(pParse, pLeft, 1, target);
#ifndef SQLITE_OMIT_FLOATING_POINT
      }else if( pLeft->op==TK_FLOAT ){
        assert( !ExprHasProperty(pExpr, EP_IntValue) );
        codeReal(v, pLeft->u.zToken, 1, target);


#endif
      }else{
        regFree1 = r1 = sqlite3GetTempReg(pParse);
        sqlite3VdbeAddOp2(v, OP_Integer, 0, r1);
        r2 = sqlite3ExprCodeTemp(pParse, pExpr->pLeft, &regFree2);
        sqlite3VdbeAddOp3(v, OP_Subtract, r2, r1, target);
        testcase( regFree2==0 );
      }
2739
2740
2741
2742
2743
2744
2745

2746
2747
2748
2749
2750
2751
2752

2753
2754
2755
2756
2757
2758
2759
      sqlite3VdbeAddOp2(v, OP_Param, p1, target);
      VdbeComment((v, "%s.%s -> $%d",
        (pExpr->iTable ? "new" : "old"),
        (pExpr->iColumn<0 ? "rowid" : pExpr->pTab->aCol[pExpr->iColumn].zName),
        target
      ));


      /* If the column has REAL affinity, it may currently be stored as an
      ** integer. Use OP_RealAffinity to make sure it is really real.  */
      if( pExpr->iColumn>=0 
       && pTab->aCol[pExpr->iColumn].affinity==SQLITE_AFF_REAL
      ){
        sqlite3VdbeAddOp1(v, OP_RealAffinity, target);
      }

      break;
    }


    /*
    ** Form A:
    **   CASE x WHEN e1 THEN r1 WHEN e2 THEN r2 ... WHEN eN THEN rN ELSE y END







>







>







2750
2751
2752
2753
2754
2755
2756
2757
2758
2759
2760
2761
2762
2763
2764
2765
2766
2767
2768
2769
2770
2771
2772
      sqlite3VdbeAddOp2(v, OP_Param, p1, target);
      VdbeComment((v, "%s.%s -> $%d",
        (pExpr->iTable ? "new" : "old"),
        (pExpr->iColumn<0 ? "rowid" : pExpr->pTab->aCol[pExpr->iColumn].zName),
        target
      ));

#ifndef SQLITE_OMIT_FLOATING_POINT
      /* If the column has REAL affinity, it may currently be stored as an
      ** integer. Use OP_RealAffinity to make sure it is really real.  */
      if( pExpr->iColumn>=0 
       && pTab->aCol[pExpr->iColumn].affinity==SQLITE_AFF_REAL
      ){
        sqlite3VdbeAddOp1(v, OP_RealAffinity, target);
      }
#endif
      break;
    }


    /*
    ** Form A:
    **   CASE x WHEN e1 THEN r1 WHEN e2 THEN r2 ... WHEN eN THEN rN ELSE y END
3411
3412
3413
3414
3415
3416
3417
3418
3419


3420
3421
3422
3423
3424
3425
3426
3427
3428
3429
3430
3431
3432
3433
3434
3435
3436
3437
3438
3439
3440
3441
3442
3443
3444
3445
3446
3447
3448
3449
3450
3451
3452
3453
3454
3455
3456
3457
3458
3459
3460
3461
3462
3463
3464
3465
3466
3467


3468
3469
3470
3471
3472
3473
3474
3475
    }
  }
  sqlite3ReleaseTempReg(pParse, regFree1);
  sqlite3ReleaseTempReg(pParse, regFree2);
}

/*
** Do a deep comparison of two expression trees.  Return TRUE (non-zero)
** if they are identical and return FALSE if they differ in any way.


**
** Sometimes this routine will return FALSE even if the two expressions
** really are equivalent.  If we cannot prove that the expressions are
** identical, we return FALSE just to be safe.  So if this routine
** returns false, then you do not really know for certain if the two
** expressions are the same.  But if you get a TRUE return, then you
** can be sure the expressions are the same.  In the places where
** this routine is used, it does not hurt to get an extra FALSE - that
** just might result in some slightly slower code.  But returning
** an incorrect TRUE could lead to a malfunction.
*/
int sqlite3ExprCompare(Expr *pA, Expr *pB){
  int i;
  if( pA==0||pB==0 ){
    return pB==pA;
  }
  assert( !ExprHasAnyProperty(pA, EP_TokenOnly|EP_Reduced) );
  assert( !ExprHasAnyProperty(pB, EP_TokenOnly|EP_Reduced) );
  if( ExprHasProperty(pA, EP_xIsSelect) || ExprHasProperty(pB, EP_xIsSelect) ){
    return 0;
  }
  if( (pA->flags & EP_Distinct)!=(pB->flags & EP_Distinct) ) return 0;
  if( pA->op!=pB->op ) return 0;
  if( !sqlite3ExprCompare(pA->pLeft, pB->pLeft) ) return 0;
  if( !sqlite3ExprCompare(pA->pRight, pB->pRight) ) return 0;

  if( pA->x.pList && pB->x.pList ){
    if( pA->x.pList->nExpr!=pB->x.pList->nExpr ) return 0;
    for(i=0; i<pA->x.pList->nExpr; i++){
      Expr *pExprA = pA->x.pList->a[i].pExpr;
      Expr *pExprB = pB->x.pList->a[i].pExpr;
      if( !sqlite3ExprCompare(pExprA, pExprB) ) return 0;
    }
  }else if( pA->x.pList || pB->x.pList ){
    return 0;
  }

  if( pA->iTable!=pB->iTable || pA->iColumn!=pB->iColumn ) return 0;
  if( ExprHasProperty(pA, EP_IntValue) ){
    if( !ExprHasProperty(pB, EP_IntValue) || pA->u.iValue!=pB->u.iValue ){
      return 0;
    }
  }else if( pA->op!=TK_COLUMN && pA->u.zToken ){
    if( ExprHasProperty(pB, EP_IntValue) || NEVER(pB->u.zToken==0) ) return 0;
    if( sqlite3StrICmp(pA->u.zToken,pB->u.zToken)!=0 ){
      return 0;
    }
  }


  return 1;
}


/*
** Add a new element to the pAggInfo->aCol[] array.  Return the index of
** the new element.  Return a negative number if malloc fails.
*/







|
|
>
>

|

|
|
|

|

|




|




|

|
|
|
|


|



|


|


|


|


|

|


>
>
|







3424
3425
3426
3427
3428
3429
3430
3431
3432
3433
3434
3435
3436
3437
3438
3439
3440
3441
3442
3443
3444
3445
3446
3447
3448
3449
3450
3451
3452
3453
3454
3455
3456
3457
3458
3459
3460
3461
3462
3463
3464
3465
3466
3467
3468
3469
3470
3471
3472
3473
3474
3475
3476
3477
3478
3479
3480
3481
3482
3483
3484
3485
3486
3487
3488
3489
3490
3491
3492
    }
  }
  sqlite3ReleaseTempReg(pParse, regFree1);
  sqlite3ReleaseTempReg(pParse, regFree2);
}

/*
** Do a deep comparison of two expression trees.  Return 0 if the two
** expressions are completely identical.  Return 1 if they differ only
** by a COLLATE operator at the top level.  Return 2 if there are differences
** other than the top-level COLLATE operator.
**
** Sometimes this routine will return 2 even if the two expressions
** really are equivalent.  If we cannot prove that the expressions are
** identical, we return 2 just to be safe.  So if this routine
** returns 2, then you do not really know for certain if the two
** expressions are the same.  But if you get a 0 or 1 return, then you
** can be sure the expressions are the same.  In the places where
** this routine is used, it does not hurt to get an extra 2 - that
** just might result in some slightly slower code.  But returning
** an incorrect 0 or 1 could lead to a malfunction.
*/
int sqlite3ExprCompare(Expr *pA, Expr *pB){
  int i;
  if( pA==0||pB==0 ){
    return pB==pA ? 0 : 2;
  }
  assert( !ExprHasAnyProperty(pA, EP_TokenOnly|EP_Reduced) );
  assert( !ExprHasAnyProperty(pB, EP_TokenOnly|EP_Reduced) );
  if( ExprHasProperty(pA, EP_xIsSelect) || ExprHasProperty(pB, EP_xIsSelect) ){
    return 2;
  }
  if( (pA->flags & EP_Distinct)!=(pB->flags & EP_Distinct) ) return 2;
  if( pA->op!=pB->op ) return 2;
  if( sqlite3ExprCompare(pA->pLeft, pB->pLeft) ) return 2;
  if( sqlite3ExprCompare(pA->pRight, pB->pRight) ) return 2;

  if( pA->x.pList && pB->x.pList ){
    if( pA->x.pList->nExpr!=pB->x.pList->nExpr ) return 2;
    for(i=0; i<pA->x.pList->nExpr; i++){
      Expr *pExprA = pA->x.pList->a[i].pExpr;
      Expr *pExprB = pB->x.pList->a[i].pExpr;
      if( sqlite3ExprCompare(pExprA, pExprB) ) return 2;
    }
  }else if( pA->x.pList || pB->x.pList ){
    return 2;
  }

  if( pA->iTable!=pB->iTable || pA->iColumn!=pB->iColumn ) return 2;
  if( ExprHasProperty(pA, EP_IntValue) ){
    if( !ExprHasProperty(pB, EP_IntValue) || pA->u.iValue!=pB->u.iValue ){
      return 2;
    }
  }else if( pA->op!=TK_COLUMN && pA->u.zToken ){
    if( ExprHasProperty(pB, EP_IntValue) || NEVER(pB->u.zToken==0) ) return 2;
    if( sqlite3StrICmp(pA->u.zToken,pB->u.zToken)!=0 ){
      return 2;
    }
  }
  if( (pA->flags & EP_ExpCollate)!=(pB->flags & EP_ExpCollate) ) return 1;
  if( (pA->flags & EP_ExpCollate)!=0 && pA->pColl!=pB->pColl ) return 2;
  return 0;
}


/*
** Add a new element to the pAggInfo->aCol[] array.  Return the index of
** the new element.  Return a negative number if malloc fails.
*/
3592
3593
3594
3595
3596
3597
3598
3599
3600
3601
3602
3603
3604
3605
3606
      ** to be ignored */
      if( pNC->nDepth==0 ){
        /* Check to see if pExpr is a duplicate of another aggregate 
        ** function that is already in the pAggInfo structure
        */
        struct AggInfo_func *pItem = pAggInfo->aFunc;
        for(i=0; i<pAggInfo->nFunc; i++, pItem++){
          if( sqlite3ExprCompare(pItem->pExpr, pExpr) ){
            break;
          }
        }
        if( i>=pAggInfo->nFunc ){
          /* pExpr is original.  Make a new entry in pAggInfo->aFunc[]
          */
          u8 enc = ENC(pParse->db);







|







3609
3610
3611
3612
3613
3614
3615
3616
3617
3618
3619
3620
3621
3622
3623
      ** to be ignored */
      if( pNC->nDepth==0 ){
        /* Check to see if pExpr is a duplicate of another aggregate 
        ** function that is already in the pAggInfo structure
        */
        struct AggInfo_func *pItem = pAggInfo->aFunc;
        for(i=0; i<pAggInfo->nFunc; i++, pItem++){
          if( sqlite3ExprCompare(pItem->pExpr, pExpr)==0 ){
            break;
          }
        }
        if( i>=pAggInfo->nFunc ){
          /* pExpr is original.  Make a new entry in pAggInfo->aFunc[]
          */
          u8 enc = ENC(pParse->db);
Changes to src/func.c.
439
440
441
442
443
444
445



446
447
448
449
450


451

452
453
454
455
456
457
458
static void last_insert_rowid(
  sqlite3_context *context, 
  int NotUsed, 
  sqlite3_value **NotUsed2
){
  sqlite3 *db = sqlite3_context_db_handle(context);
  UNUSED_PARAMETER2(NotUsed, NotUsed2);



  sqlite3_result_int64(context, sqlite3_last_insert_rowid(db));
}

/*
** Implementation of the changes() SQL function.  The return value is the


** same as the sqlite3_changes() API function.

*/
static void changes(
  sqlite3_context *context,
  int NotUsed,
  sqlite3_value **NotUsed2
){
  sqlite3 *db = sqlite3_context_db_handle(context);







>
>
>




|
>
>
|
>







439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
static void last_insert_rowid(
  sqlite3_context *context, 
  int NotUsed, 
  sqlite3_value **NotUsed2
){
  sqlite3 *db = sqlite3_context_db_handle(context);
  UNUSED_PARAMETER2(NotUsed, NotUsed2);
  /* IMP: R-51513-12026 The last_insert_rowid() SQL function is a
  ** wrapper around the sqlite3_last_insert_rowid() C/C++ interface
  ** function. */
  sqlite3_result_int64(context, sqlite3_last_insert_rowid(db));
}

/*
** Implementation of the changes() SQL function.
**
** IMP: R-62073-11209 The changes() SQL function is a wrapper
** around the sqlite3_changes() C/C++ function and hence follows the same
** rules for counting changes.
*/
static void changes(
  sqlite3_context *context,
  int NotUsed,
  sqlite3_value **NotUsed2
){
  sqlite3 *db = sqlite3_context_db_handle(context);
467
468
469
470
471
472
473


474
475
476
477
478
479
480
static void total_changes(
  sqlite3_context *context,
  int NotUsed,
  sqlite3_value **NotUsed2
){
  sqlite3 *db = sqlite3_context_db_handle(context);
  UNUSED_PARAMETER2(NotUsed, NotUsed2);


  sqlite3_result_int(context, sqlite3_total_changes(db));
}

/*
** A structure defining how to do GLOB-style comparisons.
*/
struct compareInfo {







>
>







473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
static void total_changes(
  sqlite3_context *context,
  int NotUsed,
  sqlite3_value **NotUsed2
){
  sqlite3 *db = sqlite3_context_db_handle(context);
  UNUSED_PARAMETER2(NotUsed, NotUsed2);
  /* IMP: R-52756-41993 This function is a wrapper around the
  ** sqlite3_total_changes() C/C++ interface. */
  sqlite3_result_int(context, sqlite3_total_changes(db));
}

/*
** A structure defining how to do GLOB-style comparisons.
*/
struct compareInfo {
734
735
736
737
738
739
740


741
742
743
744
745
746
747
748
749
750
751
752
753
754


755
756
757
758
759
760
761
762
*/
static void versionFunc(
  sqlite3_context *context,
  int NotUsed,
  sqlite3_value **NotUsed2
){
  UNUSED_PARAMETER2(NotUsed, NotUsed2);


  sqlite3_result_text(context, sqlite3_version, -1, SQLITE_STATIC);
}

/*
** Implementation of the sqlite_source_id() function. The result is a string
** that identifies the particular version of the source code used to build
** SQLite.
*/
static void sourceidFunc(
  sqlite3_context *context,
  int NotUsed,
  sqlite3_value **NotUsed2
){
  UNUSED_PARAMETER2(NotUsed, NotUsed2);


  sqlite3_result_text(context, SQLITE_SOURCE_ID, -1, SQLITE_STATIC);
}

/* Array for converting from half-bytes (nybbles) into ASCII hex
** digits. */
static const char hexdigits[] = {
  '0', '1', '2', '3', '4', '5', '6', '7',
  '8', '9', 'A', 'B', 'C', 'D', 'E', 'F' 







>
>
|













>
>
|







742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
*/
static void versionFunc(
  sqlite3_context *context,
  int NotUsed,
  sqlite3_value **NotUsed2
){
  UNUSED_PARAMETER2(NotUsed, NotUsed2);
  /* IMP: R-48699-48617 This function is an SQL wrapper around the
  ** sqlite3_libversion() C-interface. */
  sqlite3_result_text(context, sqlite3_libversion(), -1, SQLITE_STATIC);
}

/*
** Implementation of the sqlite_source_id() function. The result is a string
** that identifies the particular version of the source code used to build
** SQLite.
*/
static void sourceidFunc(
  sqlite3_context *context,
  int NotUsed,
  sqlite3_value **NotUsed2
){
  UNUSED_PARAMETER2(NotUsed, NotUsed2);
  /* IMP: R-24470-31136 This function is an SQL wrapper around the
  ** sqlite3_sourceid() C interface. */
  sqlite3_result_text(context, sqlite3_sourceid(), -1, SQLITE_STATIC);
}

/* Array for converting from half-bytes (nybbles) into ASCII hex
** digits. */
static const char hexdigits[] = {
  '0', '1', '2', '3', '4', '5', '6', '7',
  '8', '9', 'A', 'B', 'C', 'D', 'E', 'F' 
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
  UNUSED_PARAMETER(argc);
  n = sqlite3_value_int64(argv[0]);
  testcase( n==db->aLimit[SQLITE_LIMIT_LENGTH] );
  testcase( n==db->aLimit[SQLITE_LIMIT_LENGTH]+1 );
  if( n>db->aLimit[SQLITE_LIMIT_LENGTH] ){
    sqlite3_result_error_toobig(context);
  }else{
    sqlite3_result_zeroblob(context, (int)n);
  }
}

/*
** The replace() function.  Three arguments are all strings: call
** them A, B, and C. The result is also a string which is derived
** from A by replacing every occurance of B with C.  The match







|







889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
  UNUSED_PARAMETER(argc);
  n = sqlite3_value_int64(argv[0]);
  testcase( n==db->aLimit[SQLITE_LIMIT_LENGTH] );
  testcase( n==db->aLimit[SQLITE_LIMIT_LENGTH]+1 );
  if( n>db->aLimit[SQLITE_LIMIT_LENGTH] ){
    sqlite3_result_error_toobig(context);
  }else{
    sqlite3_result_zeroblob(context, (int)n); /* IMP: R-00293-64994 */
  }
}

/*
** The replace() function.  Three arguments are all strings: call
** them A, B, and C. The result is also a string which is derived
** from A by replacing every occurance of B with C.  The match
Changes to src/insert.c.
1711
1712
1713
1714
1715
1716
1717
1718
1719
1720
1721
1722
1723
1724
1725
      if( xferCompatibleIndex(pDestIdx, pSrcIdx) ) break;
    }
    if( pSrcIdx==0 ){
      return 0;    /* pDestIdx has no corresponding index in pSrc */
    }
  }
#ifndef SQLITE_OMIT_CHECK
  if( pDest->pCheck && !sqlite3ExprCompare(pSrc->pCheck, pDest->pCheck) ){
    return 0;   /* Tables have different CHECK constraints.  Ticket #2252 */
  }
#endif

  /* If we get this far, it means either:
  **
  **    *   We can always do the transfer if the table contains an







|







1711
1712
1713
1714
1715
1716
1717
1718
1719
1720
1721
1722
1723
1724
1725
      if( xferCompatibleIndex(pDestIdx, pSrcIdx) ) break;
    }
    if( pSrcIdx==0 ){
      return 0;    /* pDestIdx has no corresponding index in pSrc */
    }
  }
#ifndef SQLITE_OMIT_CHECK
  if( pDest->pCheck && sqlite3ExprCompare(pSrc->pCheck, pDest->pCheck) ){
    return 0;   /* Tables have different CHECK constraints.  Ticket #2252 */
  }
#endif

  /* If we get this far, it means either:
  **
  **    *   We can always do the transfer if the table contains an
Changes to src/legacy.c.
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
  int rc = SQLITE_OK;         /* Return code */
  const char *zLeftover;      /* Tail of unprocessed SQL */
  sqlite3_stmt *pStmt = 0;    /* The current SQL statement */
  char **azCols = 0;          /* Names of result columns */
  int nRetry = 0;             /* Number of retry attempts */
  int callbackIsInit;         /* True if callback data is initialized */

  if (!sqlite3SafetyCheckOk(db)) {
    return SQLITE_MISUSE;
  }
  	
  if( zSql==0 ) zSql = "";
#ifdef SQLITE_ENABLE_SQLRR
  SRRecExec(db, zSql);
#endif  
  sqlite3_mutex_enter(db->mutex);
  sqlite3Error(db, SQLITE_OK, 0);
  while( (rc==SQLITE_OK || (rc==SQLITE_SCHEMA && (++nRetry)<2)) && zSql[0] ){







|
<
<
<







40
41
42
43
44
45
46
47



48
49
50
51
52
53
54
  int rc = SQLITE_OK;         /* Return code */
  const char *zLeftover;      /* Tail of unprocessed SQL */
  sqlite3_stmt *pStmt = 0;    /* The current SQL statement */
  char **azCols = 0;          /* Names of result columns */
  int nRetry = 0;             /* Number of retry attempts */
  int callbackIsInit;         /* True if callback data is initialized */

  if( !sqlite3SafetyCheckOk(db) ) return SQLITE_MISUSE;



  if( zSql==0 ) zSql = "";
#ifdef SQLITE_ENABLE_SQLRR
  SRRecExec(db, zSql);
#endif  
  sqlite3_mutex_enter(db->mutex);
  sqlite3Error(db, SQLITE_OK, 0);
  while( (rc==SQLITE_OK || (rc==SQLITE_SCHEMA && (++nRetry)<2)) && zSql[0] ){
Changes to src/prepare.c.
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328

  /* Read the schema information out of the schema tables
  */
  assert( db->init.busy );
  {
    char *zSql;
    zSql = sqlite3MPrintf(db, 
        "SELECT name, rootpage, sql FROM '%q'.%s",
        db->aDb[iDb].zName, zMasterName);
    (void)sqlite3SafetyOff(db);
#ifndef SQLITE_OMIT_AUTHORIZATION
    {
      int (*xAuth)(void*,int,const char*,const char*,const char*,const char*);
      xAuth = db->xAuth;
      db->xAuth = 0;







|







314
315
316
317
318
319
320
321
322
323
324
325
326
327
328

  /* Read the schema information out of the schema tables
  */
  assert( db->init.busy );
  {
    char *zSql;
    zSql = sqlite3MPrintf(db, 
        "SELECT name, rootpage, sql FROM '%q'.%s ORDER BY rowid",
        db->aDb[iDb].zName, zMasterName);
    (void)sqlite3SafetyOff(db);
#ifndef SQLITE_OMIT_AUTHORIZATION
    {
      int (*xAuth)(void*,int,const char*,const char*,const char*,const char*);
      xAuth = db->xAuth;
      db->xAuth = 0;
Changes to src/printf.c.
602
603
604
605
606
607
608


609
610
611
612
613
614
615
616
          for(i=width; i>=nPad; i--){
            bufpt[i] = bufpt[i-nPad];
          }
          i = prefix!=0;
          while( nPad-- ) bufpt[i++] = '0';
          length = width;
        }


#endif
        break;
      case etSIZE:
        *(va_arg(ap,int*)) = pAccum->nChar;
        length = width = 0;
        break;
      case etPERCENT:
        buf[0] = '%';







>
>
|







602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
          for(i=width; i>=nPad; i--){
            bufpt[i] = bufpt[i-nPad];
          }
          i = prefix!=0;
          while( nPad-- ) bufpt[i++] = '0';
          length = width;
        }
#else
        length = 0;
#endif /* SQLITE_OMIT_FLOATING_POINT */
        break;
      case etSIZE:
        *(va_arg(ap,int*)) = pAccum->nChar;
        length = width = 0;
        break;
      case etPERCENT:
        buf[0] = '%';
Changes to src/resolve.c.
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
  }

  /* Try to match the ORDER BY expression against an expression
  ** in the result set.  Return an 1-based index of the matching
  ** result-set entry.
  */
  for(i=0; i<pEList->nExpr; i++){
    if( sqlite3ExprCompare(pEList->a[i].pExpr, pE) ){
      return i+1;
    }
  }

  /* If no match, return 0. */
  return 0;
}







|







682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
  }

  /* Try to match the ORDER BY expression against an expression
  ** in the result set.  Return an 1-based index of the matching
  ** result-set entry.
  */
  for(i=0; i<pEList->nExpr; i++){
    if( sqlite3ExprCompare(pEList->a[i].pExpr, pE)<2 ){
      return i+1;
    }
  }

  /* If no match, return 0. */
  return 0;
}
Changes to src/shell.c.
1497
1498
1499
1500
1501
1502
1503
1504
1505
1506
1507
1508
1509
1510
1511
1512
1513
1514
** This is the callback routine that the shell
** invokes for each row of a query result.
*/
static int shell_callback(void *pArg, int nArg, char **azArg, char **azCol, int *aiType){
  int i;
  struct callback_data *p = (struct callback_data*)pArg;

  if( p->echoOn && p->cnt==0  && p->pStmt){
    printf("%s\n", sqlite3_sql(p->pStmt));
  }

  switch( p->mode ){
    case MODE_Line: {
      int w = 5;
      if( azArg==0 ) break;
      for(i=0; i<nArg; i++){
        int len = strlen30(azCol[i] ? azCol[i] : "");
        if( len>w ) w = len;







<
<
<
<







1497
1498
1499
1500
1501
1502
1503




1504
1505
1506
1507
1508
1509
1510
** This is the callback routine that the shell
** invokes for each row of a query result.
*/
static int shell_callback(void *pArg, int nArg, char **azArg, char **azCol, int *aiType){
  int i;
  struct callback_data *p = (struct callback_data*)pArg;





  switch( p->mode ){
    case MODE_Line: {
      int w = 5;
      if( azArg==0 ) break;
      for(i=0; i<nArg; i++){
        int len = strlen30(azCol[i] ? azCol[i] : "");
        if( len>w ) w = len;
1850
1851
1852
1853
1854
1855
1856






1857
1858
1859
1860
1861
1862
1863
    }else{
      if( !pStmt ){
        /* this happens for a comment or white-space */
        zSql = zLeftover;
        while( isspace(zSql[0]) ) zSql++;
        continue;
      }







      /* perform the first step.  this will tell us if we
      ** have a result set or not and how wide it is.
      */
      rc = sqlite3_step(pStmt);
      /* if we have a result set... */
      if( SQLITE_ROW == rc ){







>
>
>
>
>
>







1846
1847
1848
1849
1850
1851
1852
1853
1854
1855
1856
1857
1858
1859
1860
1861
1862
1863
1864
1865
    }else{
      if( !pStmt ){
        /* this happens for a comment or white-space */
        zSql = zLeftover;
        while( isspace(zSql[0]) ) zSql++;
        continue;
      }

      /* echo the sql statement if echo on */
      if( pArg->echoOn ){
        char *zStmtSql = sqlite3_sql(pStmt);
        fprintf(pArg->out,"%s\n", zStmtSql ? zStmtSql : zSql);
      }

      /* perform the first step.  this will tell us if we
      ** have a result set or not and how wide it is.
      */
      rc = sqlite3_step(pStmt);
      /* if we have a result set... */
      if( SQLITE_ROW == rc ){
Changes to src/sqliteInt.h.
72
73
74
75
76
77
78





79
80
81
82
83
84
85
#ifdef HAVE_STDINT_H
#include <stdint.h>
#endif
#ifdef HAVE_INTTYPES_H
#include <inttypes.h>
#endif






#define SQLITE_INDEX_SAMPLES 10

/*
** This macro is used to "hide" some ugliness in casting an int
** value to a ptr value under the MSVC 64-bit compiler.   Casting
** non 64-bit values to ptr types results in a "hard" error with 
** the MSVC 64-bit compiler which this attempts to avoid.  







>
>
>
>
>







72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
#ifdef HAVE_STDINT_H
#include <stdint.h>
#endif
#ifdef HAVE_INTTYPES_H
#include <inttypes.h>
#endif

/*
** The number of samples of an index that SQLite takes in order to 
** construct a histogram of the table content when running ANALYZE
** and with SQLITE_ENABLE_STAT2
*/
#define SQLITE_INDEX_SAMPLES 10

/*
** This macro is used to "hide" some ugliness in casting an int
** value to a ptr value under the MSVC 64-bit compiler.   Casting
** non 64-bit values to ptr types results in a "hard" error with 
** the MSVC 64-bit compiler which this attempts to avoid.  
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166

/*
** Exactly one of the following macros must be defined in order to
** specify which memory allocation subsystem to use.
**
**     SQLITE_SYSTEM_MALLOC          // Use normal system malloc()
**     SQLITE_MEMDEBUG               // Debugging version of system malloc()
**     SQLITE_MEMORY_SIZE            // internal allocator #1
**     SQLITE_MMAP_HEAP_SIZE         // internal mmap() allocator
**     SQLITE_POW2_MEMORY_SIZE       // internal power-of-two allocator
**
** If none of the above are defined, then set SQLITE_SYSTEM_MALLOC as
** the default.
*/
#if defined(SQLITE_SYSTEM_MALLOC)+defined(SQLITE_MEMDEBUG)+\
    defined(SQLITE_MEMORY_SIZE)+defined(SQLITE_MMAP_HEAP_SIZE)+\
    defined(SQLITE_POW2_MEMORY_SIZE)>1
# error "At most one of the following compile-time configuration options\
 is allows: SQLITE_SYSTEM_MALLOC, SQLITE_MEMDEBUG, SQLITE_MEMORY_SIZE,\
 SQLITE_MMAP_HEAP_SIZE, SQLITE_POW2_MEMORY_SIZE"
#endif
#if defined(SQLITE_SYSTEM_MALLOC)+defined(SQLITE_MEMDEBUG)+\
    defined(SQLITE_MEMORY_SIZE)+defined(SQLITE_MMAP_HEAP_SIZE)+\
    defined(SQLITE_POW2_MEMORY_SIZE)==0
# define SQLITE_SYSTEM_MALLOC 1
#endif

/*
** If SQLITE_MALLOC_SOFT_LIMIT is not zero, then try to keep the
** sizes of memory allocations below this value where possible.
*/







|
|
|




|
<
<

|
<

|
<
<







141
142
143
144
145
146
147
148
149
150
151
152
153
154
155


156
157

158
159


160
161
162
163
164
165
166

/*
** Exactly one of the following macros must be defined in order to
** specify which memory allocation subsystem to use.
**
**     SQLITE_SYSTEM_MALLOC          // Use normal system malloc()
**     SQLITE_MEMDEBUG               // Debugging version of system malloc()
**
** (Historical note:  There used to be several other options, but we've
** pared it down to just these two.)
**
** If none of the above are defined, then set SQLITE_SYSTEM_MALLOC as
** the default.
*/
#if defined(SQLITE_SYSTEM_MALLOC)+defined(SQLITE_MEMDEBUG)>1


# error "At most one of the following compile-time configuration options\
 is allows: SQLITE_SYSTEM_MALLOC, SQLITE_MEMDEBUG"

#endif
#if defined(SQLITE_SYSTEM_MALLOC)+defined(SQLITE_MEMDEBUG)==0


# define SQLITE_SYSTEM_MALLOC 1
#endif

/*
** If SQLITE_MALLOC_SOFT_LIMIT is not zero, then try to keep the
** sizes of memory allocations below this value where possible.
*/
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355




356
357
358
359
360
361
362
*/
#ifdef SQLITE_OMIT_TEMPDB
#define OMIT_TEMPDB 1
#else
#define OMIT_TEMPDB 0
#endif

/*
** If the following macro is set to 1, then NULL values are considered
** distinct when determining whether or not two entries are the same
** in a UNIQUE index.  This is the way PostgreSQL, Oracle, DB2, MySQL,
** OCELOT, and Firebird all work.  The SQL92 spec explicitly says this
** is the way things are suppose to work.
**
** If the following macro is set to 0, the NULLs are indistinct for
** a UNIQUE index.  In this mode, you can only have a single NULL entry
** for a column declared UNIQUE.  This is the way Informix and SQL Server
** work.
*/
#define NULL_DISTINCT_FOR_UNIQUE 1

/*
** The "file format" number is an integer that is incremented whenever
** the VDBE-level file format changes.  The following macros define the
** the default file format for new databases and the maximum file format
** that the library can read.
*/
#define SQLITE_MAX_FILE_FORMAT 4
#ifndef SQLITE_DEFAULT_FILE_FORMAT
# define SQLITE_DEFAULT_FILE_FORMAT 1
#endif





#ifndef SQLITE_DEFAULT_RECURSIVE_TRIGGERS
# define SQLITE_DEFAULT_RECURSIVE_TRIGGERS 0
#endif

/*
** Provide a default value for SQLITE_TEMP_STORE in case it is not specified
** on the command-line







<
<
<
<
<
<
<
<
<
<
<
<
<
<











>
>
>
>







324
325
326
327
328
329
330














331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
*/
#ifdef SQLITE_OMIT_TEMPDB
#define OMIT_TEMPDB 1
#else
#define OMIT_TEMPDB 0
#endif















/*
** The "file format" number is an integer that is incremented whenever
** the VDBE-level file format changes.  The following macros define the
** the default file format for new databases and the maximum file format
** that the library can read.
*/
#define SQLITE_MAX_FILE_FORMAT 4
#ifndef SQLITE_DEFAULT_FILE_FORMAT
# define SQLITE_DEFAULT_FILE_FORMAT 1
#endif

/*
** Determine whether triggers are recursive by default.  This can be
** changed at run-time using a pragma.
*/
#ifndef SQLITE_DEFAULT_RECURSIVE_TRIGGERS
# define SQLITE_DEFAULT_RECURSIVE_TRIGGERS 0
#endif

/*
** Provide a default value for SQLITE_TEMP_STORE in case it is not specified
** on the command-line
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620

621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
/*
** Forward references to structures
*/
typedef struct AggInfo AggInfo;
typedef struct AuthContext AuthContext;
typedef struct AutoincInfo AutoincInfo;
typedef struct Bitvec Bitvec;
typedef struct RowSet RowSet;
typedef struct CollSeq CollSeq;
typedef struct Column Column;
typedef struct Db Db;
typedef struct Schema Schema;
typedef struct Expr Expr;
typedef struct ExprList ExprList;
typedef struct ExprSpan ExprSpan;
typedef struct FKey FKey;
typedef struct FuncDef FuncDef;
typedef struct FuncDefHash FuncDefHash;
typedef struct IdList IdList;
typedef struct Index Index;
typedef struct IndexSample IndexSample;
typedef struct KeyClass KeyClass;
typedef struct KeyInfo KeyInfo;
typedef struct Lookaside Lookaside;
typedef struct LookasideSlot LookasideSlot;
typedef struct Module Module;
typedef struct NameContext NameContext;
typedef struct Parse Parse;

typedef struct Savepoint Savepoint;
typedef struct Select Select;
typedef struct SrcList SrcList;
typedef struct StrAccum StrAccum;
typedef struct Table Table;
typedef struct TableLock TableLock;
typedef struct Token Token;
typedef struct TriggerPrg TriggerPrg;
typedef struct TriggerStep TriggerStep;
typedef struct Trigger Trigger;
typedef struct UnpackedRecord UnpackedRecord;
typedef struct VTable VTable;
typedef struct Walker Walker;
typedef struct WherePlan WherePlan;
typedef struct WhereInfo WhereInfo;
typedef struct WhereLevel WhereLevel;








<




















>







|
|
|







583
584
585
586
587
588
589

590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
/*
** Forward references to structures
*/
typedef struct AggInfo AggInfo;
typedef struct AuthContext AuthContext;
typedef struct AutoincInfo AutoincInfo;
typedef struct Bitvec Bitvec;

typedef struct CollSeq CollSeq;
typedef struct Column Column;
typedef struct Db Db;
typedef struct Schema Schema;
typedef struct Expr Expr;
typedef struct ExprList ExprList;
typedef struct ExprSpan ExprSpan;
typedef struct FKey FKey;
typedef struct FuncDef FuncDef;
typedef struct FuncDefHash FuncDefHash;
typedef struct IdList IdList;
typedef struct Index Index;
typedef struct IndexSample IndexSample;
typedef struct KeyClass KeyClass;
typedef struct KeyInfo KeyInfo;
typedef struct Lookaside Lookaside;
typedef struct LookasideSlot LookasideSlot;
typedef struct Module Module;
typedef struct NameContext NameContext;
typedef struct Parse Parse;
typedef struct RowSet RowSet;
typedef struct Savepoint Savepoint;
typedef struct Select Select;
typedef struct SrcList SrcList;
typedef struct StrAccum StrAccum;
typedef struct Table Table;
typedef struct TableLock TableLock;
typedef struct Token Token;
typedef struct Trigger Trigger;
typedef struct TriggerPrg TriggerPrg;
typedef struct TriggerStep TriggerStep;
typedef struct UnpackedRecord UnpackedRecord;
typedef struct VTable VTable;
typedef struct Walker Walker;
typedef struct WherePlan WherePlan;
typedef struct WhereInfo WhereInfo;
typedef struct WhereLevel WhereLevel;

689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
#ifndef SQLITE_OMIT_VIRTUALTABLE
  sqlite3 *db;         /* "Owner" connection. See comment above */
#endif
};

/*
** These macros can be used to test, set, or clear bits in the 
** Db.flags field.
*/
#define DbHasProperty(D,I,P)     (((D)->aDb[I].pSchema->flags&(P))==(P))
#define DbHasAnyProperty(D,I,P)  (((D)->aDb[I].pSchema->flags&(P))!=0)
#define DbSetProperty(D,I,P)     (D)->aDb[I].pSchema->flags|=(P)
#define DbClearProperty(D,I,P)   (D)->aDb[I].pSchema->flags&=~(P)

/*
** Allowed values for the DB.flags field.
**
** The DB_SchemaLoaded flag is set after the database schema has been
** read into internal hash tables.
**
** DB_UnresetViews means that one or more views have column names that
** have been filled out.  If the schema changes, these column names might
** changes and so the view will need to be reset.







|







|







679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
#ifndef SQLITE_OMIT_VIRTUALTABLE
  sqlite3 *db;         /* "Owner" connection. See comment above */
#endif
};

/*
** These macros can be used to test, set, or clear bits in the 
** Db.pSchema->flags field.
*/
#define DbHasProperty(D,I,P)     (((D)->aDb[I].pSchema->flags&(P))==(P))
#define DbHasAnyProperty(D,I,P)  (((D)->aDb[I].pSchema->flags&(P))!=0)
#define DbSetProperty(D,I,P)     (D)->aDb[I].pSchema->flags|=(P)
#define DbClearProperty(D,I,P)   (D)->aDb[I].pSchema->flags&=~(P)

/*
** Allowed values for the DB.pSchema->flags field.
**
** The DB_SchemaLoaded flag is set after the database schema has been
** read into internal hash tables.
**
** DB_UnresetViews means that one or more views have column names that
** have been filled out.  If the schema changes, these column names might
** changes and so the view will need to be reset.
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
** Collisions are on the FuncDef.pHash chain.
*/
struct FuncDefHash {
  FuncDef *a[23];       /* Hash table for functions */
};

/*
** Each database is an instance of the following structure.
**
** The sqlite.lastRowid records the last insert rowid generated by an
** insert statement.  Inserts on views do not affect its value.  Each
** trigger has its own context, so that lastRowid can be updated inside
** triggers as usual.  The previous value will be restored once the trigger
** exits.  Upon entering a before or instead of trigger, lastRowid is no
** longer (since after version 2.8.12) reset to -1.







|







751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
** Collisions are on the FuncDef.pHash chain.
*/
struct FuncDefHash {
  FuncDef *a[23];       /* Hash table for functions */
};

/*
** Each database connection is an instance of the following structure.
**
** The sqlite.lastRowid records the last insert rowid generated by an
** insert statement.  Inserts on views do not affect its value.  Each
** trigger has its own context, so that lastRowid can be updated inside
** triggers as usual.  The previous value will be restored once the trigger
** exits.  Upon entering a before or instead of trigger, lastRowid is no
** longer (since after version 2.8.12) reset to -1.
2528
2529
2530
2531
2532
2533
2534

2535



2536
2537
2538
2539
2540
2541
2542
  int sqlite3MutexEnd(void);
#endif

int sqlite3StatusValue(int);
void sqlite3StatusAdd(int, int);
void sqlite3StatusSet(int, int);


int sqlite3IsNaN(double);




void sqlite3VXPrintf(StrAccum*, int, const char*, va_list);
#ifndef SQLITE_OMIT_TRACE
void sqlite3XPrintf(StrAccum*, const char*, ...);
#endif
char *sqlite3MPrintf(sqlite3*,const char*, ...);
char *sqlite3VMPrintf(sqlite3*,const char*, va_list);







>

>
>
>







2518
2519
2520
2521
2522
2523
2524
2525
2526
2527
2528
2529
2530
2531
2532
2533
2534
2535
2536
  int sqlite3MutexEnd(void);
#endif

int sqlite3StatusValue(int);
void sqlite3StatusAdd(int, int);
void sqlite3StatusSet(int, int);

#ifndef SQLITE_OMIT_FLOATING_POINT
int sqlite3IsNaN(double);
#else
# define sqlite3IsNaN(X)  0
#endif

void sqlite3VXPrintf(StrAccum*, int, const char*, va_list);
#ifndef SQLITE_OMIT_TRACE
void sqlite3XPrintf(StrAccum*, const char*, ...);
#endif
char *sqlite3MPrintf(sqlite3*,const char*, ...);
char *sqlite3VMPrintf(sqlite3*,const char*, va_list);
Changes to src/test3.c.
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236

237

238
239
240
241
242
243
244
  Tcl_Interp *interp,    /* The TCL interpreter that invoked this command */
  int argc,              /* Number of arguments */
  const char **argv      /* Text of each argument */
){
  Btree *pBt;
  int iTable;
  BtCursor *pCur;
  int rc;
  int wrFlag;
  char zBuf[30];

  if( argc!=4 ){
    Tcl_AppendResult(interp, "wrong # args: should be \"", argv[0],
       " ID TABLENUM WRITEABLE\"", 0);
    return TCL_ERROR;
  }
  pBt = sqlite3TestTextToPtr(argv[1]);
  if( Tcl_GetInt(interp, argv[2], &iTable) ) return TCL_ERROR;
  if( Tcl_GetBoolean(interp, argv[3], &wrFlag) ) return TCL_ERROR;
  pCur = (BtCursor *)ckalloc(sqlite3BtreeCursorSize());
  memset(pCur, 0, sqlite3BtreeCursorSize());
  sqlite3BtreeEnter(pBt);

  rc = sqlite3BtreeLockTable(pBt, iTable, wrFlag);

  if( rc==SQLITE_OK ){
    rc = sqlite3BtreeCursor(pBt, iTable, wrFlag, 0, pCur);
  }
  sqlite3BtreeLeave(pBt);
  if( rc ){
    ckfree((char *)pCur);
    Tcl_AppendResult(interp, errorName(rc), 0);







|














>

>







215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
  Tcl_Interp *interp,    /* The TCL interpreter that invoked this command */
  int argc,              /* Number of arguments */
  const char **argv      /* Text of each argument */
){
  Btree *pBt;
  int iTable;
  BtCursor *pCur;
  int rc = SQLITE_OK;
  int wrFlag;
  char zBuf[30];

  if( argc!=4 ){
    Tcl_AppendResult(interp, "wrong # args: should be \"", argv[0],
       " ID TABLENUM WRITEABLE\"", 0);
    return TCL_ERROR;
  }
  pBt = sqlite3TestTextToPtr(argv[1]);
  if( Tcl_GetInt(interp, argv[2], &iTable) ) return TCL_ERROR;
  if( Tcl_GetBoolean(interp, argv[3], &wrFlag) ) return TCL_ERROR;
  pCur = (BtCursor *)ckalloc(sqlite3BtreeCursorSize());
  memset(pCur, 0, sqlite3BtreeCursorSize());
  sqlite3BtreeEnter(pBt);
#ifndef SQLITE_OMIT_SHARED_CACHE
  rc = sqlite3BtreeLockTable(pBt, iTable, wrFlag);
#endif
  if( rc==SQLITE_OK ){
    rc = sqlite3BtreeCursor(pBt, iTable, wrFlag, 0, pCur);
  }
  sqlite3BtreeLeave(pBt);
  if( rc ){
    ckfree((char *)pCur);
    Tcl_AppendResult(interp, errorName(rc), 0);
Changes to src/test_func.c.
337
338
339
340
341
342
343

344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362

363
364
365
366
367
368
369

/*
**      hex_to_utf16be(HEX)
**
** Convert the input string from HEX into binary.  Then return the
** result using sqlite3_result_text16le().
*/

static void testHexToUtf16be(
  sqlite3_context *pCtx, 
  int nArg,
  sqlite3_value **argv
){
  int n;
  const char *zIn;
  char *zOut;
  assert( nArg==1 );
  n = sqlite3_value_bytes(argv[0]);
  zIn = (const char*)sqlite3_value_text(argv[0]);
  zOut = sqlite3_malloc( n/2 );
  if( zOut==0 ){
    sqlite3_result_error_nomem(pCtx);
  }else{
    testHexToBin(zIn, zOut);
    sqlite3_result_text16be(pCtx, zOut, n/2, sqlite3_free);
  }
}


/*
**      hex_to_utf8(HEX)
**
** Convert the input string from HEX into binary.  Then return the
** result using sqlite3_result_text16le().
*/







>



















>







337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371

/*
**      hex_to_utf16be(HEX)
**
** Convert the input string from HEX into binary.  Then return the
** result using sqlite3_result_text16le().
*/
#ifndef SQLITE_OMIT_UTF16
static void testHexToUtf16be(
  sqlite3_context *pCtx, 
  int nArg,
  sqlite3_value **argv
){
  int n;
  const char *zIn;
  char *zOut;
  assert( nArg==1 );
  n = sqlite3_value_bytes(argv[0]);
  zIn = (const char*)sqlite3_value_text(argv[0]);
  zOut = sqlite3_malloc( n/2 );
  if( zOut==0 ){
    sqlite3_result_error_nomem(pCtx);
  }else{
    testHexToBin(zIn, zOut);
    sqlite3_result_text16be(pCtx, zOut, n/2, sqlite3_free);
  }
}
#endif

/*
**      hex_to_utf8(HEX)
**
** Convert the input string from HEX into binary.  Then return the
** result using sqlite3_result_text16le().
*/
389
390
391
392
393
394
395

396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414

415
416
417
418
419
420
421

/*
**      hex_to_utf16le(HEX)
**
** Convert the input string from HEX into binary.  Then return the
** result using sqlite3_result_text16le().
*/

static void testHexToUtf16le(
  sqlite3_context *pCtx, 
  int nArg,
  sqlite3_value **argv
){
  int n;
  const char *zIn;
  char *zOut;
  assert( nArg==1 );
  n = sqlite3_value_bytes(argv[0]);
  zIn = (const char*)sqlite3_value_text(argv[0]);
  zOut = sqlite3_malloc( n/2 );
  if( zOut==0 ){
    sqlite3_result_error_nomem(pCtx);
  }else{
    testHexToBin(zIn, zOut);
    sqlite3_result_text16le(pCtx, zOut, n/2, sqlite3_free);
  }
}


static int registerTestFunctions(sqlite3 *db){
  static const struct {
     char *zName;
     signed char nArg;
     unsigned char eTextRep; /* 1: UTF-16.  0: UTF-8 */
     void (*xFunc)(sqlite3_context*,int,sqlite3_value **);







>



















>







391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425

/*
**      hex_to_utf16le(HEX)
**
** Convert the input string from HEX into binary.  Then return the
** result using sqlite3_result_text16le().
*/
#ifndef SQLITE_OMIT_UTF16
static void testHexToUtf16le(
  sqlite3_context *pCtx, 
  int nArg,
  sqlite3_value **argv
){
  int n;
  const char *zIn;
  char *zOut;
  assert( nArg==1 );
  n = sqlite3_value_bytes(argv[0]);
  zIn = (const char*)sqlite3_value_text(argv[0]);
  zOut = sqlite3_malloc( n/2 );
  if( zOut==0 ){
    sqlite3_result_error_nomem(pCtx);
  }else{
    testHexToBin(zIn, zOut);
    sqlite3_result_text16le(pCtx, zOut, n/2, sqlite3_free);
  }
}
#endif

static int registerTestFunctions(sqlite3 *db){
  static const struct {
     char *zName;
     signed char nArg;
     unsigned char eTextRep; /* 1: UTF-16.  0: UTF-8 */
     void (*xFunc)(sqlite3_context*,int,sqlite3_value **);
Changes to src/util.c.
27
28
29
30
31
32
33

34
35
36
37
38
39
40
#ifdef SQLITE_COVERAGE_TEST
void sqlite3Coverage(int x){
  static int dummy = 0;
  dummy += x;
}
#endif


/*
** Return true if the floating point value is Not a Number (NaN).
**
** Use the math library isnan() function if compiled with SQLITE_HAVE_ISNAN.
** Otherwise, we have our own implementation that works on most systems.
*/
int sqlite3IsNaN(double x){







>







27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
#ifdef SQLITE_COVERAGE_TEST
void sqlite3Coverage(int x){
  static int dummy = 0;
  dummy += x;
}
#endif

#ifndef SQLITE_OMIT_FLOATING_POINT
/*
** Return true if the floating point value is Not a Number (NaN).
**
** Use the math library isnan() function if compiled with SQLITE_HAVE_ISNAN.
** Otherwise, we have our own implementation that works on most systems.
*/
int sqlite3IsNaN(double x){
71
72
73
74
75
76
77

78
79
80
81
82
83
84
  rc = (y!=z);
#else  /* if defined(SQLITE_HAVE_ISNAN) */
  rc = isnan(x);
#endif /* SQLITE_HAVE_ISNAN */
  testcase( rc );
  return rc;
}


/*
** Compute a string length that is limited to what can be stored in
** lower 30 bits of a 32-bit signed integer.
**
** The value returned will never be negative.  Nor will it ever be greater
** than the actual length of the string.  For very long strings (greater







>







72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
  rc = (y!=z);
#else  /* if defined(SQLITE_HAVE_ISNAN) */
  rc = isnan(x);
#endif /* SQLITE_HAVE_ISNAN */
  testcase( rc );
  return rc;
}
#endif /* SQLITE_OMIT_FLOATING_POINT */

/*
** Compute a string length that is limited to what can be stored in
** lower 30 bits of a 32-bit signed integer.
**
** The value returned will never be negative.  Nor will it ever be greater
** than the actual length of the string.  For very long strings (greater
251
252
253
254
255
256
257

258
259
260
261
262
263
264
265
266
267
268
269
270

271
272
273
274
275
276
277
  if( *z=='-' || *z=='+' ) z += incr;
  if( !sqlite3Isdigit(*z) ){
    return 0;
  }
  z += incr;
  *realnum = 0;
  while( sqlite3Isdigit(*z) ){ z += incr; }

  if( *z=='.' ){
    z += incr;
    if( !sqlite3Isdigit(*z) ) return 0;
    while( sqlite3Isdigit(*z) ){ z += incr; }
    *realnum = 1;
  }
  if( *z=='e' || *z=='E' ){
    z += incr;
    if( *z=='+' || *z=='-' ) z += incr;
    if( !sqlite3Isdigit(*z) ) return 0;
    while( sqlite3Isdigit(*z) ){ z += incr; }
    *realnum = 1;
  }

  return *z==0;
}

/*
** The string z[] is an ASCII representation of a real number.
** Convert this string to a double.
**







>













>







253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
  if( *z=='-' || *z=='+' ) z += incr;
  if( !sqlite3Isdigit(*z) ){
    return 0;
  }
  z += incr;
  *realnum = 0;
  while( sqlite3Isdigit(*z) ){ z += incr; }
#ifndef SQLITE_OMIT_FLOATING_POINT
  if( *z=='.' ){
    z += incr;
    if( !sqlite3Isdigit(*z) ) return 0;
    while( sqlite3Isdigit(*z) ){ z += incr; }
    *realnum = 1;
  }
  if( *z=='e' || *z=='E' ){
    z += incr;
    if( *z=='+' || *z=='-' ) z += incr;
    if( !sqlite3Isdigit(*z) ) return 0;
    while( sqlite3Isdigit(*z) ){ z += incr; }
    *realnum = 1;
  }
#endif
  return *z==0;
}

/*
** The string z[] is an ASCII representation of a real number.
** Convert this string to a double.
**
425
426
427
428
429
430
431



432
433
434
435
436
437
438
** will return -8.
*/
static int compare2pow63(const char *zNum){
  int c;
  c = memcmp(zNum,"922337203685477580",18)*10;
  if( c==0 ){
    c = zNum[18] - '8';



  }
  return c;
}


/*
** Return TRUE if zNum is a 64-bit signed integer and write







>
>
>







429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
** will return -8.
*/
static int compare2pow63(const char *zNum){
  int c;
  c = memcmp(zNum,"922337203685477580",18)*10;
  if( c==0 ){
    c = zNum[18] - '8';
    testcase( c==(-1) );
    testcase( c==0 );
    testcase( c==(+1) );
  }
  return c;
}


/*
** Return TRUE if zNum is a 64-bit signed integer and write
461
462
463
464
465
466
467



468
469
470
471
472
473
474
  }
  zStart = zNum;
  while( zNum[0]=='0' ){ zNum++; } /* Skip over leading zeros. Ticket #2454 */
  for(i=0; (c=zNum[i])>='0' && c<='9'; i++){
    v = v*10 + c - '0';
  }
  *pNum = neg ? -v : v;



  if( c!=0 || (i==0 && zStart==zNum) || i>19 ){
    /* zNum is empty or contains non-numeric text or is longer
    ** than 19 digits (thus guaranting that it is too large) */
    return 0;
  }else if( i<19 ){
    /* Less than 19 digits, so we know that it fits in 64 bits */
    return 1;







>
>
>







468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
  }
  zStart = zNum;
  while( zNum[0]=='0' ){ zNum++; } /* Skip over leading zeros. Ticket #2454 */
  for(i=0; (c=zNum[i])>='0' && c<='9'; i++){
    v = v*10 + c - '0';
  }
  *pNum = neg ? -v : v;
  testcase( i==18 );
  testcase( i==19 );
  testcase( i==20 );
  if( c!=0 || (i==0 && zStart==zNum) || i>19 ){
    /* zNum is empty or contains non-numeric text or is longer
    ** than 19 digits (thus guaranting that it is too large) */
    return 0;
  }else if( i<19 ){
    /* Less than 19 digits, so we know that it fits in 64 bits */
    return 1;
504
505
506
507
508
509
510



511
512
513
514
515
516
517
  assert( zNum[0]>='0' && zNum[0]<='9' ); /* zNum is an unsigned number */

  if( negFlag ) neg = 1-neg;
  while( *zNum=='0' ){
    zNum++;   /* Skip leading zeros.  Ticket #2454 */
  }
  for(i=0; zNum[i]; i++){ assert( zNum[i]>='0' && zNum[i]<='9' ); }



  if( i<19 ){
    /* Guaranteed to fit if less than 19 digits */
    return 1;
  }else if( i>19 ){
    /* Guaranteed to be too big if greater than 19 digits */
    return 0;
  }else{







>
>
>







514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
  assert( zNum[0]>='0' && zNum[0]<='9' ); /* zNum is an unsigned number */

  if( negFlag ) neg = 1-neg;
  while( *zNum=='0' ){
    zNum++;   /* Skip leading zeros.  Ticket #2454 */
  }
  for(i=0; zNum[i]; i++){ assert( zNum[i]>='0' && zNum[i]<='9' ); }
  testcase( i==18 );
  testcase( i==19 );
  testcase( i==20 );
  if( i<19 ){
    /* Guaranteed to fit if less than 19 digits */
    return 1;
  }else if( i>19 ){
    /* Guaranteed to be too big if greater than 19 digits */
    return 0;
  }else{
544
545
546
547
548
549
550

551
552
553

554
555
556
557
558
559
560
  }

  /* The longest decimal representation of a 32 bit integer is 10 digits:
  **
  **             1234567890
  **     2^31 -> 2147483648
  */

  if( i>10 ){
    return 0;
  }

  if( v-neg>2147483647 ){
    return 0;
  }
  if( neg ){
    v = -v;
  }
  *pValue = (int)v;







>



>







557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
  }

  /* The longest decimal representation of a 32 bit integer is 10 digits:
  **
  **             1234567890
  **     2^31 -> 2147483648
  */
  testcase( i==10 );
  if( i>10 ){
    return 0;
  }
  testcase( v-neg==2147483647 );
  if( v-neg>2147483647 ){
    return 0;
  }
  if( neg ){
    v = -v;
  }
  *pValue = (int)v;
Changes to src/vacuum.c.
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
  ** restored before returning. Then set the writable-schema flag, and
  ** disable CHECK and foreign key constraints.  */
  saved_flags = db->flags;
  saved_nChange = db->nChange;
  saved_nTotalChange = db->nTotalChange;
  saved_xTrace = db->xTrace;
  db->flags |= SQLITE_WriteSchema | SQLITE_IgnoreChecks;
  db->flags &= ~SQLITE_ForeignKeys;
  db->xTrace = 0;

  pMain = db->aDb[0].pBt;
  isMemDb = sqlite3PagerIsMemdb(sqlite3BtreePager(pMain));

  /* Attach the temporary database as 'vacuum_db'. The synchronous pragma
  ** can be set to 'off' for this file, as it is not recovered if a crash







|







100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
  ** restored before returning. Then set the writable-schema flag, and
  ** disable CHECK and foreign key constraints.  */
  saved_flags = db->flags;
  saved_nChange = db->nChange;
  saved_nTotalChange = db->nTotalChange;
  saved_xTrace = db->xTrace;
  db->flags |= SQLITE_WriteSchema | SQLITE_IgnoreChecks;
  db->flags &= ~(SQLITE_ForeignKeys | SQLITE_ReverseOrder);
  db->xTrace = 0;

  pMain = db->aDb[0].pBt;
  isMemDb = sqlite3PagerIsMemdb(sqlite3BtreePager(pMain));

  /* Attach the temporary database as 'vacuum_db'. The synchronous pragma
  ** can be set to 'off' for this file, as it is not recovered if a crash
Changes to src/vdbe.c.
880
881
882
883
884
885
886

887
888
889
890
891
892
893
894
895
896
897

898
899
900
901
902
903
904
*/
case OP_Int64: {           /* out2-prerelease */
  assert( pOp->p4.pI64!=0 );
  pOut->u.i = *pOp->p4.pI64;
  break;
}


/* Opcode: Real * P2 * P4 *
**
** P4 is a pointer to a 64-bit floating point value.
** Write that value into register P2.
*/
case OP_Real: {            /* same as TK_FLOAT, out2-prerelease */
  pOut->flags = MEM_Real;
  assert( !sqlite3IsNaN(*pOp->p4.pReal) );
  pOut->r = *pOp->p4.pReal;
  break;
}


/* Opcode: String8 * P2 * P4 *
**
** P4 points to a nul terminated UTF-8 string. This opcode is transformed 
** into an OP_String before it is executed for the first time.
*/
case OP_String8: {         /* same as TK_STRING, out2-prerelease */







>











>







880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
*/
case OP_Int64: {           /* out2-prerelease */
  assert( pOp->p4.pI64!=0 );
  pOut->u.i = *pOp->p4.pI64;
  break;
}

#ifndef SQLITE_OMIT_FLOATING_POINT
/* Opcode: Real * P2 * P4 *
**
** P4 is a pointer to a 64-bit floating point value.
** Write that value into register P2.
*/
case OP_Real: {            /* same as TK_FLOAT, out2-prerelease */
  pOut->flags = MEM_Real;
  assert( !sqlite3IsNaN(*pOp->p4.pReal) );
  pOut->r = *pOp->p4.pReal;
  break;
}
#endif

/* Opcode: String8 * P2 * P4 *
**
** P4 points to a nul terminated UTF-8 string. This opcode is transformed 
** into an OP_String before it is executed for the first time.
*/
case OP_String8: {         /* same as TK_STRING, out2-prerelease */
1291
1292
1293
1294
1295
1296
1297




1298
1299
1300
1301
1302
1303
1304
1305

1306
1307
1308
1309
1310
1311
1312
        iB = (i64)rB;
        if( iA==0 ) goto arithmetic_result_is_null;
        if( iA==-1 ) iA = 1;
        rB = (double)(iB % iA);
        break;
      }
    }




    if( sqlite3IsNaN(rB) ){
      goto arithmetic_result_is_null;
    }
    pOut->r = rB;
    MemSetTypeFlag(pOut, MEM_Real);
    if( (flags & MEM_Real)==0 ){
      sqlite3VdbeIntegerAffinity(pOut);
    }

  }
  break;

arithmetic_result_is_null:
  sqlite3VdbeMemSetNull(pOut);
  break;
}







>
>
>
>








>







1293
1294
1295
1296
1297
1298
1299
1300
1301
1302
1303
1304
1305
1306
1307
1308
1309
1310
1311
1312
1313
1314
1315
1316
1317
1318
1319
        iB = (i64)rB;
        if( iA==0 ) goto arithmetic_result_is_null;
        if( iA==-1 ) iA = 1;
        rB = (double)(iB % iA);
        break;
      }
    }
#ifdef SQLITE_OMIT_FLOATING_POINT
    pOut->u.i = rB;
    MemSetTypeFlag(pOut, MEM_Int);
#else
    if( sqlite3IsNaN(rB) ){
      goto arithmetic_result_is_null;
    }
    pOut->r = rB;
    MemSetTypeFlag(pOut, MEM_Real);
    if( (flags & MEM_Real)==0 ){
      sqlite3VdbeIntegerAffinity(pOut);
    }
#endif
  }
  break;

arithmetic_result_is_null:
  sqlite3VdbeMemSetNull(pOut);
  break;
}
1526
1527
1528
1529
1530
1531
1532

1533
1534
1535
1536
1537
1538
1539
1540
1541
1542
1543
1544
1545
1546
1547
1548

1549
1550
1551
1552
1553
1554
1555
    }
  }else{
    MemSetTypeFlag(pIn1, MEM_Int);
  }
  break;
}


/* Opcode: RealAffinity P1 * * * *
**
** If register P1 holds an integer convert it to a real value.
**
** This opcode is used when extracting information from a column that
** has REAL affinity.  Such column values may still be stored as
** integers, for space efficiency, but after extraction we want them
** to have only a real value.
*/
case OP_RealAffinity: {                  /* in1 */
  pIn1 = &aMem[pOp->p1];
  if( pIn1->flags & MEM_Int ){
    sqlite3VdbeMemRealify(pIn1);
  }
  break;
}


#ifndef SQLITE_OMIT_CAST
/* Opcode: ToText P1 * * * *
**
** Force the value in register P1 to be text.
** If the value is numeric, convert it to a string using the
** equivalent of printf().  Blob values are unchanged and







>
















>







1533
1534
1535
1536
1537
1538
1539
1540
1541
1542
1543
1544
1545
1546
1547
1548
1549
1550
1551
1552
1553
1554
1555
1556
1557
1558
1559
1560
1561
1562
1563
1564
    }
  }else{
    MemSetTypeFlag(pIn1, MEM_Int);
  }
  break;
}

#ifndef SQLITE_OMIT_FLOATING_POINT
/* Opcode: RealAffinity P1 * * * *
**
** If register P1 holds an integer convert it to a real value.
**
** This opcode is used when extracting information from a column that
** has REAL affinity.  Such column values may still be stored as
** integers, for space efficiency, but after extraction we want them
** to have only a real value.
*/
case OP_RealAffinity: {                  /* in1 */
  pIn1 = &aMem[pOp->p1];
  if( pIn1->flags & MEM_Int ){
    sqlite3VdbeMemRealify(pIn1);
  }
  break;
}
#endif

#ifndef SQLITE_OMIT_CAST
/* Opcode: ToText P1 * * * *
**
** Force the value in register P1 to be text.
** If the value is numeric, convert it to a string using the
** equivalent of printf().  Blob values are unchanged and
1625
1626
1627
1628
1629
1630
1631
1632
1633
1634
1635
1636
1637
1638
1639
1640
1641
1642
1643
1644
1645
1646
1647
1648
1649
1650
1651
1652
1653
1654
1655
1656
  pIn1 = &aMem[pOp->p1];
  if( (pIn1->flags & MEM_Null)==0 ){
    sqlite3VdbeMemIntegerify(pIn1);
  }
  break;
}

#ifndef SQLITE_OMIT_CAST
/* Opcode: ToReal P1 * * * *
**
** Force the value in register P1 to be a floating point number.
** If The value is currently an integer, convert it.
** If the value is text or blob, try to convert it to an integer using the
** equivalent of atoi() and store 0.0 if no such conversion is possible.
**
** A NULL value is not changed by this routine.  It remains NULL.
*/
case OP_ToReal: {                  /* same as TK_TO_REAL, in1 */
  pIn1 = &aMem[pOp->p1];
  if( (pIn1->flags & MEM_Null)==0 ){
    sqlite3VdbeMemRealify(pIn1);
  }
  break;
}
#endif /* SQLITE_OMIT_CAST */

/* Opcode: Lt P1 P2 P3 P4 P5
**
** Compare the values in register P1 and P3.  If reg(P3)<reg(P1) then
** jump to address P2.  
**
** If the SQLITE_JUMPIFNULL bit of P5 is set and either reg(P1) or







|
















|







1634
1635
1636
1637
1638
1639
1640
1641
1642
1643
1644
1645
1646
1647
1648
1649
1650
1651
1652
1653
1654
1655
1656
1657
1658
1659
1660
1661
1662
1663
1664
1665
  pIn1 = &aMem[pOp->p1];
  if( (pIn1->flags & MEM_Null)==0 ){
    sqlite3VdbeMemIntegerify(pIn1);
  }
  break;
}

#if !defined(SQLITE_OMIT_CAST) && !defined(SQLITE_OMIT_FLOATING_POINT)
/* Opcode: ToReal P1 * * * *
**
** Force the value in register P1 to be a floating point number.
** If The value is currently an integer, convert it.
** If the value is text or blob, try to convert it to an integer using the
** equivalent of atoi() and store 0.0 if no such conversion is possible.
**
** A NULL value is not changed by this routine.  It remains NULL.
*/
case OP_ToReal: {                  /* same as TK_TO_REAL, in1 */
  pIn1 = &aMem[pOp->p1];
  if( (pIn1->flags & MEM_Null)==0 ){
    sqlite3VdbeMemRealify(pIn1);
  }
  break;
}
#endif /* !defined(SQLITE_OMIT_CAST) && !defined(SQLITE_OMIT_FLOATING_POINT) */

/* Opcode: Lt P1 P2 P3 P4 P5
**
** Compare the values in register P1 and P3.  If reg(P3)<reg(P1) then
** jump to address P2.  
**
** If the SQLITE_JUMPIFNULL bit of P5 is set and either reg(P1) or
4568
4569
4570
4571
4572
4573
4574
4575
4576
4577
4578
4579
4580
4581
4582
  sqlite3BtreeEnterAll(db);
  if( pOp->p2 || DbHasProperty(db, iDb, DB_SchemaLoaded) ){
    zMaster = SCHEMA_TABLE(iDb);
    initData.db = db;
    initData.iDb = pOp->p1;
    initData.pzErrMsg = &p->zErrMsg;
    zSql = sqlite3MPrintf(db,
       "SELECT name, rootpage, sql FROM '%q'.%s WHERE %s",
       db->aDb[iDb].zName, zMaster, pOp->p4.z);
    if( zSql==0 ){
      rc = SQLITE_NOMEM;
    }else{
      (void)sqlite3SafetyOff(db);
      assert( db->init.busy==0 );
      db->init.busy = 1;







|







4577
4578
4579
4580
4581
4582
4583
4584
4585
4586
4587
4588
4589
4590
4591
  sqlite3BtreeEnterAll(db);
  if( pOp->p2 || DbHasProperty(db, iDb, DB_SchemaLoaded) ){
    zMaster = SCHEMA_TABLE(iDb);
    initData.db = db;
    initData.iDb = pOp->p1;
    initData.pzErrMsg = &p->zErrMsg;
    zSql = sqlite3MPrintf(db,
       "SELECT name, rootpage, sql FROM '%q'.%s WHERE %s ORDER BY rowid",
       db->aDb[iDb].zName, zMaster, pOp->p4.z);
    if( zSql==0 ){
      rc = SQLITE_NOMEM;
    }else{
      (void)sqlite3SafetyOff(db);
      assert( db->init.busy==0 );
      db->init.busy = 1;
5660
5661
5662
5663
5664
5665
5666

5667
5668
5669
5670
5671
5672
5673
/*
** The magic Explain opcode are only inserted when explain==2 (which
** is to say when the EXPLAIN QUERY PLAN syntax is used.)
** This opcode records information from the optimizer.  It is the
** the same as a no-op.  This opcodesnever appears in a real VM program.
*/
default: {          /* This is really OP_Noop and OP_Explain */

  break;
}

/*****************************************************************************
** The cases of the switch statement above this line should all be indented
** by 6 spaces.  But the left-most 6 spaces have been removed to improve the
** readability.  From this point on down, the normal indentation rules are







>







5669
5670
5671
5672
5673
5674
5675
5676
5677
5678
5679
5680
5681
5682
5683
/*
** The magic Explain opcode are only inserted when explain==2 (which
** is to say when the EXPLAIN QUERY PLAN syntax is used.)
** This opcode records information from the optimizer.  It is the
** the same as a no-op.  This opcodesnever appears in a real VM program.
*/
default: {          /* This is really OP_Noop and OP_Explain */
  assert( pOp->opcode==OP_Noop || pOp->opcode==OP_Explain );
  break;
}

/*****************************************************************************
** The cases of the switch statement above this line should all be indented
** by 6 spaces.  But the left-most 6 spaces have been removed to improve the
** readability.  From this point on down, the normal indentation rules are
Changes to src/vdbeInt.h.
358
359
360
361
362
363
364



365

366
367
368
369
370
371
372
int sqlite3VdbeMemTooBig(Mem*);
int sqlite3VdbeMemCopy(Mem*, const Mem*);
void sqlite3VdbeMemShallowCopy(Mem*, const Mem*, int);
void sqlite3VdbeMemMove(Mem*, Mem*);
int sqlite3VdbeMemNulTerminate(Mem*);
int sqlite3VdbeMemSetStr(Mem*, const char*, int, u8, void(*)(void*));
void sqlite3VdbeMemSetInt64(Mem*, i64);



void sqlite3VdbeMemSetDouble(Mem*, double);

void sqlite3VdbeMemSetNull(Mem*);
void sqlite3VdbeMemSetZeroBlob(Mem*,int);
void sqlite3VdbeMemSetRowSet(Mem*);
int sqlite3VdbeMemMakeWriteable(Mem*);
int sqlite3VdbeMemStringify(Mem*, int);
i64 sqlite3VdbeIntValue(Mem*);
int sqlite3VdbeMemIntegerify(Mem*);







>
>
>

>







358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
int sqlite3VdbeMemTooBig(Mem*);
int sqlite3VdbeMemCopy(Mem*, const Mem*);
void sqlite3VdbeMemShallowCopy(Mem*, const Mem*, int);
void sqlite3VdbeMemMove(Mem*, Mem*);
int sqlite3VdbeMemNulTerminate(Mem*);
int sqlite3VdbeMemSetStr(Mem*, const char*, int, u8, void(*)(void*));
void sqlite3VdbeMemSetInt64(Mem*, i64);
#ifdef SQLITE_OMIT_FLOATING_POINT
# define sqlite3VdbeMemSetDouble sqlite3VdbeMemSetInt64
#else
void sqlite3VdbeMemSetDouble(Mem*, double);
#endif
void sqlite3VdbeMemSetNull(Mem*);
void sqlite3VdbeMemSetZeroBlob(Mem*,int);
void sqlite3VdbeMemSetRowSet(Mem*);
int sqlite3VdbeMemMakeWriteable(Mem*);
int sqlite3VdbeMemStringify(Mem*, int);
i64 sqlite3VdbeIntValue(Mem*);
int sqlite3VdbeMemIntegerify(Mem*);
Changes to src/vdbeapi.c.
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
    rc = SQLITE_OK;
  }else{
    Vdbe *v = (Vdbe*)pStmt;
#ifdef SQLITE_ENABLE_SQLRR
    SRRecFinalize(pStmt);
#endif
    sqlite3 *db = v->db;
    if( db==0 ){
      return SQLITE_MISUSE;
    }
#if SQLITE_THREADSAFE
    sqlite3_mutex *mutex = v->db->mutex;
#endif
    sqlite3_mutex_enter(mutex);
    rc = sqlite3VdbeFinalize(v);
    rc = sqlite3ApiExit(db, rc);
    sqlite3_mutex_leave(mutex);







<
|
<







49
50
51
52
53
54
55

56

57
58
59
60
61
62
63
    rc = SQLITE_OK;
  }else{
    Vdbe *v = (Vdbe*)pStmt;
#ifdef SQLITE_ENABLE_SQLRR
    SRRecFinalize(pStmt);
#endif
    sqlite3 *db = v->db;

    if( db==0 ) return SQLITE_MISUSE;

#if SQLITE_THREADSAFE
    sqlite3_mutex *mutex = v->db->mutex;
#endif
    sqlite3_mutex_enter(mutex);
    rc = sqlite3VdbeFinalize(v);
    rc = sqlite3ApiExit(db, rc);
    sqlite3_mutex_leave(mutex);
414
415
416
417
418
419
420
421
422
423

424
425
426
427
428
429
430
431
432
433
434
** This is the top-level implementation of sqlite3_step().  Call
** sqlite3Step() to do most of the work.  If a schema error occurs,
** call sqlite3Reprepare() and try again.
*/
int sqlite3_step(sqlite3_stmt *pStmt){
  int rc = SQLITE_MISUSE;
  Vdbe *v = (Vdbe*)pStmt;
  sqlite3 *db;
  if( v && ((db = v->db) != NULL)){
    int cnt = 0;

#ifdef SQLITE_ENABLE_SQLRR
    SRRecStep(pStmt);
#endif
    
    sqlite3_mutex_enter(db->mutex);
    while( (rc = sqlite3Step(v))==SQLITE_SCHEMA
           && cnt++ < 5
           && (rc = sqlite3Reprepare(v))==SQLITE_OK ){
      sqlite3_reset(pStmt);
      v->expired = 0;
    }







<
|

>



<







412
413
414
415
416
417
418

419
420
421
422
423
424

425
426
427
428
429
430
431
** This is the top-level implementation of sqlite3_step().  Call
** sqlite3Step() to do most of the work.  If a schema error occurs,
** call sqlite3Reprepare() and try again.
*/
int sqlite3_step(sqlite3_stmt *pStmt){
  int rc = SQLITE_MISUSE;
  Vdbe *v = (Vdbe*)pStmt;

  if( v && (v->db)!=0 ){
    int cnt = 0;
    sqlite3 *db = v->db;
#ifdef SQLITE_ENABLE_SQLRR
    SRRecStep(pStmt);
#endif

    sqlite3_mutex_enter(db->mutex);
    while( (rc = sqlite3Step(v))==SQLITE_SCHEMA
           && cnt++ < 5
           && (rc = sqlite3Reprepare(v))==SQLITE_OK ){
      sqlite3_reset(pStmt);
      v->expired = 0;
    }
Changes to src/vdbeaux.c.
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
}

/*
** Return the SQL associated with a prepared statement
*/
const char *sqlite3_sql(sqlite3_stmt *pStmt){
  Vdbe *p = (Vdbe *)pStmt;
  return ((p && p->isPrepareV2) ? p->zSql : 0);
}

/*
** Swap all content between two VDBE structures.
*/
void sqlite3VdbeSwap(Vdbe *pA, Vdbe *pB){
  Vdbe tmp, *pTmp;







|







62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
}

/*
** Return the SQL associated with a prepared statement
*/
const char *sqlite3_sql(sqlite3_stmt *pStmt){
  Vdbe *p = (Vdbe *)pStmt;
  return (p && p->isPrepareV2) ? p->zSql : 0;
}

/*
** Swap all content between two VDBE structures.
*/
void sqlite3VdbeSwap(Vdbe *pA, Vdbe *pB){
  Vdbe tmp, *pTmp;
2332
2333
2334
2335
2336
2337
2338
2339
2340
2341
2342
2343
2344
2345
2346
  releaseMemArray(p->aColName, p->nResColumn*COLNAME_N);
  vdbeFreeOpArray(db, p->aOp, p->nOp);
  sqlite3DbFree(db, p->aLabel);
  sqlite3DbFree(db, p->aColName);
  sqlite3DbFree(db, p->zSql);
  p->magic = VDBE_MAGIC_DEAD;
  sqlite3DbFree(db, p->pFree);
  p->db = NULL;
  sqlite3DbFree(db, p);
}

/*
** Make sure the cursor p is ready to read or write the row to which it
** was last positioned.  Return an error code if an OOM fault or I/O error
** prevents us from positioning the cursor to its correct position.







|







2332
2333
2334
2335
2336
2337
2338
2339
2340
2341
2342
2343
2344
2345
2346
  releaseMemArray(p->aColName, p->nResColumn*COLNAME_N);
  vdbeFreeOpArray(db, p->aOp, p->nOp);
  sqlite3DbFree(db, p->aLabel);
  sqlite3DbFree(db, p->aColName);
  sqlite3DbFree(db, p->zSql);
  p->magic = VDBE_MAGIC_DEAD;
  sqlite3DbFree(db, p->pFree);
  p->db = 0;
  sqlite3DbFree(db, p);
}

/*
** Make sure the cursor p is ready to read or write the row to which it
** was last positioned.  Return an error code if an OOM fault or I/O error
** prevents us from positioning the cursor to its correct position.
Changes to src/vdbemem.c.
311
312
313
314
315
316
317




318
319
320
321
322
323
324
** there are reports that windows throws an expection
** if the floating point value is out of range. (See ticket #2880.)
** Because we do not completely understand the problem, we will
** take the conservative approach and always do range tests
** before attempting the conversion.
*/
static i64 doubleToInt64(double r){




  /*
  ** Many compilers we encounter do not define constants for the
  ** minimum and maximum 64-bit integers, or they define them
  ** inconsistently.  And many do not understand the "LL" notation.
  ** So we define our own static constants here using nothing
  ** larger than a 32-bit integer constant.
  */







>
>
>
>







311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
** there are reports that windows throws an expection
** if the floating point value is out of range. (See ticket #2880.)
** Because we do not completely understand the problem, we will
** take the conservative approach and always do range tests
** before attempting the conversion.
*/
static i64 doubleToInt64(double r){
#ifdef SQLITE_OMIT_FLOATING_POINT
  /* When floating-point is omitted, double and int64 are the same thing */
  return r;
#else
  /*
  ** Many compilers we encounter do not define constants for the
  ** minimum and maximum 64-bit integers, or they define them
  ** inconsistently.  And many do not understand the "LL" notation.
  ** So we define our own static constants here using nothing
  ** larger than a 32-bit integer constant.
  */
332
333
334
335
336
337
338

339
340
341
342
343
344
345
    ** a very large positive number to an integer results in a very large
    ** negative integer.  This makes no sense, but it is what x86 hardware
    ** does so for compatibility we will do the same in software. */
    return minInt;
  }else{
    return (i64)r;
  }

}

/*
** Return some kind of integer value which is the best we can do
** at representing the value that *pMem describes as an integer.
** If pMem is an integer, then the value is exact.  If pMem is
** a floating-point then the value returned is the integer part.







>







336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
    ** a very large positive number to an integer results in a very large
    ** negative integer.  This makes no sense, but it is what x86 hardware
    ** does so for compatibility we will do the same in software. */
    return minInt;
  }else{
    return (i64)r;
  }
#endif
}

/*
** Return some kind of integer value which is the best we can do
** at representing the value that *pMem describes as an integer.
** If pMem is an integer, then the value is exact.  If pMem is
** a floating-point then the value returned is the integer part.
525
526
527
528
529
530
531

532
533
534
535
536
537
538
539
540
541
542
543
544
545

546
547
548
549
550
551
552
void sqlite3VdbeMemSetInt64(Mem *pMem, i64 val){
  sqlite3VdbeMemRelease(pMem);
  pMem->u.i = val;
  pMem->flags = MEM_Int;
  pMem->type = SQLITE_INTEGER;
}


/*
** Delete any previous value and set the value stored in *pMem to val,
** manifest type REAL.
*/
void sqlite3VdbeMemSetDouble(Mem *pMem, double val){
  if( sqlite3IsNaN(val) ){
    sqlite3VdbeMemSetNull(pMem);
  }else{
    sqlite3VdbeMemRelease(pMem);
    pMem->r = val;
    pMem->flags = MEM_Real;
    pMem->type = SQLITE_FLOAT;
  }
}


/*
** Delete any previous value and set the value of pMem to be an
** empty boolean index.
*/
void sqlite3VdbeMemSetRowSet(Mem *pMem){
  sqlite3 *db = pMem->db;







>














>







530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
void sqlite3VdbeMemSetInt64(Mem *pMem, i64 val){
  sqlite3VdbeMemRelease(pMem);
  pMem->u.i = val;
  pMem->flags = MEM_Int;
  pMem->type = SQLITE_INTEGER;
}

#ifndef SQLITE_OMIT_FLOATING_POINT
/*
** Delete any previous value and set the value stored in *pMem to val,
** manifest type REAL.
*/
void sqlite3VdbeMemSetDouble(Mem *pMem, double val){
  if( sqlite3IsNaN(val) ){
    sqlite3VdbeMemSetNull(pMem);
  }else{
    sqlite3VdbeMemRelease(pMem);
    pMem->r = val;
    pMem->flags = MEM_Real;
    pMem->type = SQLITE_FLOAT;
  }
}
#endif

/*
** Delete any previous value and set the value of pMem to be an
** empty boolean index.
*/
void sqlite3VdbeMemSetRowSet(Mem *pMem){
  sqlite3 *db = pMem->db;
Changes to test/auth2.test.
98
99
100
101
102
103
104

105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123

124
125
126
127
128
129
130
SQLITE_UPDATE sqlite_master rootpage main {}
SQLITE_UPDATE sqlite_master sql main {}
SQLITE_READ sqlite_master ROWID main {}
SQLITE_READ sqlite_master name main {}
SQLITE_READ sqlite_master rootpage main {}
SQLITE_READ sqlite_master sql main {}
SQLITE_READ sqlite_master tbl_name main {}

}
do_test auth2-2.2 {
  set ::authargs {}
  db eval {
    CREATE VIEW v2 AS SELECT x+y AS a, y+z AS b from t2;
  }
  set ::authargs
} {SQLITE_INSERT sqlite_master {} main {}
SQLITE_CREATE_VIEW v2 {} main {}
SQLITE_UPDATE sqlite_master type main {}
SQLITE_UPDATE sqlite_master name main {}
SQLITE_UPDATE sqlite_master tbl_name main {}
SQLITE_UPDATE sqlite_master rootpage main {}
SQLITE_UPDATE sqlite_master sql main {}
SQLITE_READ sqlite_master ROWID main {}
SQLITE_READ sqlite_master name main {}
SQLITE_READ sqlite_master rootpage main {}
SQLITE_READ sqlite_master sql main {}
SQLITE_READ sqlite_master tbl_name main {}

}
do_test auth2-2.3 {
  set ::authargs {}
  db eval {
    SELECT a, b FROM v2;
  }
  set ::authargs







>



















>







98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
SQLITE_UPDATE sqlite_master rootpage main {}
SQLITE_UPDATE sqlite_master sql main {}
SQLITE_READ sqlite_master ROWID main {}
SQLITE_READ sqlite_master name main {}
SQLITE_READ sqlite_master rootpage main {}
SQLITE_READ sqlite_master sql main {}
SQLITE_READ sqlite_master tbl_name main {}
SQLITE_READ sqlite_master ROWID main {}
}
do_test auth2-2.2 {
  set ::authargs {}
  db eval {
    CREATE VIEW v2 AS SELECT x+y AS a, y+z AS b from t2;
  }
  set ::authargs
} {SQLITE_INSERT sqlite_master {} main {}
SQLITE_CREATE_VIEW v2 {} main {}
SQLITE_UPDATE sqlite_master type main {}
SQLITE_UPDATE sqlite_master name main {}
SQLITE_UPDATE sqlite_master tbl_name main {}
SQLITE_UPDATE sqlite_master rootpage main {}
SQLITE_UPDATE sqlite_master sql main {}
SQLITE_READ sqlite_master ROWID main {}
SQLITE_READ sqlite_master name main {}
SQLITE_READ sqlite_master rootpage main {}
SQLITE_READ sqlite_master sql main {}
SQLITE_READ sqlite_master tbl_name main {}
SQLITE_READ sqlite_master ROWID main {}
}
do_test auth2-2.3 {
  set ::authargs {}
  db eval {
    SELECT a, b FROM v2;
  }
  set ::authargs
Changes to test/backup.test.
900
901
902
903
904
905
906


























































907
    execsql { PRAGMA integrity_check } db3
  } {ok}

  db2 close
  db3 close
}



























































finish_test







>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>

900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
    execsql { PRAGMA integrity_check } db3
  } {ok}

  db2 close
  db3 close
}


# Test that if the database is written to via the same database handle being
# used as the source by a backup operation:
#
#   10.1.*: If the db is in-memory, the backup is restarted.
#   10.2.*: If the db is a file, the backup is not restarted.
#
db close
file delete -force test.db test.db-journal
foreach {tn file rc} {
  1 test.db  SQLITE_DONE
  2 :memory: SQLITE_OK
} {
  do_test backup-10.$tn.1 {
    sqlite3 db $file
    execsql { 
      CREATE TABLE t1(a INTEGER PRIMARY KEY, b BLOB);
      BEGIN;
        INSERT INTO t1 VALUES(NULL, randomblob(200));
        INSERT INTO t1 SELECT NULL, randomblob(200) FROM t1;
        INSERT INTO t1 SELECT NULL, randomblob(200) FROM t1;
        INSERT INTO t1 SELECT NULL, randomblob(200) FROM t1;
        INSERT INTO t1 SELECT NULL, randomblob(200) FROM t1;
        INSERT INTO t1 SELECT NULL, randomblob(200) FROM t1;
        INSERT INTO t1 SELECT NULL, randomblob(200) FROM t1;
        INSERT INTO t1 SELECT NULL, randomblob(200) FROM t1;
        INSERT INTO t1 SELECT NULL, randomblob(200) FROM t1;
      COMMIT;
      SELECT count(*) FROM t1;
    }
  } {256}

  do_test backup-10.$tn.2 {
    set pgs [execsql {pragma page_count}]
    expr {$pgs > 50 && $pgs < 75}
  } {1}

  do_test backup-10.$tn.3 {
    file delete -force bak.db bak.db-journal
    sqlite3 db2 bak.db
    sqlite3_backup B db2 main db main
    B step 50
  } {SQLITE_OK}

  do_test backup-10.$tn.4 {
    execsql { UPDATE t1 SET b = randomblob(200) WHERE a IN (1, 250) }
  } {}

  do_test backup-10.$tn.5 {
    B step 50
  } $rc

  do_test backup-10.6 {
    B finish
  } {SQLITE_OK}
}

db2 close
finish_test
Changes to test/e_fts3.test.
60
61
62
63
64
65
66








67
68
69
70
71
72
73
# [foreach] loop is testing with OOM errors, disable the lookaside buffer.
#
db close
file delete -force test.db test.db-journal
sqlite3 db test.db
if {$DO_MALLOC_TEST} { sqlite3_db_config_lookaside db 0 0 0 }
db eval "PRAGMA encoding = '$enc'"









##########################################################################
# Test the example CREATE VIRTUAL TABLE statements in section 1.1 
# of fts3.in.
#
ddl_test   1.1.1.1 {CREATE VIRTUAL TABLE data USING fts3()}
read_test  1.1.1.2 {PRAGMA table_info(data)} {0 content {} 0 {} 0}







>
>
>
>
>
>
>
>







60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
# [foreach] loop is testing with OOM errors, disable the lookaside buffer.
#
db close
file delete -force test.db test.db-journal
sqlite3 db test.db
if {$DO_MALLOC_TEST} { sqlite3_db_config_lookaside db 0 0 0 }
db eval "PRAGMA encoding = '$enc'"

proc mit {blob} {
  set scan(littleEndian) i*
  set scan(bigEndian) I*
  binary scan $blob $scan($::tcl_platform(byteOrder)) r
  return $r
}
db func mit mit

##########################################################################
# Test the example CREATE VIRTUAL TABLE statements in section 1.1 
# of fts3.in.
#
ddl_test   1.1.1.1 {CREATE VIRTUAL TABLE data USING fts3()}
read_test  1.1.1.2 {PRAGMA table_info(data)} {0 content {} 0 {} 0}
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426

















427
428
429
430
431
432
433
} {{1 0 5 7 1 0 30 7}}
read_test  1.7.1.6 { 
  SELECT offsets(mail) FROM mail WHERE mail MATCH '"serious mail"'
} {{1 0 28 7 1 1 36 4}}

ddl_test   1.7.2.1 { CREATE VIRTUAL TABLE text USING fts3() }

write_test 3.2.2 text_content {
  INSERT INTO text VALUES('
    During 30 Nov-1 Dec, 2-3oC drops. Cool in the upper portion, minimum temperature 14-16oC and cool elsewhere, minimum temperature 17-20oC. Cold to very cold on mountaintops, minimum temperature 6-12oC. Northeasterly winds 15-30 km/hr. After that, temperature increases. Northeasterly winds 15-30 km/hr.
  ');
}

read_test  1.7.2.3 {
  SELECT snippet(text) FROM text WHERE text MATCH 'cold'
} {{<b>...</b> elsewhere, minimum temperature 17-20oC. <b>Cold</b> to very <b>cold</b> on mountaintops, minimum <b>...</b>}}

read_test  1.7.2.4 {
  SELECT snippet(text, '[', ']', '...') FROM text WHERE text MATCH '"min* tem*"'
} {{... 2-3oC drops. Cool in the upper portion, [minimum] [temperature] 14-16oC and cool elsewhere, [minimum] ...}}


















##########################################################################
# Test the example in section 5 (custom tokenizers).
#
ddl_test   1.8.1.1 { CREATE VIRTUAL TABLE simple USING fts3(tokenize=simple) } 
write_test 1.8.1.2 simple_content { 
  INSERT INTO simple VALUES('Right now they''re very frustrated')







|







|



|
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>







415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
} {{1 0 5 7 1 0 30 7}}
read_test  1.7.1.6 { 
  SELECT offsets(mail) FROM mail WHERE mail MATCH '"serious mail"'
} {{1 0 28 7 1 1 36 4}}

ddl_test   1.7.2.1 { CREATE VIRTUAL TABLE text USING fts3() }

write_test 1.7.2.2 text_content {
  INSERT INTO text VALUES('
    During 30 Nov-1 Dec, 2-3oC drops. Cool in the upper portion, minimum temperature 14-16oC and cool elsewhere, minimum temperature 17-20oC. Cold to very cold on mountaintops, minimum temperature 6-12oC. Northeasterly winds 15-30 km/hr. After that, temperature increases. Northeasterly winds 15-30 km/hr.
  ');
}

read_test  1.7.2.3 {
  SELECT snippet(text) FROM text WHERE text MATCH 'cold'
} {{<b>...</b>cool elsewhere, minimum temperature 17-20oC. <b>Cold</b> to very <b>cold</b> on mountaintops, minimum temperature 6<b>...</b>}}

read_test  1.7.2.4 {
  SELECT snippet(text, '[', ']', '...') FROM text WHERE text MATCH '"min* tem*"'
} {{...the upper portion, [minimum] [temperature] 14-16oC and cool elsewhere, [minimum] [temperature] 17-20oC. Cold...}}

ddl_test   1.7.3.1 { DROP TABLE IF EXISTS t1 }
ddl_test   1.7.3.2 { CREATE VIRTUAL TABLE t1 USING fts3(a, b) }
write_test 1.7.3.3 t1_content { 
  INSERT INTO t1 VALUES(
    'transaction default models default', 'Non transaction reads');
}
write_test 1.7.3.4 t1_content { 
  INSERT INTO t1 VALUES('the default transaction', 'these semantics present');
}
write_test 1.7.3.5 t1_content { 
  INSERT INTO t1 VALUES('single request', 'default data');
}
read_test  1.7.3.6 { 
  SELECT mit(matchinfo(t1)) FROM t1 
    WHERE t1 MATCH 'default transaction "these semantics"';
} {{3 2 1 3 2 0 1 1 1 2 2 0 1 1 0 0 0 1 1 1}}

##########################################################################
# Test the example in section 5 (custom tokenizers).
#
ddl_test   1.8.1.1 { CREATE VIRTUAL TABLE simple USING fts3(tokenize=simple) } 
write_test 1.8.1.2 simple_content { 
  INSERT INTO simple VALUES('Right now they''re very frustrated')
448
449
450
451
452
453
454

455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
#-------------------------------------------------------------------------

#-------------------------------------------------------------------------
# Test that errors in the arguments passed to the snippet and offsets
# functions are handled correctly.
#
set DO_MALLOC_TEST 0

ddl_test   2.1.1 { CREATE VIRTUAL TABLE t1 USING fts3(a, b) }
write_test 2.1.2 t1_content { 
  INSERT INTO t1 VALUES('one two three', x'A1B2C3D4E5F6');
}
error_test 2.1.3 {
  SELECT offsets(a) FROM t1 WHERE a MATCH 'one'
} {illegal first argument to offsets}
error_test 2.1.4 {
  SELECT offsets(b) FROM t1 WHERE a MATCH 'one'
} {illegal first argument to offsets}
error_test 2.1.5 {
  SELECT optimize(a) FROM t1 LIMIT 1
} {illegal first argument to optimize}
error_test 2.1.6 {
  SELECT snippet(a) FROM t1 WHERE a MATCH 'one'
} {illegal first argument to snippet}
error_test 2.1.7 {
  SELECT snippet() FROM t1 WHERE a MATCH 'one'
} {unable to use function snippet in the requested context}
error_test 2.1.8 {
  SELECT snippet(a, b, 'A', 'B', 'C') FROM t1 WHERE a MATCH 'one'
} {wrong number of arguments to function snippet()}
#-------------------------------------------------------------------------

#-------------------------------------------------------------------------
# Test the effect of an OOM error while installing the FTS3 module (i.e.
# opening a database handle). This case was not tested by the OOM testing
# of the document examples above.







>




















|







473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
#-------------------------------------------------------------------------

#-------------------------------------------------------------------------
# Test that errors in the arguments passed to the snippet and offsets
# functions are handled correctly.
#
set DO_MALLOC_TEST 0
ddl_test   2.1.0 { DROP TABLE IF EXISTS t1 }
ddl_test   2.1.1 { CREATE VIRTUAL TABLE t1 USING fts3(a, b) }
write_test 2.1.2 t1_content { 
  INSERT INTO t1 VALUES('one two three', x'A1B2C3D4E5F6');
}
error_test 2.1.3 {
  SELECT offsets(a) FROM t1 WHERE a MATCH 'one'
} {illegal first argument to offsets}
error_test 2.1.4 {
  SELECT offsets(b) FROM t1 WHERE a MATCH 'one'
} {illegal first argument to offsets}
error_test 2.1.5 {
  SELECT optimize(a) FROM t1 LIMIT 1
} {illegal first argument to optimize}
error_test 2.1.6 {
  SELECT snippet(a) FROM t1 WHERE a MATCH 'one'
} {illegal first argument to snippet}
error_test 2.1.7 {
  SELECT snippet() FROM t1 WHERE a MATCH 'one'
} {unable to use function snippet in the requested context}
error_test 2.1.8 {
  SELECT snippet(a, b, 'A', 'B', 'C', 'D', 'E') FROM t1 WHERE a MATCH 'one'
} {wrong number of arguments to function snippet()}
#-------------------------------------------------------------------------

#-------------------------------------------------------------------------
# Test the effect of an OOM error while installing the FTS3 module (i.e.
# opening a database handle). This case was not tested by the OOM testing
# of the document examples above.
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
set DO_MALLOC_TEST 0
ddl_test   5.1 { CREATE VIRTUAL TABLE t5 USING fts3(x) }
write_test 5.2 t5_content {
  INSERT INTO t5 VALUES('In Xanadu did Kubla Khan A stately pleasure-dome decree Where Alph, the sacred river, ran Through caverns measureless to man Down to a sunless sea.  So twice five miles of fertile ground With walls and towers were girdled round : And there were gardens bright with sinuous rills, Where blossomed many an incense-bearing tree ; And here were forests ancient as the hills, Enfolding sunny spots of greenery.');
}
read_test 5.3 { 
  SELECT snippet(t5) FROM t5 WHERE t5 MATCH 'miles'
} {{<b>...</b> Down to a sunless sea.  So twice five <b>miles</b> of fertile ground With walls and towers were <b>...</b>}}
read_test 5.4 { 
  SELECT snippet(t5, '<i>') FROM t5 WHERE t5 MATCH 'miles'
} {{<b>...</b> Down to a sunless sea.  So twice five <i>miles</b> of fertile ground With walls and towers were <b>...</b>}}
read_test 5.5 { 
  SELECT snippet(t5, '<i>', '</i>') FROM t5 WHERE t5 MATCH 'miles'
} {{<b>...</b> Down to a sunless sea.  So twice five <i>miles</i> of fertile ground With walls and towers were <b>...</b>}}
read_test 5.6 { 
  SELECT snippet(t5, '<i>', '</i>', 'XXX') FROM t5 WHERE t5 MATCH 'miles'
} {{XXX Down to a sunless sea.  So twice five <i>miles</i> of fertile ground With walls and towers were XXX}}
#-------------------------------------------------------------------------

#-------------------------------------------------------------------------
# Test that an empty MATCH expression returns an empty result set. As
# does passing a NULL value as a MATCH expression.
#
set DO_MALLOC_TEST 0







|


|


|


|







540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
set DO_MALLOC_TEST 0
ddl_test   5.1 { CREATE VIRTUAL TABLE t5 USING fts3(x) }
write_test 5.2 t5_content {
  INSERT INTO t5 VALUES('In Xanadu did Kubla Khan A stately pleasure-dome decree Where Alph, the sacred river, ran Through caverns measureless to man Down to a sunless sea.  So twice five miles of fertile ground With walls and towers were girdled round : And there were gardens bright with sinuous rills, Where blossomed many an incense-bearing tree ; And here were forests ancient as the hills, Enfolding sunny spots of greenery.');
}
read_test 5.3 { 
  SELECT snippet(t5) FROM t5 WHERE t5 MATCH 'miles'
} {{<b>...</b>to a sunless sea.  So twice five <b>miles</b> of fertile ground With walls and towers<b>...</b>}}
read_test 5.4 { 
  SELECT snippet(t5, '<i>') FROM t5 WHERE t5 MATCH 'miles'
} {{<b>...</b>to a sunless sea.  So twice five <i>miles</b> of fertile ground With walls and towers<b>...</b>}}
read_test 5.5 { 
  SELECT snippet(t5, '<i>', '</i>') FROM t5 WHERE t5 MATCH 'miles'
} {{<b>...</b>to a sunless sea.  So twice five <i>miles</i> of fertile ground With walls and towers<b>...</b>}}
read_test 5.6 { 
  SELECT snippet(t5, '<i>', '</i>', 'XXX') FROM t5 WHERE t5 MATCH 'miles'
} {{XXXto a sunless sea.  So twice five <i>miles</i> of fertile ground With walls and towersXXX}}
#-------------------------------------------------------------------------

#-------------------------------------------------------------------------
# Test that an empty MATCH expression returns an empty result set. As
# does passing a NULL value as a MATCH expression.
#
set DO_MALLOC_TEST 0
Changes to test/fts3_common.tcl.
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
}

proc doPassiveTest {isRestart name sql catchres} {
  if {![info exists ::DO_MALLOC_TEST]} { set ::DO_MALLOC_TEST 1 }

  switch $::DO_MALLOC_TEST {
    0 { # No malloc failures.
      do_test $name [list catchsql $sql] $catchres
      return
    }
    1 { # Simulate transient failures.
      set nRepeat 1
      set zName "transient"
      set nStartLimit 100000
      set nBackup 1







|







322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
}

proc doPassiveTest {isRestart name sql catchres} {
  if {![info exists ::DO_MALLOC_TEST]} { set ::DO_MALLOC_TEST 1 }

  switch $::DO_MALLOC_TEST {
    0 { # No malloc failures.
      do_test $name [list set {} [uplevel [list catchsql $sql]]] $catchres
      return
    }
    1 { # Simulate transient failures.
      set nRepeat 1
      set zName "transient"
      set nStartLimit 100000
      set nBackup 1
Changes to test/fts3ac.test.
1127
1128
1129
1130
1131
1132
1133
1134
1135
1136
1137
1138
1139
1140
1141
1142
1143
1144
1145
1146
1147
1148
1149
1150
1151
1152
1153
1154
1155
1156
1157
1158
1159
1160
1161
1162
1163
1164
1165
1166
1167
1168
1169
1170
1171
1172
1173
1174
1175
1176
1177
1178
1179
1180
1181
1182
1183
1184
1185
1186
1187
1188
1189
1190
1191
1192
1193
1194
1195
1196
1197
1198
1199
1200
1201
1202
1203
1204
1205
1206
1207
1208
1209
1210
1211
1212
1213
  }
} {{Alert Posted 10:00 AM November 20,2000: E-<b>GAS</b> Request <b>Reminder</b>}}
do_test fts3ac-4.2 {
  execsql {
    SELECT snippet(email) FROM email
     WHERE email MATCH 'christmas candlelight'
  }
} {{<b>...</b> place.? What do you think about going here <b>Christmas</b> 
eve?? They have an 11:00 a.m. service and a <b>candlelight</b> service at 5:00 p.m., 
among others. <b>...</b>}}

do_test fts3ac-4.3 {
  execsql {
    SELECT snippet(email) FROM email
     WHERE email MATCH 'deal sheet potential reuse'
  }
} {{EOL-Accenture <b>Deal</b> <b>Sheet</b> <b>...</b> intent
     Review Enron asset base for <b>potential</b> <b>reuse</b>/ licensing
     Contract negotiations <b>...</b>}}
do_test fts3ac-4.4 {
  execsql {
    SELECT snippet(email,'<<<','>>>',' ') FROM email
     WHERE email MATCH 'deal sheet potential reuse'
  }
} {{EOL-Accenture <<<Deal>>> <<<Sheet>>>  intent
     Review Enron asset base for <<<potential>>> <<<reuse>>>/ licensing
     Contract negotiations  }}
do_test fts3ac-4.5 {
  execsql {
    SELECT snippet(email,'<<<','>>>',' ') FROM email
     WHERE email MATCH 'first things'
  }
} {{Re: <<<First>>> Polish Deal!  Congrats!  <<<Things>>> seem to be building rapidly now on the  }}
do_test fts3ac-4.6 {
  execsql {
    SELECT snippet(email) FROM email
     WHERE email MATCH 'chris is here'
  }
} {{<b>chris</b>.germany@enron.com <b>...</b> Sounds good to me.  I bet this <b>is</b> next to the Warick?? Hotel. <b>...</b> place.? What do you think about going <b>here</b> Christmas 
eve?? They have an 11:00 a.m. <b>...</b>}}
do_test fts3ac-4.7 {
  execsql {
    SELECT snippet(email) FROM email
     WHERE email MATCH '"pursuant to"'
  }
} {{Erin:

<b>Pursuant</b> <b>to</b> your request, attached are the Schedule to <b>...</b>}}
do_test fts3ac-4.8 {
  execsql {
    SELECT snippet(email) FROM email
     WHERE email MATCH 'ancillary load davis'
  }
} {{pete.<b>davis</b>@enron.com <b>...</b> Start Date: 4/22/01; HourAhead hour: 3;  No <b>ancillary</b> schedules awarded.  
Variances detected.
Variances detected in <b>Load</b> schedule.

    LOG MESSAGES:

PARSING <b>...</b>}}

# Combinations of AND and OR operators:
#
do_test fts3ac-5.1 {
  execsql {
    SELECT snippet(email) FROM email
     WHERE email MATCH 'questar enron OR com'
  }
} {{matt.smith@<b>enron</b>.<b>com</b> <b>...</b> six reports:  

31 Keystone Receipts
15 <b>Questar</b> Pipeline
40 Rockies Production
22 West_2 <b>...</b>}}
do_test fts3ac-5.2 {
  execsql {
    SELECT snippet(email) FROM email
     WHERE email MATCH 'enron OR com questar'
  }
} {{matt.smith@<b>enron</b>.<b>com</b> <b>...</b> six reports:  

31 Keystone Receipts
15 <b>Questar</b> Pipeline
40 Rockies Production
22 West_2 <b>...</b>}}

finish_test







|
|
<






|
<






|
<






|





|
|







|





|
<
<

<
<
|








|
<
<

|
|





|
<
<

<
|


1127
1128
1129
1130
1131
1132
1133
1134
1135

1136
1137
1138
1139
1140
1141
1142

1143
1144
1145
1146
1147
1148
1149

1150
1151
1152
1153
1154
1155
1156
1157
1158
1159
1160
1161
1162
1163
1164
1165
1166
1167
1168
1169
1170
1171
1172
1173
1174
1175
1176
1177


1178


1179
1180
1181
1182
1183
1184
1185
1186
1187
1188


1189
1190
1191
1192
1193
1194
1195
1196
1197


1198

1199
1200
1201
  }
} {{Alert Posted 10:00 AM November 20,2000: E-<b>GAS</b> Request <b>Reminder</b>}}
do_test fts3ac-4.2 {
  execsql {
    SELECT snippet(email) FROM email
     WHERE email MATCH 'christmas candlelight'
  }
} {{<b>...</b>here <b>Christmas</b> 
eve?? They have an 11:00 a.m. service and a <b>candlelight</b> service<b>...</b>}}


do_test fts3ac-4.3 {
  execsql {
    SELECT snippet(email) FROM email
     WHERE email MATCH 'deal sheet potential reuse'
  }
} {{EOL-Accenture <b>Deal</b> <b>Sheet</b><b>...</b>asset base for <b>potential</b> <b>reuse</b>/ licensing

     Contract negotiations <b>...</b>}}
do_test fts3ac-4.4 {
  execsql {
    SELECT snippet(email,'<<<','>>>',' ') FROM email
     WHERE email MATCH 'deal sheet potential reuse'
  }
} {{EOL-Accenture <<<Deal>>> <<<Sheet>>> asset base for <<<potential>>> <<<reuse>>>/ licensing

     Contract negotiations  }}
do_test fts3ac-4.5 {
  execsql {
    SELECT snippet(email,'<<<','>>>',' ') FROM email
     WHERE email MATCH 'first things'
  }
} {{Re: <<<First>>> Polish Deal! Congrats!  <<<Things>>> seem to be building rapidly now }}
do_test fts3ac-4.6 {
  execsql {
    SELECT snippet(email) FROM email
     WHERE email MATCH 'chris is here'
  }
} {{<b>...</b><b>chris</b>.germany@enron.com'" <<b>chris</b><b>...</b>bet this <b>is</b> next to<b>...</b>about going <b>here</b> Christmas 
eve<b>...</b>}}
do_test fts3ac-4.7 {
  execsql {
    SELECT snippet(email) FROM email
     WHERE email MATCH '"pursuant to"'
  }
} {{Erin:

<b>Pursuant</b> <b>to</b> your request, attached are the Schedule to the ISDA Master Agreement, together<b>...</b>}}
do_test fts3ac-4.8 {
  execsql {
    SELECT snippet(email) FROM email
     WHERE email MATCH 'ancillary load davis'
  }
} {{pete.<b>davis</b>@enron.com<b>...</b>3;  No <b>ancillary</b> schedules awarded<b>...</b>detected in <b>Load</b> schedule.





    LOG<b>...</b>}}

# Combinations of AND and OR operators:
#
do_test fts3ac-5.1 {
  execsql {
    SELECT snippet(email) FROM email
     WHERE email MATCH 'questar enron OR com'
  }
} {{matt.smith@<b>enron</b>.<b>com</b><b>...</b>31 Keystone Receipts


15 <b>Questar</b> Pipeline
40 Rockies<b>...</b>}}

do_test fts3ac-5.2 {
  execsql {
    SELECT snippet(email) FROM email
     WHERE email MATCH 'enron OR com questar'
  }
} {{matt.smith@<b>enron</b>.<b>com</b><b>...</b>31 Keystone Receipts


15 <b>Questar</b> Pipeline

40 Rockies<b>...</b>}}

finish_test
Changes to test/fts3al.test.
49
50
51
52
53
54
55




56
57
58
59
60
61
62
63
64
65
66
67
68
69
# exercised.  The version with a couple extra spaces should cause the
# other isspace() call to be exercised.  [Both cases have been tested
# in the debugger, but I'm hoping to continue to catch it if simple
# constant changes change things slightly.
#
# The trailing and leading hi-bit chars help with code which tests for
# isspace() to coalesce multiple spaces.





set word "\x80xxxxx\x80xxxxx\x80xxxxx\x80xxxxx\x80xxxxx\x80xxxxx\x80"
set phrase1 "$word $word $word target $word $word $word"
set phrase2 "$word $word $word    target    $word $word $word"

db eval {CREATE VIRTUAL TABLE t4 USING fts3(content)}
db eval "INSERT INTO t4 (content) VALUES ('$phrase1')"
db eval "INSERT INTO t4 (content) VALUES ('$phrase2')"

do_test fts3al-1.4 {
  execsql {SELECT rowid, length(snippet(t4)) FROM t4 WHERE t4 MATCH 'target'}
} {1 111 2 117}

finish_test







>
>
>
>











|


49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
# exercised.  The version with a couple extra spaces should cause the
# other isspace() call to be exercised.  [Both cases have been tested
# in the debugger, but I'm hoping to continue to catch it if simple
# constant changes change things slightly.
#
# The trailing and leading hi-bit chars help with code which tests for
# isspace() to coalesce multiple spaces.
#
# UPDATE: The above is no longer true; there is no such code in fts3.
# But leave the test in just the same.
# 

set word "\x80xxxxx\x80xxxxx\x80xxxxx\x80xxxxx\x80xxxxx\x80xxxxx\x80"
set phrase1 "$word $word $word target $word $word $word"
set phrase2 "$word $word $word    target    $word $word $word"

db eval {CREATE VIRTUAL TABLE t4 USING fts3(content)}
db eval "INSERT INTO t4 (content) VALUES ('$phrase1')"
db eval "INSERT INTO t4 (content) VALUES ('$phrase2')"

do_test fts3al-1.4 {
  execsql {SELECT rowid, length(snippet(t4)) FROM t4 WHERE t4 MATCH 'target'}
} {1 241 2 247}

finish_test
Changes to test/fts3near.test.
72
73
74
75
76
77
78











79
80
81
82
83
84
85
do_test fts3near-1.14 {
  execsql {SELECT docid FROM t1 WHERE content MATCH 'four NEAR four'}
} {} 
do_test fts3near-1.15 {
  execsql {SELECT docid FROM t1 WHERE content MATCH 'one NEAR two NEAR one'}
} {3} 













# Output format of the offsets() function:
#
#     <column number> <term number> <starting offset> <number of bytes>
#
db eval {
  INSERT INTO t1(content) VALUES('A X B C D A B');







>
>
>
>
>
>
>
>
>
>
>







72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
do_test fts3near-1.14 {
  execsql {SELECT docid FROM t1 WHERE content MATCH 'four NEAR four'}
} {} 
do_test fts3near-1.15 {
  execsql {SELECT docid FROM t1 WHERE content MATCH 'one NEAR two NEAR one'}
} {3} 

do_test fts3near-1.16 {
  execsql {
    SELECT docid FROM t1 WHERE content MATCH '"one three" NEAR/0 "four five"'
  }
} {1} 
do_test fts3near-1.17 {
  execsql {
    SELECT docid FROM t1 WHERE content MATCH '"four five" NEAR/0 "one three"'
  }
} {1} 


# Output format of the offsets() function:
#
#     <column number> <term number> <starting offset> <number of bytes>
#
db eval {
  INSERT INTO t1(content) VALUES('A X B C D A B');
150
151
152
153
154
155
156

157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
  execsql {SELECT offsets(t1) FROM t1 WHERE content MATCH 'two NEAR/2 three'}
} {{0 0 4 3 0 1 8 5 0 0 14 3 0 1 27 5}}
do_test fts3near-3.6 {
  execsql {
    SELECT offsets(t1) FROM t1 WHERE content MATCH 'three NEAR/0 "two four"'
  }
} {{0 0 8 5 0 1 14 3 0 2 18 4}}

do_test fts3near-3.7 {
  execsql {
    SELECT offsets(t1) FROM t1 WHERE content MATCH '"two four" NEAR/0 three'}
} {{0 2 8 5 0 0 14 3 0 1 18 4}}

db eval {
  INSERT INTO t1(content) VALUES('
    This specification defines Cascading Style Sheets, level 2 (CSS2). CSS2 is a style sheet language that allows authors and users to attach style (e.g., fonts, spacing, and aural cues) to structured documents (e.g., HTML documents and XML applications). By separating the presentation style of documents from the content of documents, CSS2 simplifies Web authoring and site maintenance.

    CSS2 builds on CSS1 (see [CSS1]) and, with very few exceptions, all valid CSS1 style sheets are valid CSS2 style sheets. CSS2 supports media-specific style sheets so that authors may tailor the presentation of their documents to visual browsers, aural devices, printers, braille devices, handheld devices, etc. This specification also supports content positioning, downloadable fonts, table layout, features for internationalization, automatic counters and numbering, and some properties related to user interface.
  ') 
}
do_test fts3near-4.1 {
  execsql {
    SELECT snippet(t1) FROM t1 WHERE content MATCH 'specification NEAR supports'
  }
} {{<b>...</b> devices, handheld devices, etc. This <b>specification</b> also <b>supports</b> content positioning, downloadable fonts, <b>...</b>}}

do_test fts3near-5.1 {
  execsql {
    SELECT docid FROM t1 WHERE content MATCH 'specification attach'
  }
} {2}
do_test fts3near-5.2 {







>
















|







161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
  execsql {SELECT offsets(t1) FROM t1 WHERE content MATCH 'two NEAR/2 three'}
} {{0 0 4 3 0 1 8 5 0 0 14 3 0 1 27 5}}
do_test fts3near-3.6 {
  execsql {
    SELECT offsets(t1) FROM t1 WHERE content MATCH 'three NEAR/0 "two four"'
  }
} {{0 0 8 5 0 1 14 3 0 2 18 4}}
breakpoint
do_test fts3near-3.7 {
  execsql {
    SELECT offsets(t1) FROM t1 WHERE content MATCH '"two four" NEAR/0 three'}
} {{0 2 8 5 0 0 14 3 0 1 18 4}}

db eval {
  INSERT INTO t1(content) VALUES('
    This specification defines Cascading Style Sheets, level 2 (CSS2). CSS2 is a style sheet language that allows authors and users to attach style (e.g., fonts, spacing, and aural cues) to structured documents (e.g., HTML documents and XML applications). By separating the presentation style of documents from the content of documents, CSS2 simplifies Web authoring and site maintenance.

    CSS2 builds on CSS1 (see [CSS1]) and, with very few exceptions, all valid CSS1 style sheets are valid CSS2 style sheets. CSS2 supports media-specific style sheets so that authors may tailor the presentation of their documents to visual browsers, aural devices, printers, braille devices, handheld devices, etc. This specification also supports content positioning, downloadable fonts, table layout, features for internationalization, automatic counters and numbering, and some properties related to user interface.
  ') 
}
do_test fts3near-4.1 {
  execsql {
    SELECT snippet(t1) FROM t1 WHERE content MATCH 'specification NEAR supports'
  }
} {{<b>...</b>braille devices, handheld devices, etc. This <b>specification</b> also <b>supports</b> content positioning, downloadable fonts, table layout<b>...</b>}}

do_test fts3near-5.1 {
  execsql {
    SELECT docid FROM t1 WHERE content MATCH 'specification attach'
  }
} {2}
do_test fts3near-5.2 {
Changes to test/fts3query.test.
95
96
97
98
99
100
101
102
103
104
105
  binary scan $blob $scan($::tcl_platform(byteOrder)) r
  return $r
}
db func mit mit

do_test fts3query-3.3 {
  execsql { SELECT mit(matchinfo(foobar)) FROM foobar WHERE foobar MATCH 'the' }
} {{1 1 3 3}}

finish_test








|



95
96
97
98
99
100
101
102
103
104
105
  binary scan $blob $scan($::tcl_platform(byteOrder)) r
  return $r
}
db func mit mit

do_test fts3query-3.3 {
  execsql { SELECT mit(matchinfo(foobar)) FROM foobar WHERE foobar MATCH 'the' }
} {{1 1 3 3 1}}

finish_test

Changes to test/fts3rnd.test.
155
156
157
158
159
160
161


162
163
164





165
166
167
168
169
170
171
172
173

174




175




176
177
178
179
180
181
182
183
184
185
186
187
    }
  }

  #lsort -uniq -integer $ret
  set ret
}



proc simple_token_matchinfo {zToken} {
  set total(0) 0
  set total(1) 0





  set total(2) 0

  foreach key [lsort -integer [array names ::t1]] {
    set value $::t1($key)
    set cnt [list]
    foreach i {0 1 2} col $value {
      set n [llength [lsearch -all $col $zToken]]
      lappend cnt $n
      incr total($i) $n

    }




    if {[lindex [lsort $cnt] end]} {




      lappend ret $key [concat 1 3 XXX $cnt]
    }
  }
  
  string map [list XXX "$total(0) $total(1) $total(2)"] $ret
} 

proc simple_near {termlist nNear} {
  set ret [list]

  foreach {key value} [array get ::t1] {
    foreach v $value {







>
>

|
|
>
>
>
>
>
|

|

|

|
|
|
>

>
>
>
>
|
>
>
>
>
|



|







155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
    }
  }

  #lsort -uniq -integer $ret
  set ret
}

# This [proc] is used to test the FTS3 matchinfo() function.
# 
proc simple_token_matchinfo {zToken} {

  set nDoc(0) 0
  set nDoc(1) 0
  set nDoc(2) 0
  set nHit(0) 0
  set nHit(1) 0
  set nHit(2) 0


  foreach key [array names ::t1] {
    set value $::t1($key)
    set a($key) [list]
    foreach i {0 1 2} col $value {
      set hit [llength [lsearch -all $col $zToken]]
      lappend a($key) $hit
      incr nHit($i) $hit
      if {$hit>0} { incr nDoc($i) }
    }
  }

  set ret [list]
  foreach docid [lsort -integer [array names a]] {
    if { [lindex [lsort -integer $a($docid)] end] } {
      set matchinfo [list 1 3]
      foreach i {0 1 2} hit $a($docid) {
        lappend matchinfo $hit $nHit($i) $nDoc($i)
      }
      lappend ret $docid $matchinfo
    }
  }
  
  set ret
} 

proc simple_near {termlist nNear} {
  set ret [list]

  foreach {key value} [array get ::t1] {
    foreach v $value {
Added test/fts3snippet.test.










































































































































































































































































































































































































































































































































































































































































































































































































































































































>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
# 2010 January 07
#
# The author disclaims copyright to this source code.  In place of
# a legal notice, here is a blessing:
#
#    May you do good and not evil.
#    May you find forgiveness for yourself and forgive others.
#    May you share freely, never taking more than you give.
#
#*************************************************************************
#
# The tests in this file test the FTS3 auxillary functions offsets(), 
# snippet() and matchinfo() work. At time of writing, running this file 
# provides full coverage of fts3_snippet.c.
#

set testdir [file dirname $argv0]
source $testdir/tester.tcl

# If SQLITE_ENABLE_FTS3 is not defined, omit this file.
ifcapable !fts3 { finish_test ; return }
source $testdir/fts3_common.tcl
source $testdir/malloc_common.tcl

set sqlite_fts3_enable_parentheses 1
set DO_MALLOC_TEST 0

# Transform the list $L to its "normal" form. So that it can be compared to
# another list with the same set of elements using [string compare].
#
proc normalize {L} {
  set ret [list]
  foreach l $L {lappend ret $l}
  return $ret
}

proc do_offsets_test {name expr args} {
  set result [list]
  foreach a $args {
    lappend result [normalize $a]
  }
  do_select_test $name {
    SELECT offsets(ft) FROM ft WHERE ft MATCH $expr
  } $result
}
  
# Document text used by a few tests. Contains the English names of all
# integers between 1 and 300.
#
set numbers [normalize {
  one two three four five six seven eight nine ten eleven twelve thirteen
  fourteen fifteen sixteen seventeen eighteen nineteen twenty twentyone
  twentytwo twentythree twentyfour twentyfive twentysix twentyseven
  twentyeight twentynine thirty thirtyone thirtytwo thirtythree thirtyfour
  thirtyfive thirtysix thirtyseven thirtyeight thirtynine forty fortyone
  fortytwo fortythree fortyfour fortyfive fortysix fortyseven fortyeight
  fortynine fifty fiftyone fiftytwo fiftythree fiftyfour fiftyfive fiftysix
  fiftyseven fiftyeight fiftynine sixty sixtyone sixtytwo sixtythree sixtyfour
  sixtyfive sixtysix sixtyseven sixtyeight sixtynine seventy seventyone
  seventytwo seventythree seventyfour seventyfive seventysix seventyseven
  seventyeight seventynine eighty eightyone eightytwo eightythree eightyfour
  eightyfive eightysix eightyseven eightyeight eightynine ninety ninetyone
  ninetytwo ninetythree ninetyfour ninetyfive ninetysix ninetyseven
  ninetyeight ninetynine onehundred onehundredone onehundredtwo
  onehundredthree onehundredfour onehundredfive onehundredsix onehundredseven
  onehundredeight onehundrednine onehundredten onehundredeleven
  onehundredtwelve onehundredthirteen onehundredfourteen onehundredfifteen
  onehundredsixteen onehundredseventeen onehundredeighteen onehundrednineteen
  onehundredtwenty onehundredtwentyone onehundredtwentytwo
  onehundredtwentythree onehundredtwentyfour onehundredtwentyfive
  onehundredtwentysix onehundredtwentyseven onehundredtwentyeight
  onehundredtwentynine onehundredthirty onehundredthirtyone
  onehundredthirtytwo onehundredthirtythree onehundredthirtyfour
  onehundredthirtyfive onehundredthirtysix onehundredthirtyseven
  onehundredthirtyeight onehundredthirtynine onehundredforty
  onehundredfortyone onehundredfortytwo onehundredfortythree
  onehundredfortyfour onehundredfortyfive onehundredfortysix
  onehundredfortyseven onehundredfortyeight onehundredfortynine
  onehundredfifty onehundredfiftyone onehundredfiftytwo onehundredfiftythree
  onehundredfiftyfour onehundredfiftyfive onehundredfiftysix
  onehundredfiftyseven onehundredfiftyeight onehundredfiftynine
  onehundredsixty onehundredsixtyone onehundredsixtytwo onehundredsixtythree
  onehundredsixtyfour onehundredsixtyfive onehundredsixtysix
  onehundredsixtyseven onehundredsixtyeight onehundredsixtynine
  onehundredseventy onehundredseventyone onehundredseventytwo
  onehundredseventythree onehundredseventyfour onehundredseventyfive
  onehundredseventysix onehundredseventyseven onehundredseventyeight
  onehundredseventynine onehundredeighty onehundredeightyone
  onehundredeightytwo onehundredeightythree onehundredeightyfour
  onehundredeightyfive onehundredeightysix onehundredeightyseven
  onehundredeightyeight onehundredeightynine onehundredninety
  onehundredninetyone onehundredninetytwo onehundredninetythree
  onehundredninetyfour onehundredninetyfive onehundredninetysix
  onehundredninetyseven onehundredninetyeight onehundredninetynine twohundred
  twohundredone twohundredtwo twohundredthree twohundredfour twohundredfive
  twohundredsix twohundredseven twohundredeight twohundrednine twohundredten
  twohundredeleven twohundredtwelve twohundredthirteen twohundredfourteen
  twohundredfifteen twohundredsixteen twohundredseventeen twohundredeighteen
  twohundrednineteen twohundredtwenty twohundredtwentyone twohundredtwentytwo
  twohundredtwentythree twohundredtwentyfour twohundredtwentyfive
  twohundredtwentysix twohundredtwentyseven twohundredtwentyeight
  twohundredtwentynine twohundredthirty twohundredthirtyone
  twohundredthirtytwo twohundredthirtythree twohundredthirtyfour
  twohundredthirtyfive twohundredthirtysix twohundredthirtyseven
  twohundredthirtyeight twohundredthirtynine twohundredforty
  twohundredfortyone twohundredfortytwo twohundredfortythree
  twohundredfortyfour twohundredfortyfive twohundredfortysix
  twohundredfortyseven twohundredfortyeight twohundredfortynine
  twohundredfifty twohundredfiftyone twohundredfiftytwo twohundredfiftythree
  twohundredfiftyfour twohundredfiftyfive twohundredfiftysix
  twohundredfiftyseven twohundredfiftyeight twohundredfiftynine
  twohundredsixty twohundredsixtyone twohundredsixtytwo twohundredsixtythree
  twohundredsixtyfour twohundredsixtyfive twohundredsixtysix
  twohundredsixtyseven twohundredsixtyeight twohundredsixtynine
  twohundredseventy twohundredseventyone twohundredseventytwo
  twohundredseventythree twohundredseventyfour twohundredseventyfive
  twohundredseventysix twohundredseventyseven twohundredseventyeight
  twohundredseventynine twohundredeighty twohundredeightyone
  twohundredeightytwo twohundredeightythree twohundredeightyfour
  twohundredeightyfive twohundredeightysix twohundredeightyseven
  twohundredeightyeight twohundredeightynine twohundredninety
  twohundredninetyone twohundredninetytwo twohundredninetythree
  twohundredninetyfour twohundredninetyfive twohundredninetysix
  twohundredninetyseven twohundredninetyeight twohundredninetynine
  threehundred
}]

foreach {DO_MALLOC_TEST enc} {
  0 utf8
  1 utf8
  1 utf16
} {

  db close
  file delete -force test.db
  sqlite3 db test.db
  db eval "PRAGMA encoding = \"$enc\""

  # Set variable $T to the test name prefix for this iteration of the loop.
  #
  set T "fts3snippet-$enc"

  ##########################################################################
  # Test the offset function.
  #
  do_test $T.1.1 {
    execsql {
      CREATE VIRTUAL TABLE ft USING fts3;
      INSERT INTO ft VALUES('xxx xxx xxx xxx');
    }
  } {}
  do_offsets_test $T.1.2 {xxx} {0 0 0 3 0 0 4 3 0 0 8 3 0 0 12 3}
  do_offsets_test $T.1.3 {"xxx xxx"} {
      0 0  0 3     0 0  4 3     0 1  4 3     0 0  8 3 
      0 1  8 3     0 1 12 3
  }
  do_offsets_test $T.1.4 {"xxx xxx" xxx} {
      0 0  0 3     0 2  0 3     0 0  4 3     0 1  4 3 
      0 2  4 3     0 0  8 3     0 1  8 3     0 2  8 3 
      0 1 12 3     0 2 12 3
  }
  do_offsets_test $T.1.5 {xxx "xxx xxx"} {
      0 0  0 3     0 1  0 3     0 0  4 3     0 1  4 3 
      0 2  4 3     0 0  8 3     0 1  8 3     0 2  8 3 
      0 0 12 3     0 2 12 3
  }

  do_test $T.2.1 {
    set v1 [lrange $numbers 0 99]
    execsql {
      DROP TABLE IF EXISTS ft;
      CREATE VIRTUAL TABLE ft USING fts3(a, b);
      INSERT INTO ft VALUES($v1, $numbers);
      INSERT INTO ft VALUES($v1, NULL);
    }
  } {}

  set off [string first "twohundred " $numbers]
  do_offsets_test $T.2.1 {twohundred} [list 1 0 $off 10]

  set off [string first "onehundred " $numbers]
  do_offsets_test $T.2.2 {onehundred} \
    [list 0 0 $off 10 1 0 $off 10] [list 0 0 $off 10]

  # Test a corruption case:
  execsql { UPDATE ft_content SET c1b = 'hello world' WHERE c1b = $numbers }
  do_error_test $T.2.3 {
    SELECT offsets(ft) FROM ft WHERE ft MATCH 'onehundred'
  } {database disk image is malformed}
  
  ##########################################################################
  # Test the snippet function.
  #
  proc do_snippet_test {name expr iCol nTok args} {
    set res [list]
    foreach a $args { lappend res [string trim $a] }
    do_select_test $name {
      SELECT snippet(ft,'{','}','...',$iCol,$nTok) FROM ft WHERE ft MATCH $expr
    } $res
  }
  do_test $T.3.1 {
    execsql {
      DROP TABLE IF EXISTS ft;
      CREATE VIRTUAL TABLE ft USING fts3;
      INSERT INTO ft VALUES('one two three four five six seven eight nine ten');
    }
  } {}
  do_snippet_test $T.3.2  one    0 5 "{one} two three four five..."
  do_snippet_test $T.3.3  two    0 5 "one {two} three four five..."
  do_snippet_test $T.3.4  three  0 5 "one two {three} four five..."
  do_snippet_test $T.3.5  four   0 5 "...two three {four} five six..."
  do_snippet_test $T.3.6  five   0 5 "...three four {five} six seven..."
  do_snippet_test $T.3.7  six    0 5 "...four five {six} seven eight..."
  do_snippet_test $T.3.8  seven  0 5 "...five six {seven} eight nine..."
  do_snippet_test $T.3.9  eight  0 5 "...six seven {eight} nine ten"
  do_snippet_test $T.3.10 nine   0 5 "...six seven eight {nine} ten"
  do_snippet_test $T.3.11 ten    0 5 "...six seven eight nine {ten}"
  
  do_test $T.4.1 {
    execsql {
      INSERT INTO ft VALUES(
           'one two three four five '
        || 'six seven eight nine ten '
        || 'eleven twelve thirteen fourteen fifteen '
        || 'sixteen seventeen eighteen nineteen twenty '
        || 'one two three four five '
        || 'six seven eight nine ten '
        || 'eleven twelve thirteen fourteen fifteen '
        || 'sixteen seventeen eighteen nineteen twenty'
      );
    }
  } {}
  
  do_snippet_test $T.4.2 {one nine} 0 5 {
     {one} two three...eight {nine} ten
  } {
     {one} two three...eight {nine} ten...
  }
  
  do_snippet_test $T.4.3 {one nine} 0 -5 {
     {one} two three four five...six seven eight {nine} ten
  } {
     {one} two three four five...seven eight {nine} ten eleven...
  }
  do_snippet_test $T.4.3 {one nineteen} 0 -5 {
     ...eighteen {nineteen} twenty {one} two...
  }
  do_snippet_test $T.4.4 {two nineteen} 0 -5 {
     ...eighteen {nineteen} twenty one {two}...
  }
  do_snippet_test $T.4.5 {three nineteen} 0 -5 {
     ...{nineteen} twenty one two {three}...
  }
  
  do_snippet_test $T.4.6 {four nineteen} 0 -5 {
     ...two three {four} five six...seventeen eighteen {nineteen} twenty one...
  }
  do_snippet_test $T.4.7 {four NEAR nineteen} 0 -5 {
     ...seventeen eighteen {nineteen} twenty one...two three {four} five six...
  }
  
  do_snippet_test $T.4.8 {four nineteen} 0 5 {
     ...three {four} five...eighteen {nineteen} twenty...
  }
  do_snippet_test $T.4.9 {four NEAR nineteen} 0 5 {
     ...eighteen {nineteen} twenty...three {four} five...
  }
  do_snippet_test $T.4.10 {four NEAR nineteen} 0 -5 {
     ...seventeen eighteen {nineteen} twenty one...two three {four} five six...
  }
  do_snippet_test $T.4.11 {four NOT (nineteen twentyone)} 0 5 {
     ...two three {four} five six...
  } {
     ...two three {four} five six...
  }
  do_snippet_test $T.4.12 {four OR nineteen NEAR twentyone} 0 5 {
     ...two three {four} five six...
  } {
     ...two three {four} five six...
  }
  
  do_test $T.5.1 {
    execsql {
      DROP TABLE IF EXISTS ft;
      CREATE VIRTUAL TABLE ft USING fts3(a, b, c);
      INSERT INTO ft VALUES(
        'one two three four five', 
        'four five six seven eight', 
        'seven eight nine ten eleven'
      );
    }
  } {}
  
  do_snippet_test $T.5.2 {five} -1 3 {...three four {five}}
  do_snippet_test $T.5.3 {five}  0 3 {...three four {five}}
  do_snippet_test $T.5.4 {five}  1 3 {four {five} six...}
  do_snippet_test $T.5.5 {five}  2 3 {seven eight nine...}
  
  do_test $T.5.6 {
    execsql { UPDATE ft SET b = NULL }
  } {}
  
  do_snippet_test $T.5.7  {five} -1 3 {...three four {five}}
  do_snippet_test $T.5.8  {five}  0 3 {...three four {five}}
  do_snippet_test $T.5.9  {five}  1 3 {}
  do_snippet_test $T.5.10 {five}  2 3 {seven eight nine...}
  
  do_snippet_test $T.5.11 {one "seven eight nine"} -1 -3 {
    {one} two three...{seven} {eight} {nine}...
  }

  do_test $T.6.1 {
    execsql {
      DROP TABLE IF EXISTS ft;
      CREATE VIRTUAL TABLE ft USING fts3(x);
      INSERT INTO ft VALUES($numbers);
    }
  } {}
  do_snippet_test $T.6.2 {
    one fifty onehundred onehundredfifty twohundredfifty threehundred
  } -1 4 {
    {one}...{fifty}...{onehundred}...{onehundredfifty}...
  }
  do_snippet_test $T.6.3 {
    one fifty onehundred onehundredfifty twohundredfifty threehundred
  } -1 -4 {
    {one} two three four...fortyeight fortynine {fifty} fiftyone...ninetyeight ninetynine {onehundred} onehundredone...onehundredfortyeight onehundredfortynine {onehundredfifty} onehundredfiftyone...
  }

  do_test $T.7.1 {
    execsql {
      BEGIN;
        DROP TABLE IF EXISTS ft;
        CREATE VIRTUAL TABLE ft USING fts3(x);
    }
    set testresults [list]
    for {set i 1} {$i < 150} {incr i} {
      set commas [string repeat , $i]
      execsql {INSERT INTO ft VALUES('one' || $commas || 'two')}
      lappend testresults "{one}$commas{two}"
    }
    execsql COMMIT
  } {}
  do_snippet_test $T.7.2 {one two} -1 3 {*}$testresults
  
  ##########################################################################
  # Test the matchinfo function.
  #
  proc mit {blob} {
    set scan(littleEndian) i*
    set scan(bigEndian) I*
    binary scan $blob $scan($::tcl_platform(byteOrder)) r
    return $r
  }
  db func mit mit
  proc do_matchinfo_test {name expr args} {
    set res [list]
    foreach a $args { lappend res [normalize $a] }
    do_select_test $name {
      SELECT mit(matchinfo(ft)) FROM ft WHERE ft MATCH $expr
    } $res
  }
  do_test $T.8.1 {
    set ten {one two three four five six seven eight nine ten}
    execsql {
      DROP TABLE IF EXISTS ft;
      CREATE VIRTUAL TABLE ft USING fts3;
      INSERT INTO ft VALUES($ten);
      INSERT INTO ft VALUES($ten || ' ' || $ten);
    }
  } {}
  
  do_matchinfo_test $T.8.2 "one" {1 1  1 3 2} {1 1  2 3 2}
  do_matchinfo_test $T.8.3 "one NEAR/3 ten" {2 1  1 1 1 1 1 1}
  do_matchinfo_test $T.8.4 "five NEAR/4 ten" \
    {2 1  1 3 2  1 3 2} {2 1  2 3 2  2 3 2}
  do_matchinfo_test $T.8.5 "six NEAR/3 ten NEAR/3 two" \
    {3 1  1 1 1  1 1 1  1 1 1}
  do_matchinfo_test $T.8.6 "five NEAR/4 ten NEAR/3 two" \
    {3 1  2 2 1  1 1 1  1 1 1}

  do_test $T.9.1 {
    execsql {
      DROP TABLE IF EXISTS ft;
      CREATE VIRTUAL TABLE ft USING fts3(x, y);
    }
    foreach n {1 2 3} {
      set v1 [lrange $numbers 0 [expr $n*100]]
      set v2 [string trim [string repeat "$numbers " $n]]
      set docid [expr $n * 1000000]
      execsql { INSERT INTO ft(docid, x, y) VALUES($docid, $v1, $v2) }
    }
  } {}
  do_matchinfo_test $T.9.2 {two*}     \
    { 1 2    1   105 3   101 606 3}   \
    { 1 2    3   105 3   202 606 3}   \
    { 1 2    101 105 3   303 606 3}

  do_matchinfo_test $T.9.4 {"one* two*"}  \
    { 1 2    1 5 3   2 12 3}              \
    { 1 2    2 5 3   4 12 3}              \
    { 1 2    2 5 3   6 12 3}

  do_matchinfo_test $T.9.5 {twohundredfifty}  \
    { 1 2    0 1 1   1 6 3}                   \
    { 1 2    0 1 1   2 6 3}                   \
    { 1 2    1 1 1   3 6 3}

  do_matchinfo_test $T.9.6 {"threehundred one"} \
    { 1 2    0 0 0   1 3 2}                     \
    { 1 2    0 0 0   2 3 2}

  do_matchinfo_test $T.9.7 {one OR fivehundred} \
    { 2 2    1 3 3   1 6 3   0 0 0   0 0 0 }    \
    { 2 2    1 3 3   2 6 3   0 0 0   0 0 0 }    \
    { 2 2    1 3 3   3 6 3   0 0 0   0 0 0 }

  do_matchinfo_test $T.9.8 {two OR "threehundred one"} \
    { 2 2    1 3 3   1 6 3   0 0 0   0 3 2 }           \
    { 2 2    1 3 3   2 6 3   0 0 0   1 3 2 }           \
    { 2 2    1 3 3   3 6 3   0 0 0   2 3 2 }

  do_select_test $T.9.9 {
    SELECT mit(matchinfo(ft)), mit(matchinfo(ft))
    FROM ft WHERE ft MATCH 'two OR "threehundred one"' 
  } [normalize {
    {2 2 1 3 3 1 6 3 0 0 0 0 3 2}
    {2 2 1 3 3 1 6 3 0 0 0 0 3 2}
    {2 2 1 3 3 2 6 3 0 0 0 1 3 2}
    {2 2 1 3 3 2 6 3 0 0 0 1 3 2}
    {2 2 1 3 3 3 6 3 0 0 0 2 3 2}          
    {2 2 1 3 3 3 6 3 0 0 0 2 3 2}
  }]
}

set sqlite_fts3_enable_parentheses 0
finish_test
Changes to test/insert4.test.
297
298
299
300
301
302
303





304


















305
306
    CREATE INDEX t2_i1 ON t2(x ASC, y COLLATE RTRIM);
    INSERT INTO t2 SELECT * FROM t3;
  }
  set ::sqlite3_xferopt_count
} {0}



























finish_test







>
>
>
>
>
|
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>


297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
    CREATE INDEX t2_i1 ON t2(x ASC, y COLLATE RTRIM);
    INSERT INTO t2 SELECT * FROM t3;
  }
  set ::sqlite3_xferopt_count
} {0}


do_test insert4-6.5 {
  execsql {
    CREATE TABLE t6a(x CHECK( x<>'abc' ));
    INSERT INTO t6a VALUES('ABC');
    SELECT * FROM t6a;
  }
} {ABC}
do_test insert4-6.6 {
  execsql {
    CREATE TABLE t6b(x CHECK( x<>'abc' COLLATE nocase ));
  }
  catchsql {
    INSERT INTO t6b SELECT * FROM t6a;
  }
} {1 {constraint failed}}
do_test insert4-6.7 {
  execsql {
    DROP TABLE t6b;
    CREATE TABLE t6b(x CHECK( x COLLATE nocase <>'abc' ));
  }
  catchsql {
    INSERT INTO t6b SELECT * FROM t6a;
  }
} {1 {constraint failed}}

finish_test
Changes to test/minmax3.test.
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205

































206
207
208


209
210
211







212
213
214


215





















216
217
do_test minmax3-2.7 {
  execsql { SELECT min(b) FROM t2 WHERE a = 1 AND b<1; }
} {{}}
do_test minmax3-2.8 {
  execsql { SELECT min(b) FROM t2 WHERE a = 3 AND b<1; }
} {{}}

do_test minmax3-2.1 {
  execsql {
    DROP TABLE t2;
    CREATE TABLE t2(a, b);
    CREATE INDEX i3 ON t2(a, b DESC);
    INSERT INTO t2 VALUES(1, NULL);
    INSERT INTO t2 VALUES(1, 1);
    INSERT INTO t2 VALUES(1, 2);
    INSERT INTO t2 VALUES(1, 3);
    INSERT INTO t2 VALUES(2, NULL);
    INSERT INTO t2 VALUES(2, 1);
    INSERT INTO t2 VALUES(2, 2);
    INSERT INTO t2 VALUES(2, 3);
    INSERT INTO t2 VALUES(3, 1);
    INSERT INTO t2 VALUES(3, 2);
    INSERT INTO t2 VALUES(3, 3);
  }
} {}
do_test minmax3-2.2 {
  execsql { SELECT min(b) FROM t2 WHERE a = 1; }
} {1}
do_test minmax3-2.3 {
  execsql { SELECT min(b) FROM t2 WHERE a = 1 AND b>1; }
} {2}
do_test minmax3-2.4 {
  execsql { SELECT min(b) FROM t2 WHERE a = 1 AND b>-1; }
} {1}
do_test minmax3-2.5 {
  execsql { SELECT min(b) FROM t2 WHERE a = 1; }

































} {1}
do_test minmax3-2.6 {
  execsql { SELECT min(b) FROM t2 WHERE a = 1 AND b<2; }


} {1}
do_test minmax3-2.7 {
  execsql { SELECT min(b) FROM t2 WHERE a = 1 AND b<1; }







} {{}}
do_test minmax3-2.8 {
  execsql { SELECT min(b) FROM t2 WHERE a = 3 AND b<1; }


} {{}}






















finish_test







|

















|


|


|


|

>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
|
|
|
>
>
|
|
|
>
>
>
>
>
>
>
|
|
|
>
>
|
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>


170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
do_test minmax3-2.7 {
  execsql { SELECT min(b) FROM t2 WHERE a = 1 AND b<1; }
} {{}}
do_test minmax3-2.8 {
  execsql { SELECT min(b) FROM t2 WHERE a = 3 AND b<1; }
} {{}}

do_test minmax3-3.1 {
  execsql {
    DROP TABLE t2;
    CREATE TABLE t2(a, b);
    CREATE INDEX i3 ON t2(a, b DESC);
    INSERT INTO t2 VALUES(1, NULL);
    INSERT INTO t2 VALUES(1, 1);
    INSERT INTO t2 VALUES(1, 2);
    INSERT INTO t2 VALUES(1, 3);
    INSERT INTO t2 VALUES(2, NULL);
    INSERT INTO t2 VALUES(2, 1);
    INSERT INTO t2 VALUES(2, 2);
    INSERT INTO t2 VALUES(2, 3);
    INSERT INTO t2 VALUES(3, 1);
    INSERT INTO t2 VALUES(3, 2);
    INSERT INTO t2 VALUES(3, 3);
  }
} {}
do_test minmax3-3.2 {
  execsql { SELECT min(b) FROM t2 WHERE a = 1; }
} {1}
do_test minmax3-3.3 {
  execsql { SELECT min(b) FROM t2 WHERE a = 1 AND b>1; }
} {2}
do_test minmax3-3.4 {
  execsql { SELECT min(b) FROM t2 WHERE a = 1 AND b>-1; }
} {1}
do_test minmax3-3.5 {
  execsql { SELECT min(b) FROM t2 WHERE a = 1; }
} {1}
do_test minmax3-3.6 {
  execsql { SELECT min(b) FROM t2 WHERE a = 1 AND b<2; }
} {1}
do_test minmax3-3.7 {
  execsql { SELECT min(b) FROM t2 WHERE a = 1 AND b<1; }
} {{}}
do_test minmax3-3.8 {
  execsql { SELECT min(b) FROM t2 WHERE a = 3 AND b<1; }
} {{}}

do_test minmax3-4.1 {
  execsql {
    CREATE TABLE t4(x);
    INSERT INTO t4 VALUES('abc');
    INSERT INTO t4 VALUES('BCD');
    SELECT max(x) FROM t4;
  }
} {abc}
do_test minmax3-4.2 {
  execsql {
    SELECT max(x COLLATE nocase) FROM t4;
  }
} {BCD}
do_test minmax3-4.3 {
  execsql {
    SELECT max(x), max(x COLLATE nocase) FROM t4;
  }
} {abc BCD}
do_test minmax3-4.4 {
  execsql {
    SELECT max(x COLLATE binary), max(x COLLATE nocase) FROM t4;
  }
} {abc BCD}
do_test minmax3-4.5 {
  execsql {
    SELECT max(x COLLATE nocase), max(x COLLATE rtrim) FROM t4;
  }
} {BCD abc}
do_test minmax3-4.6 {
  execsql {
    SELECT max(x COLLATE nocase), max(x) FROM t4;
  }
} {BCD abc}
do_test minmax3-4.10 {
  execsql {
    SELECT min(x) FROM t4;
  }
} {BCD}
do_test minmax3-4.11 {
  execsql {
    SELECT min(x COLLATE nocase) FROM t4;
  }
} {abc}
do_test minmax3-4.12 {
  execsql {
    SELECT min(x), min(x COLLATE nocase) FROM t4;
  }
} {BCD abc}
do_test minmax3-4.13 {
  execsql {
    SELECT min(x COLLATE binary), min(x COLLATE nocase) FROM t4;
  }
} {BCD abc}
do_test minmax3-4.14 {
  execsql {
    SELECT min(x COLLATE nocase), min(x COLLATE rtrim) FROM t4;
  }
} {abc BCD}
do_test minmax3-4.15 {
  execsql {
    SELECT min(x COLLATE nocase), min(x) FROM t4;
  }
} {abc BCD}


finish_test
Changes to test/whereA.test.
29
30
31
32
33
34
35










36
37
38
39
40




















41
42
43
44
45
46
47
  db eval {
    PRAGMA reverse_unordered_selects=1;
    SELECT * FROM t1;
  }
} {3 4.53 {} 2 hello world 1 2 3}

do_test whereA-1.3 {










  db eval {
    PRAGMA reverse_unordered_selects=1;
    SELECT * FROM t1 ORDER BY rowid;
  }
} {1 2 3 2 hello world 3 4.53 {}}





















do_test whereA-2.1 {
  db eval {
    PRAGMA reverse_unordered_selects=0;
    SELECT * FROM t1 WHERE a>0;
  }
} {1 2 3 2 hello world 3 4.53 {}}







>
>
>
>
>
>
>
>
>
>





>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>







29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
  db eval {
    PRAGMA reverse_unordered_selects=1;
    SELECT * FROM t1;
  }
} {3 4.53 {} 2 hello world 1 2 3}

do_test whereA-1.3 {
  db close
  sqlite3 db test.db
  db eval {
    PRAGMA reverse_unordered_selects=1;
    SELECT * FROM t1;
  }
} {3 4.53 {} 2 hello world 1 2 3}
do_test whereA-1.4 {
  db close
  sqlite3 db test.db
  db eval {
    PRAGMA reverse_unordered_selects=1;
    SELECT * FROM t1 ORDER BY rowid;
  }
} {1 2 3 2 hello world 3 4.53 {}}
do_test whereA-1.5 {
  db eval {
    VACUUM;
    SELECT * FROM t1 ORDER BY rowid;
  }
} {1 2 3 2 hello world 3 4.53 {}}
do_test whereA-1.6 {
  db eval {
    PRAGMA reverse_unordered_selects;
  }
} {1}
do_test whereA-1.7 {
  db close
  sqlite3 db test.db
  db eval {
    PRAGMA reverse_unordered_selects=1;
    VACUUM;
    SELECT * FROM t1;
  }
} {3 4.53 {} 2 hello world 1 2 3}

do_test whereA-2.1 {
  db eval {
    PRAGMA reverse_unordered_selects=0;
    SELECT * FROM t1 WHERE a>0;
  }
} {1 2 3 2 hello world 3 4.53 {}}
Changes to tool/lemon.c.
405
406
407
408
409
410
411
412
















413
414
415
416
417
418
419
/********************** New code to implement the "acttab" module ***********/
/*
** This module implements routines use to construct the yy_action[] table.
*/

/*
** The state of the yy_action table under construction is an instance of
** the following structure
















*/
typedef struct acttab acttab;
struct acttab {
  int nAction;                 /* Number of used slots in aAction[] */
  int nActionAlloc;            /* Slots allocated for aAction[] */
  struct {
    int lookahead;             /* Value of the lookahead token */







|
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>







405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
/********************** New code to implement the "acttab" module ***********/
/*
** This module implements routines use to construct the yy_action[] table.
*/

/*
** The state of the yy_action table under construction is an instance of
** the following structure.
**
** The yy_action table maps the pair (state_number, lookahead) into an
** action_number.  The table is an array of integers pairs.  The state_number
** determines an initial offset into the yy_action array.  The lookahead
** value is then added to this initial offset to get an index X into the
** yy_action array. If the aAction[X].lookahead equals the value of the
** of the lookahead input, then the value of the action_number output is
** aAction[X].action.  If the lookaheads do not match then the
** default action for the state_number is returned.
**
** All actions associated with a single state_number are first entered
** into aLookahead[] using multiple calls to acttab_action().  Then the 
** actions for that single state_number are placed into the aAction[] 
** array with a single call to acttab_insert().  The acttab_insert() call
** also resets the aLookahead[] array in preparation for the next
** state number.
*/
typedef struct acttab acttab;
struct acttab {
  int nAction;                 /* Number of used slots in aAction[] */
  int nActionAlloc;            /* Slots allocated for aAction[] */
  struct {
    int lookahead;             /* Value of the lookahead token */
450
451
452
453
454
455
456
457



458
459
460
461
462
463
464
    fprintf(stderr,"Unable to allocate memory for a new acttab.");
    exit(1);
  }
  memset(p, 0, sizeof(*p));
  return p;
}

/* Add a new action to the current transaction set



*/
void acttab_action(acttab *p, int lookahead, int action){
  if( p->nLookahead>=p->nLookaheadAlloc ){
    p->nLookaheadAlloc += 25;
    p->aLookahead = realloc( p->aLookahead,
                             sizeof(p->aLookahead[0])*p->nLookaheadAlloc );
    if( p->aLookahead==0 ){







|
>
>
>







466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
    fprintf(stderr,"Unable to allocate memory for a new acttab.");
    exit(1);
  }
  memset(p, 0, sizeof(*p));
  return p;
}

/* Add a new action to the current transaction set.  
**
** This routine is called once for each lookahead for a particular
** state.
*/
void acttab_action(acttab *p, int lookahead, int action){
  if( p->nLookahead>=p->nLookaheadAlloc ){
    p->nLookaheadAlloc += 25;
    p->aLookahead = realloc( p->aLookahead,
                             sizeof(p->aLookahead[0])*p->nLookaheadAlloc );
    if( p->aLookahead==0 ){
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527


528
529
530
531
532
533
534
535



536
537
538
539
540
541
542
543
544
545





546
547



548
549
550
551
552
553
554
555
** into the current action table.  Then reset the transaction set back
** to an empty set in preparation for a new round of acttab_action() calls.
**
** Return the offset into the action table of the new transaction.
*/
int acttab_insert(acttab *p){
  int i, j, k, n;
  int nActtab;     /* Number of slots in the p->aAction[] table */
  assert( p->nLookahead>0 );

  /* Make sure we have enough space to hold the expanded action table
  ** in the worst case.  The worst case occurs if the transaction set
  ** must be appended to the current action table
  */
  n = p->mxLookahead + 1;
  nActtab = p->nAction + n;
  if( nActtab >= p->nActionAlloc ){
    int oldAlloc = p->nActionAlloc;
    p->nActionAlloc = p->nAction + n + p->nActionAlloc + 20;
    p->aAction = realloc( p->aAction,
                          sizeof(p->aAction[0])*p->nActionAlloc);
    if( p->aAction==0 ){
      fprintf(stderr,"malloc failed\n");
      exit(1);
    }
    for(i=oldAlloc; i<p->nActionAlloc; i++){
      p->aAction[i].lookahead = -1;
      p->aAction[i].action = -1;
    }
  }

  /* Scan the existing action table looking for an offset where we can
  ** insert the current transaction set.  Fall out of the loop when that
  ** offset is found.  In the worst case, we fall out of the loop when
  ** i reaches nActtab, which means we append the new transaction set.
  **
  ** i is the index in p->aAction[] where p->mnLookahead is inserted.
  */
  for(i=nActtab-1; i>=0; i--){
    /* First look for an existing action table entry that can be reused */
    if( p->aAction[i].lookahead==p->mnLookahead ){


      if( p->aAction[i].action!=p->mnAction ) continue;
      for(j=0; j<p->nLookahead; j++){
        k = p->aLookahead[j].lookahead - p->mnLookahead + i;
        if( k<0 || k>=p->nAction ) break;
        if( p->aLookahead[j].lookahead!=p->aAction[k].lookahead ) break;
        if( p->aLookahead[j].action!=p->aAction[k].action ) break;
      }
      if( j<p->nLookahead ) continue;



      n = 0;
      for(j=0; j<p->nAction; j++){
        if( p->aAction[j].lookahead<0 ) continue;
        if( p->aAction[j].lookahead==j+p->mnLookahead-i ) n++;
      }
      if( n==p->nLookahead ){
        break;  /* Same as a prior transaction set */
      }
    }
  }





  if( i<0 ){
    /* If no reusable entry is found, look for an empty slot */



    for(i=0; i<nActtab; i++){
      if( p->aAction[i].lookahead<0 ){
        for(j=0; j<p->nLookahead; j++){
          k = p->aLookahead[j].lookahead - p->mnLookahead + i;
          if( k<0 ) break;
          if( p->aAction[k].lookahead>=0 ) break;
        }
        if( j<p->nLookahead ) continue;







<







<
|














|
|
|
<



|
<

>
>








>
>
>






|



>
>
>
>
>

|
>
>
>
|







506
507
508
509
510
511
512

513
514
515
516
517
518
519

520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537

538
539
540
541

542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
** into the current action table.  Then reset the transaction set back
** to an empty set in preparation for a new round of acttab_action() calls.
**
** Return the offset into the action table of the new transaction.
*/
int acttab_insert(acttab *p){
  int i, j, k, n;

  assert( p->nLookahead>0 );

  /* Make sure we have enough space to hold the expanded action table
  ** in the worst case.  The worst case occurs if the transaction set
  ** must be appended to the current action table
  */
  n = p->mxLookahead + 1;

  if( p->nAction + n >= p->nActionAlloc ){
    int oldAlloc = p->nActionAlloc;
    p->nActionAlloc = p->nAction + n + p->nActionAlloc + 20;
    p->aAction = realloc( p->aAction,
                          sizeof(p->aAction[0])*p->nActionAlloc);
    if( p->aAction==0 ){
      fprintf(stderr,"malloc failed\n");
      exit(1);
    }
    for(i=oldAlloc; i<p->nActionAlloc; i++){
      p->aAction[i].lookahead = -1;
      p->aAction[i].action = -1;
    }
  }

  /* Scan the existing action table looking for an offset that is a 
  ** duplicate of the current transaction set.  Fall out of the loop
  ** if and when the duplicate is found.

  **
  ** i is the index in p->aAction[] where p->mnLookahead is inserted.
  */
  for(i=p->nAction-1; i>=0; i--){

    if( p->aAction[i].lookahead==p->mnLookahead ){
      /* All lookaheads and actions in the aLookahead[] transaction
      ** must match against the candidate aAction[i] entry. */
      if( p->aAction[i].action!=p->mnAction ) continue;
      for(j=0; j<p->nLookahead; j++){
        k = p->aLookahead[j].lookahead - p->mnLookahead + i;
        if( k<0 || k>=p->nAction ) break;
        if( p->aLookahead[j].lookahead!=p->aAction[k].lookahead ) break;
        if( p->aLookahead[j].action!=p->aAction[k].action ) break;
      }
      if( j<p->nLookahead ) continue;

      /* No possible lookahead value that is not in the aLookahead[]
      ** transaction is allowed to match aAction[i] */
      n = 0;
      for(j=0; j<p->nAction; j++){
        if( p->aAction[j].lookahead<0 ) continue;
        if( p->aAction[j].lookahead==j+p->mnLookahead-i ) n++;
      }
      if( n==p->nLookahead ){
        break;  /* An exact match is found at offset i */
      }
    }
  }

  /* If no existing offsets exactly match the current transaction, find an
  ** an empty offset in the aAction[] table in which we can add the
  ** aLookahead[] transaction.
  */
  if( i<0 ){
    /* Look for holes in the aAction[] table that fit the current
    ** aLookahead[] transaction.  Leave i set to the offset of the hole.
    ** If no holes are found, i is left at p->nAction, which means the
    ** transaction will be appended. */
    for(i=0; i<p->nActionAlloc - p->mxLookahead; i++){
      if( p->aAction[i].lookahead<0 ){
        for(j=0; j<p->nLookahead; j++){
          k = p->aLookahead[j].lookahead - p->mnLookahead + i;
          if( k<0 ) break;
          if( p->aAction[k].lookahead>=0 ) break;
        }
        if( j<p->nLookahead ) continue;
Added tool/restore_jrnl.tcl.


















































































































































































































































































































































































































































































>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
# 2010 January 7
#
# The author disclaims copyright to this source code.  In place of
# a legal notice, here is a blessing:
#
#    May you do good and not evil.
#    May you find forgiveness for yourself and forgive others.
#    May you share freely, never taking more than you give.
#
#***********************************************************************
# This file implements utility functions for SQLite library.
#
# This file attempts to restore the header of a journal.
# This may be useful for rolling-back the last committed 
# transaction from a recovered journal.
#

package require sqlite3

set parm_error 0
set fix_chksums 0
set dump_pages 0
set db_name ""

for {set i 0} {$i<$argc} {incr i} {
  if {[lindex $argv $i] == "-fix_chksums"} {
    set fix_chksums -1
  } elseif {[lindex $argv $i] == "-dump_pages"} {
    set dump_pages -1
  } elseif {$db_name == ""} {
    set db_name [lindex $argv $i]
    set jrnl_name $db_name-journal
  } else {
    set parm_error -1
  }
}
if {$parm_error || $db_name == ""} {
  puts "USAGE: restore_jrnl.tcl \[-fix_chksums\] \[-dump_pages\] db_name"
  puts "Example: restore_jrnl.tcl foo.sqlite"
  return
}

# is there a way to determine this?
set sectsz 512

# Copy file $from into $to
#
proc copy_file {from to} {
  file copy -force $from $to
}

# Execute some SQL
#
proc catchsql {sql} {
  set rc [catch {uplevel [list db eval $sql]} msg]
  list $rc $msg
}

# Perform a test
#
proc do_test {name cmd expected} {
  puts -nonewline "$name ..."
  set res [uplevel $cmd]
  if {$res eq $expected} {
    puts Ok
  } else {
    puts Error
    puts "  Got: $res"
    puts "  Expected: $expected"
  }
}

# Calc checksum nonce from journal page data.
#
proc calc_nonce {jrnl_pgno} {
  global sectsz
  global db_pgsz
  global jrnl_name
  set jrnl_pg_offset [expr $sectsz+((4+$db_pgsz+4)*$jrnl_pgno)]
  set nonce [hexio_get_int [hexio_read $jrnl_name [expr $jrnl_pg_offset+4+$db_pgsz] 4]]
  for {set i [expr $db_pgsz-200]} {$i>0} {set i [expr $i-200]} {
    set byte [hexio_get_int [hexio_read $jrnl_name [expr $jrnl_pg_offset+4+$i] 1]]
    set nonce [expr $nonce-$byte]
  }
  return $nonce
}

# Calc checksum from journal page data.
#
proc calc_chksum {jrnl_pgno} {
  global sectsz
  global db_pgsz
  global jrnl_name
  global nonce
  set jrnl_pg_offset [expr $sectsz+((4+$db_pgsz+4)*$jrnl_pgno)]
  set chksum $nonce
  for {set i [expr $db_pgsz-200]} {$i>0} {set i [expr $i-200]} {
    set byte [hexio_get_int [hexio_read $jrnl_name [expr $jrnl_pg_offset+4+$i] 1]]
    set chksum [expr $chksum+$byte]
  }
  return $chksum
}

# Print journal page data in hex dump form
#
proc dump_jrnl_page {jrnl_pgno} {
  global sectsz
  global db_pgsz
  global jrnl_name

  # print a header block for the page
  puts [string repeat "-" 79]
  set jrnl_pg_offset [expr $sectsz+((4+$db_pgsz+4)*$jrnl_pgno)]
  set db_pgno [hexio_get_int [hexio_read $jrnl_name [expr $jrnl_pg_offset] 4]]
  set chksum [hexio_get_int [hexio_read $jrnl_name [expr $jrnl_pg_offset+4+$db_pgsz] 4]]
  set nonce [calc_nonce $jrnl_pgno]
  puts [ format {jrnl_pg_offset: %08x (%d)  jrnl_pgno: %d  db_pgno: %d} \
      $jrnl_pg_offset $jrnl_pg_offset \
      $jrnl_pgno $db_pgno]
  puts [ format {nonce: %08x chksum: %08x} \
      $nonce $chksum]

  # now hex dump the data
  # This is derived from the Tcler's WIKI
  set fid [open $jrnl_name r]
  fconfigure $fid -translation binary -encoding binary
  seek $fid [expr $jrnl_pg_offset+4]
  set data [read $fid $db_pgsz]
  close $fid
  for {set addr 0} {$addr<$db_pgsz} {set addr [expr $addr+16]} {
    # get 16 bytes of data
    set s [string range $data $addr [expr $addr+16]]
    
    # Convert the data to hex and to characters.
    binary scan $s H*@0a* hex ascii

    # Replace non-printing characters in the data.
    regsub -all -- {[^[:graph:] ]} $ascii {.} ascii

    # Split the 16 bytes into two 8-byte chunks
    regexp -- {(.{16})(.{0,16})} $hex -> hex1 hex2

    # Convert the hex to pairs of hex digits
    regsub -all -- {..} $hex1 {& } hex1
    regsub -all -- {..} $hex2 {& } hex2

    # Print the hex and ascii data
    puts [ format {%08x %-24s %-24s %-16s} \
        $addr $hex1 $hex2 $ascii ]
  }
}

# Setup for the tests.  Make a backup copy of the files.
#
if [file exist $db_name.org] {
  puts "ERROR: during back-up: $db_name.org exists already."
  return;
}
if [file exist $jrnl_name.org] {
  puts "ERROR: during back-up: $jrnl_name.org exists already."
  return
}
copy_file $db_name $db_name.org
copy_file $jrnl_name $jrnl_name.org

set db_fsize [file size $db_name]
set db_pgsz [hexio_get_int [hexio_read $db_name 16 2]]
set db_npage [expr {$db_fsize / $db_pgsz}]

set jrnl_fsize [file size $jrnl_name]
set jrnl_npage [expr {($jrnl_fsize - $sectsz) / (4 + $db_pgsz + 4)}]

# calculate checksum nonce for first page
set nonce [calc_nonce 0]

# verify all the pages in the journal use the same nonce
for {set i 1} {$i<$jrnl_npage} {incr i} {
  set tnonce [calc_nonce $i]
  if {$tnonce != $nonce} {
    puts "WARNING: different nonces: 0=$nonce $i=$tnonce"
    if {$fix_chksums } {
      set jrnl_pg_offset [expr $sectsz+((4+$db_pgsz+4)*$i)]
      set tchksum [calc_chksum $i]
      hexio_write $jrnl_name [expr $jrnl_pg_offset+4+$db_pgsz] [format %08x $tchksum]
      puts "INFO: fixing chksum: $i=$tchksum"
    }
  }
}

# verify all the page numbers in the journal
for {set i 0} {$i<$jrnl_npage} {incr i} {
  set jrnl_pg_offset [expr $sectsz+((4+$db_pgsz+4)*$i)]
  set db_pgno [hexio_get_int [hexio_read $jrnl_name $jrnl_pg_offset 4]]
  if {$db_pgno < 1} {
    puts "WARNING: page number < 1: $i=$db_pgno"
  }
  if {$db_pgno >= $db_npage} {
    puts "WARNING: page number >= $db_npage: $i=$db_pgno"
  }
}

# dump page data
if {$dump_pages} {
  for {set i 0} {$i<$jrnl_npage} {incr i} {
    dump_jrnl_page $i
  }
}

# write the 8 byte magic string
hexio_write $jrnl_name 0 d9d505f920a163d7

# write -1 for number of records
hexio_write $jrnl_name 8 ffffffff

# write 00 for checksum nonce
hexio_write $jrnl_name 12 [format %08x $nonce]

# write page count
hexio_write $jrnl_name 16 [format %08x $db_npage]

# write sector size
hexio_write $jrnl_name 20 [format %08x $sectsz]

# write page size
hexio_write $jrnl_name 24 [format %08x $db_pgsz]

# check the integrity of the database with the patched journal
sqlite3 db $db_name
do_test restore_jrnl-1.0 {
  catchsql {PRAGMA integrity_check}
} {0 ok}
db close

Changes to tool/shell2.test.
95
96
97
98
99
100
101























































































































    END;

    UPDATE OR REPLACE t5 SET a = 4 WHERE a = 1;
  }
} {1 {Error: near line 9: too many levels of trigger recursion}}
































































































































>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
    END;

    UPDATE OR REPLACE t5 SET a = 4 WHERE a = 1;
  }
} {1 {Error: near line 9: too many levels of trigger recursion}}



# Shell not echoing all commands with echo on.
# Ticket [eb620916be].

# Test with echo off
# NB. whitespace is important
do_test shell2-1.4.1 {
  file delete -force foo.db
  catchcmd "foo.db" {CREATE TABLE foo(a);
INSERT INTO foo(a) VALUES(1);
SELECT * FROM foo;}
} {0 1}

# Test with echo on using command line option
# NB. whitespace is important
do_test shell2-1.4.2 {
  file delete -force foo.db
  catchcmd "-echo foo.db" {CREATE TABLE foo(a);
INSERT INTO foo(a) VALUES(1);
SELECT * FROM foo;}
} {0 {CREATE TABLE foo(a);
INSERT INTO foo(a) VALUES(1);
SELECT * FROM foo;
1}}

# Test with echo on using dot command
# NB. whitespace is important
do_test shell2-1.4.3 {
  file delete -force foo.db
  catchcmd "foo.db" {.echo ON
CREATE TABLE foo(a);
INSERT INTO foo(a) VALUES(1);
SELECT * FROM foo;}
} {0 {CREATE TABLE foo(a);
INSERT INTO foo(a) VALUES(1);
SELECT * FROM foo;
1}}

# Test with echo on using dot command and 
# turning off mid- processing.
# NB. whitespace is important
do_test shell2-1.4.4 {
  file delete -force foo.db
  catchcmd "foo.db" {.echo ON
CREATE TABLE foo(a);
.echo OFF
INSERT INTO foo(a) VALUES(1);
SELECT * FROM foo;}
} {0 {CREATE TABLE foo(a);
.echo OFF
1}}

# Test with echo on using dot command and 
# multiple commands per line.
# NB. whitespace is important
do_test shell2-1.4.5 {
  file delete -force foo.db
  catchcmd "foo.db" {.echo ON
CREATE TABLE foo1(a);
INSERT INTO foo1(a) VALUES(1);
CREATE TABLE foo2(b);
INSERT INTO foo2(b) VALUES(1);
SELECT * FROM foo1; SELECT * FROM foo2;
INSERT INTO foo1(a) VALUES(2); INSERT INTO foo2(b) VALUES(2);
SELECT * FROM foo1; SELECT * FROM foo2;
}
} {0 {CREATE TABLE foo1(a);
INSERT INTO foo1(a) VALUES(1);
CREATE TABLE foo2(b);
INSERT INTO foo2(b) VALUES(1);
SELECT * FROM foo1;
1
SELECT * FROM foo2;
1
INSERT INTO foo1(a) VALUES(2);
INSERT INTO foo2(b) VALUES(2);
SELECT * FROM foo1;
1
2
SELECT * FROM foo2;
1
2}}

# Test with echo on and headers on using dot command and 
# multiple commands per line.
# NB. whitespace is important
do_test shell2-1.4.6 {
  file delete -force foo.db
  catchcmd "foo.db" {.echo ON
.headers ON
CREATE TABLE foo1(a);
INSERT INTO foo1(a) VALUES(1);
CREATE TABLE foo2(b);
INSERT INTO foo2(b) VALUES(1);
SELECT * FROM foo1; SELECT * FROM foo2;
INSERT INTO foo1(a) VALUES(2); INSERT INTO foo2(b) VALUES(2);
SELECT * FROM foo1; SELECT * FROM foo2;
}
} {0 {.headers ON
CREATE TABLE foo1(a);
INSERT INTO foo1(a) VALUES(1);
CREATE TABLE foo2(b);
INSERT INTO foo2(b) VALUES(1);
SELECT * FROM foo1;
a
1
SELECT * FROM foo2;
b
1
INSERT INTO foo1(a) VALUES(2);
INSERT INTO foo2(b) VALUES(2);
SELECT * FROM foo1;
a
1
2
SELECT * FROM foo2;
b
1
2}}