SQLite

Check-in [6dc6472175]
Login

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

Overview
Comment:Merge branches "fix-onerow-opt", "faster-openread", "fts5-delay-tokenizer" and "enhanced-raise", each containing minor enhancements prepared for 3.46, into this branch.
Downloads: Tarball | ZIP archive
Timelines: family | ancestors | descendants | both | pending-3.46
Files: files | file ages | folders
SHA3-256: 6dc6472175bccbed15ebf6811c209d1a0b5fad60158fb32040210f2cdae916a6
User & Date: dan 2024-05-17 14:26:32.984
Original Comment: Merge branches "fix-onerow-opt", "faster-openread", "fts5-delay-tokenizer" and "enhanced-raise", each containing minor enhancements prepared for 3.47, into this branch.
Context
2024-05-23
15:05
Merge various enhancements that were being held on a branch until after the 3.46.0 release. (1) Allow arbitrary expressions in the second argument to the RAISE() function. (2) Performance optimization to the OP_OpenRead and OP_OpenWrite bytecode operators. (3) Allow FTS5 tables to be dropped even if the associated tokenizer is not available. (4) Performance optimizations in FTS5. (5) Generate better bytecode to improve performance of ONEROW scans. (check-in: 1e5385ffc3 user: drh tags: trunk)
2024-05-17
14:26
Merge branches "fix-onerow-opt", "faster-openread", "fts5-delay-tokenizer" and "enhanced-raise", each containing minor enhancements prepared for 3.46, into this branch. (Closed-Leaf check-in: 6dc6472175 user: dan tags: pending-3.46)
13:56
Fix a problem in where.c preventing SQLite from identifying some ONEROW scans, leading to (slightly) sub-optimal byte-code. (Closed-Leaf check-in: b41e71208a user: dan tags: fix-onerow-opt)
2024-05-16
16:22
Add extra tests for the error messages generated by sqlite3_declare_vtab() when passed something other than a well-formed CREATE TABLE statement. (check-in: 5fa0c2a026 user: dan tags: trunk)
2024-05-14
19:41
Optimize queries against fts5vocab(row) tables that do use the "cnt" column. (Closed-Leaf check-in: 96a591c202 user: dan tags: fts5-delay-tokenizer)
2024-05-12
00:26
Optimization in the implementation of OP_OpenRead and OP_OpenWrite (Closed-Leaf check-in: 3a256a1667 user: drh tags: faster-openread)
2024-05-08
17:42
Allow arbitrary expressions as the second argument to RAISE(). (Closed-Leaf check-in: 003e1c8c27 user: drh tags: enhanced-raise)
Changes
Unified Diff Ignore Whitespace Patch
Changes to ext/fts5/fts5Int.h.
138
139
140
141
142
143
144









145
146
147
148
149
150
151

/**************************************************************************
** Interface to code in fts5_config.c. fts5_config.c contains contains code
** to parse the arguments passed to the CREATE VIRTUAL TABLE statement.
*/

typedef struct Fts5Config Fts5Config;










/*
** An instance of the following structure encodes all information that can
** be gleaned from the CREATE VIRTUAL TABLE statement.
**
** And all information loaded from the %_config table.
**







>
>
>
>
>
>
>
>
>







138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160

/**************************************************************************
** Interface to code in fts5_config.c. fts5_config.c contains contains code
** to parse the arguments passed to the CREATE VIRTUAL TABLE statement.
*/

typedef struct Fts5Config Fts5Config;
typedef struct Fts5TokenizerConfig Fts5TokenizerConfig;

struct Fts5TokenizerConfig {
  Fts5Tokenizer *pTok;
  fts5_tokenizer *pTokApi;
  const char **azArg;
  int nArg;
  int ePattern;                   /* FTS_PATTERN_XXX constant */
};

/*
** An instance of the following structure encodes all information that can
** be gleaned from the CREATE VIRTUAL TABLE statement.
**
** And all information loaded from the %_config table.
**
180
181
182
183
184
185
186

187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
**   are ignored. This value is configured using:
**
**       INSERT INTO tbl(tbl, rank) VALUES('prefix-index', $bPrefixIndex);
**
*/
struct Fts5Config {
  sqlite3 *db;                    /* Database handle */

  char *zDb;                      /* Database holding FTS index (e.g. "main") */
  char *zName;                    /* Name of FTS index */
  int nCol;                       /* Number of columns */
  char **azCol;                   /* Column names */
  u8 *abUnindexed;                /* True for unindexed columns */
  int nPrefix;                    /* Number of prefix indexes */
  int *aPrefix;                   /* Sizes in bytes of nPrefix prefix indexes */
  int eContent;                   /* An FTS5_CONTENT value */
  int bContentlessDelete;         /* "contentless_delete=" option (dflt==0) */
  char *zContent;                 /* content table */ 
  char *zContentRowid;            /* "content_rowid=" option value */ 
  int bColumnsize;                /* "columnsize=" option value (dflt==1) */
  int bTokendata;                 /* "tokendata=" option value (dflt==0) */
  int eDetail;                    /* FTS5_DETAIL_XXX value */
  char *zContentExprlist;
  Fts5Tokenizer *pTok;
  fts5_tokenizer *pTokApi;
  int bLock;                      /* True when table is preparing statement */
  int ePattern;                   /* FTS_PATTERN_XXX constant */

  /* Values loaded from the %_config table */
  int iVersion;                   /* fts5 file format 'version' */
  int iCookie;                    /* Incremented when %_config is modified */
  int pgsz;                       /* Approximate page size used in %_data */
  int nAutomerge;                 /* 'automerge' setting */
  int nCrisisMerge;               /* Maximum allowed segments per level */







>















|
<

<







189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212

213

214
215
216
217
218
219
220
**   are ignored. This value is configured using:
**
**       INSERT INTO tbl(tbl, rank) VALUES('prefix-index', $bPrefixIndex);
**
*/
struct Fts5Config {
  sqlite3 *db;                    /* Database handle */
  Fts5Global *pGlobal;            /* Global fts5 object for handle db */
  char *zDb;                      /* Database holding FTS index (e.g. "main") */
  char *zName;                    /* Name of FTS index */
  int nCol;                       /* Number of columns */
  char **azCol;                   /* Column names */
  u8 *abUnindexed;                /* True for unindexed columns */
  int nPrefix;                    /* Number of prefix indexes */
  int *aPrefix;                   /* Sizes in bytes of nPrefix prefix indexes */
  int eContent;                   /* An FTS5_CONTENT value */
  int bContentlessDelete;         /* "contentless_delete=" option (dflt==0) */
  char *zContent;                 /* content table */ 
  char *zContentRowid;            /* "content_rowid=" option value */ 
  int bColumnsize;                /* "columnsize=" option value (dflt==1) */
  int bTokendata;                 /* "tokendata=" option value (dflt==0) */
  int eDetail;                    /* FTS5_DETAIL_XXX value */
  char *zContentExprlist;
  Fts5TokenizerConfig t;

  int bLock;                      /* True when table is preparing statement */


  /* Values loaded from the %_config table */
  int iVersion;                   /* fts5 file format 'version' */
  int iCookie;                    /* Incremented when %_config is modified */
  int pgsz;                       /* Approximate page size used in %_data */
  int nAutomerge;                 /* 'automerge' setting */
  int nCrisisMerge;               /* Maximum allowed segments per level */
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
typedef struct Fts5Table Fts5Table;
struct Fts5Table {
  sqlite3_vtab base;              /* Base class used by SQLite core */
  Fts5Config *pConfig;            /* Virtual table configuration */
  Fts5Index *pIndex;              /* Full-text index */
};

int sqlite3Fts5GetTokenizer(
  Fts5Global*, 
  const char **azArg,
  int nArg,
  Fts5Config*,
  char **pzErr
);

Fts5Table *sqlite3Fts5TableFromCsrid(Fts5Global*, i64);

int sqlite3Fts5FlushToDisk(Fts5Table*);

/*
** End of interface to code in fts5.c.







|
<
<
<
<
<
<







601
602
603
604
605
606
607
608






609
610
611
612
613
614
615
typedef struct Fts5Table Fts5Table;
struct Fts5Table {
  sqlite3_vtab base;              /* Base class used by SQLite core */
  Fts5Config *pConfig;            /* Virtual table configuration */
  Fts5Index *pIndex;              /* Full-text index */
};

int sqlite3Fts5LoadTokenizer(Fts5Config *pConfig);







Fts5Table *sqlite3Fts5TableFromCsrid(Fts5Global*, i64);

int sqlite3Fts5FlushToDisk(Fts5Table*);

/*
** End of interface to code in fts5.c.
862
863
864
865
866
867
868

869
870
871
872
873
874
875
*/

int sqlite3Fts5TokenizerInit(fts5_api*);
int sqlite3Fts5TokenizerPattern(
    int (*xCreate)(void*, const char**, int, Fts5Tokenizer**),
    Fts5Tokenizer *pTok
);

/*
** End of interface to code in fts5_tokenizer.c.
**************************************************************************/

/**************************************************************************
** Interface to code in fts5_vocab.c. 
*/







>







864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
*/

int sqlite3Fts5TokenizerInit(fts5_api*);
int sqlite3Fts5TokenizerPattern(
    int (*xCreate)(void*, const char**, int, Fts5Tokenizer**),
    Fts5Tokenizer *pTok
);
int sqlite3Fts5TokenizerPreload(Fts5TokenizerConfig*);
/*
** End of interface to code in fts5_tokenizer.c.
**************************************************************************/

/**************************************************************************
** Interface to code in fts5_vocab.c. 
*/
Changes to ext/fts5/fts5_config.c.
294
295
296
297
298
299
300
301
302
303
304
305

306
307
308
309
310
311
312
313
    assert( pConfig->nPrefix<=FTS5_MAX_PREFIX_INDEXES );
    return rc;
  }

  if( sqlite3_strnicmp("tokenize", zCmd, nCmd)==0 ){
    const char *p = (const char*)zArg;
    sqlite3_int64 nArg = strlen(zArg) + 1;
    char **azArg = sqlite3Fts5MallocZero(&rc, sizeof(char*) * nArg);
    char *pDel = sqlite3Fts5MallocZero(&rc, nArg * 2);
    char *pSpace = pDel;

    if( azArg && pSpace ){

      if( pConfig->pTok ){
        *pzErr = sqlite3_mprintf("multiple tokenize=... directives");
        rc = SQLITE_ERROR;
      }else{
        for(nArg=0; p && *p; nArg++){
          const char *p2 = fts5ConfigSkipWhitespace(p);
          if( *p2=='\'' ){
            p = fts5ConfigSkipLiteral(p2);







|
<
<

|
>
|







294
295
296
297
298
299
300
301


302
303
304
305
306
307
308
309
310
311
312
    assert( pConfig->nPrefix<=FTS5_MAX_PREFIX_INDEXES );
    return rc;
  }

  if( sqlite3_strnicmp("tokenize", zCmd, nCmd)==0 ){
    const char *p = (const char*)zArg;
    sqlite3_int64 nArg = strlen(zArg) + 1;
    char **azArg = sqlite3Fts5MallocZero(&rc, (sizeof(char*) + 2) * nArg);



    if( azArg ){
      char *pSpace = (char*)&azArg[nArg];
      if( pConfig->t.azArg ){
        *pzErr = sqlite3_mprintf("multiple tokenize=... directives");
        rc = SQLITE_ERROR;
      }else{
        for(nArg=0; p && *p; nArg++){
          const char *p2 = fts5ConfigSkipWhitespace(p);
          if( *p2=='\'' ){
            p = fts5ConfigSkipLiteral(p2);
322
323
324
325
326
327
328
329
330

331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
            p = fts5ConfigSkipWhitespace(p);
          }
        }
        if( p==0 ){
          *pzErr = sqlite3_mprintf("parse error in tokenize directive");
          rc = SQLITE_ERROR;
        }else{
          rc = sqlite3Fts5GetTokenizer(pGlobal, 
              (const char**)azArg, (int)nArg, pConfig,

              pzErr
          );
        }
      }
    }

    sqlite3_free(azArg);
    sqlite3_free(pDel);
    return rc;
  }

  if( sqlite3_strnicmp("content", zCmd, nCmd)==0 ){
    if( pConfig->eContent!=FTS5_CONTENT_NORMAL ){
      *pzErr = sqlite3_mprintf("multiple content=... directives");
      rc = SQLITE_ERROR;







<
|
>
|
<



<

|







321
322
323
324
325
326
327

328
329
330

331
332
333

334
335
336
337
338
339
340
341
342
            p = fts5ConfigSkipWhitespace(p);
          }
        }
        if( p==0 ){
          *pzErr = sqlite3_mprintf("parse error in tokenize directive");
          rc = SQLITE_ERROR;
        }else{

          pConfig->t.azArg = (const char**)azArg;
          pConfig->t.nArg = nArg;
          azArg = 0;

        }
      }
    }

    sqlite3_free(azArg);

    return rc;
  }

  if( sqlite3_strnicmp("content", zCmd, nCmd)==0 ){
    if( pConfig->eContent!=FTS5_CONTENT_NORMAL ){
      *pzErr = sqlite3_mprintf("multiple content=... directives");
      rc = SQLITE_ERROR;
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
    return rc;
  }

  *pzErr = sqlite3_mprintf("unrecognized option: \"%.*s\"", nCmd, zCmd);
  return SQLITE_ERROR;
}

/*
** Allocate an instance of the default tokenizer ("simple") at 
** Fts5Config.pTokenizer. Return SQLITE_OK if successful, or an SQLite error
** code if an error occurs.
*/
static int fts5ConfigDefaultTokenizer(Fts5Global *pGlobal, Fts5Config *pConfig){
  assert( pConfig->pTok==0 && pConfig->pTokApi==0 );
  return sqlite3Fts5GetTokenizer(pGlobal, 0, 0, pConfig, 0);
}

/*
** Gobble up the first bareword or quoted word from the input buffer zIn.
** Return a pointer to the character immediately following the last in
** the gobbled word if successful, or a NULL pointer otherwise (failed
** to find close-quote character).
**
** Before returning, set pzOut to point to a new buffer containing a







<
<
<
<
<
<
<
<
<
<







405
406
407
408
409
410
411










412
413
414
415
416
417
418
    return rc;
  }

  *pzErr = sqlite3_mprintf("unrecognized option: \"%.*s\"", nCmd, zCmd);
  return SQLITE_ERROR;
}











/*
** Gobble up the first bareword or quoted word from the input buffer zIn.
** Return a pointer to the character immediately following the last in
** the gobbled word if successful, or a NULL pointer otherwise (failed
** to find close-quote character).
**
** Before returning, set pzOut to point to a new buffer containing a
550
551
552
553
554
555
556

557
558
559
560
561
562
563
  Fts5Config *pRet;               /* New object to return */
  int i;
  sqlite3_int64 nByte;

  *ppOut = pRet = (Fts5Config*)sqlite3_malloc(sizeof(Fts5Config));
  if( pRet==0 ) return SQLITE_NOMEM;
  memset(pRet, 0, sizeof(Fts5Config));

  pRet->db = db;
  pRet->iCookie = -1;

  nByte = nArg * (sizeof(char*) + sizeof(u8));
  pRet->azCol = (char**)sqlite3Fts5MallocZero(&rc, nByte);
  pRet->abUnindexed = pRet->azCol ? (u8*)&pRet->azCol[nArg] : 0;
  pRet->zDb = sqlite3Fts5Strndup(&rc, azArg[1], -1);







>







537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
  Fts5Config *pRet;               /* New object to return */
  int i;
  sqlite3_int64 nByte;

  *ppOut = pRet = (Fts5Config*)sqlite3_malloc(sizeof(Fts5Config));
  if( pRet==0 ) return SQLITE_NOMEM;
  memset(pRet, 0, sizeof(Fts5Config));
  pRet->pGlobal = pGlobal;
  pRet->db = db;
  pRet->iCookie = -1;

  nByte = nArg * (sizeof(char*) + sizeof(u8));
  pRet->azCol = (char**)sqlite3Fts5MallocZero(&rc, nByte);
  pRet->abUnindexed = pRet->azCol ? (u8*)&pRet->azCol[nArg] : 0;
  pRet->zDb = sqlite3Fts5Strndup(&rc, azArg[1], -1);
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
  if( rc==SQLITE_OK && pRet->bContentlessDelete && pRet->bColumnsize==0 ){
    *pzErr = sqlite3_mprintf(
        "contentless_delete=1 is incompatible with columnsize=0"
    );
    rc = SQLITE_ERROR;
  }

  /* If a tokenizer= option was successfully parsed, the tokenizer has
  ** already been allocated. Otherwise, allocate an instance of the default
  ** tokenizer (unicode61) now.  */
  if( rc==SQLITE_OK && pRet->pTok==0 ){
    rc = fts5ConfigDefaultTokenizer(pGlobal, pRet);
  }

  /* If no zContent option was specified, fill in the default values. */
  if( rc==SQLITE_OK && pRet->zContent==0 ){
    const char *zTail = 0;
    assert( pRet->eContent==FTS5_CONTENT_NORMAL 
         || pRet->eContent==FTS5_CONTENT_NONE 
    );
    if( pRet->eContent==FTS5_CONTENT_NORMAL ){







<
<
<
<
<
<
<







624
625
626
627
628
629
630







631
632
633
634
635
636
637
  if( rc==SQLITE_OK && pRet->bContentlessDelete && pRet->bColumnsize==0 ){
    *pzErr = sqlite3_mprintf(
        "contentless_delete=1 is incompatible with columnsize=0"
    );
    rc = SQLITE_ERROR;
  }








  /* If no zContent option was specified, fill in the default values. */
  if( rc==SQLITE_OK && pRet->zContent==0 ){
    const char *zTail = 0;
    assert( pRet->eContent==FTS5_CONTENT_NORMAL 
         || pRet->eContent==FTS5_CONTENT_NONE 
    );
    if( pRet->eContent==FTS5_CONTENT_NORMAL ){
684
685
686
687
688
689
690
691
692
693

694
695
696
697
698
699
700

/*
** Free the configuration object passed as the only argument.
*/
void sqlite3Fts5ConfigFree(Fts5Config *pConfig){
  if( pConfig ){
    int i;
    if( pConfig->pTok ){
      pConfig->pTokApi->xDelete(pConfig->pTok);
    }

    sqlite3_free(pConfig->zDb);
    sqlite3_free(pConfig->zName);
    for(i=0; i<pConfig->nCol; i++){
      sqlite3_free(pConfig->azCol[i]);
    }
    sqlite3_free(pConfig->azCol);
    sqlite3_free(pConfig->aPrefix);







|
|

>







665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682

/*
** Free the configuration object passed as the only argument.
*/
void sqlite3Fts5ConfigFree(Fts5Config *pConfig){
  if( pConfig ){
    int i;
    if( pConfig->t.pTok ){
      pConfig->t.pTokApi->xDelete(pConfig->t.pTok);
    }
    sqlite3_free(pConfig->t.azArg);
    sqlite3_free(pConfig->zDb);
    sqlite3_free(pConfig->zName);
    for(i=0; i<pConfig->nCol; i++){
      sqlite3_free(pConfig->azCol[i]);
    }
    sqlite3_free(pConfig->azCol);
    sqlite3_free(pConfig->aPrefix);
761
762
763
764
765
766
767

768




769
770
771



772
773
774
775
776
777
778
int sqlite3Fts5Tokenize(
  Fts5Config *pConfig,            /* FTS5 Configuration object */
  int flags,                      /* FTS5_TOKENIZE_* flags */
  const char *pText, int nText,   /* Text to tokenize */
  void *pCtx,                     /* Context passed to xToken() */
  int (*xToken)(void*, int, const char*, int, int, int)    /* Callback */
){

  if( pText==0 ) return SQLITE_OK;




  return pConfig->pTokApi->xTokenize(
      pConfig->pTok, pCtx, flags, pText, nText, xToken
  );



}

/*
** Argument pIn points to the first character in what is expected to be
** a comma-separated list of SQL literals followed by a ')' character.
** If it actually is this, return a pointer to the ')'. Otherwise, return
** NULL to indicate a parse error.







>
|
>
>
>
>
|
|
|
>
>
>







743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
int sqlite3Fts5Tokenize(
  Fts5Config *pConfig,            /* FTS5 Configuration object */
  int flags,                      /* FTS5_TOKENIZE_* flags */
  const char *pText, int nText,   /* Text to tokenize */
  void *pCtx,                     /* Context passed to xToken() */
  int (*xToken)(void*, int, const char*, int, int, int)    /* Callback */
){
  int rc = SQLITE_OK;
  if( pText ){
    if( pConfig->t.pTok==0 ){
      rc = sqlite3Fts5LoadTokenizer(pConfig);
    }
    if( rc==SQLITE_OK ){
      rc = pConfig->t.pTokApi->xTokenize(
          pConfig->t.pTok, pCtx, flags, pText, nText, xToken
      );
    }
  }
  return rc;
}

/*
** Argument pIn points to the first character in what is expected to be
** a comma-separated list of SQL literals followed by a ')' character.
** If it actually is this, return a pointer to the ')'. Otherwise, return
** NULL to indicate a parse error.
Changes to ext/fts5/fts5_expr.c.
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
      sParse.apPhrase = 0;
    }
  }else{
    sqlite3Fts5ParseNodeFree(sParse.pExpr);
  }

  sqlite3_free(sParse.apPhrase);
  *pzErr = sParse.zErr;
  return sParse.rc;
}

/*
** Assuming that buffer z is at least nByte bytes in size and contains a
** valid utf-8 string, return the number of characters in the string.
*/







|







320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
      sParse.apPhrase = 0;
    }
  }else{
    sqlite3Fts5ParseNodeFree(sParse.pExpr);
  }

  sqlite3_free(sParse.apPhrase);
  if( 0==*pzErr ) *pzErr = sParse.zErr;
  return sParse.rc;
}

/*
** Assuming that buffer z is at least nByte bytes in size and contains a
** valid utf-8 string, return the number of characters in the string.
*/
Changes to ext/fts5/fts5_main.c.
373
374
375
376
377
378
379

380
381



382
383
384
385
386
387
388
  /* Allocate the new vtab object and parse the configuration */
  pTab = (Fts5FullTable*)sqlite3Fts5MallocZero(&rc, sizeof(Fts5FullTable));
  if( rc==SQLITE_OK ){
    rc = sqlite3Fts5ConfigParse(pGlobal, db, argc, azConfig, &pConfig, pzErr);
    assert( (rc==SQLITE_OK && *pzErr==0) || pConfig==0 );
  }
  if( rc==SQLITE_OK ){

    pTab->p.pConfig = pConfig;
    pTab->pGlobal = pGlobal;



  }

  /* Open the index sub-system */
  if( rc==SQLITE_OK ){
    rc = sqlite3Fts5IndexOpen(pConfig, bCreate, &pTab->p.pIndex, pzErr);
  }








>


>
>
>







373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
  /* Allocate the new vtab object and parse the configuration */
  pTab = (Fts5FullTable*)sqlite3Fts5MallocZero(&rc, sizeof(Fts5FullTable));
  if( rc==SQLITE_OK ){
    rc = sqlite3Fts5ConfigParse(pGlobal, db, argc, azConfig, &pConfig, pzErr);
    assert( (rc==SQLITE_OK && *pzErr==0) || pConfig==0 );
  }
  if( rc==SQLITE_OK ){
    pConfig->pzErrmsg = pzErr;
    pTab->p.pConfig = pConfig;
    pTab->pGlobal = pGlobal;
    if( bCreate || sqlite3Fts5TokenizerPreload(&pConfig->t) ){
      rc = sqlite3Fts5LoadTokenizer(pConfig);
    }
  }

  /* Open the index sub-system */
  if( rc==SQLITE_OK ){
    rc = sqlite3Fts5IndexOpen(pConfig, bCreate, &pTab->p.pIndex, pzErr);
  }

396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416

417
418
419
420
421
422
423
  /* Call sqlite3_declare_vtab() */
  if( rc==SQLITE_OK ){
    rc = sqlite3Fts5ConfigDeclareVtab(pConfig);
  }

  /* Load the initial configuration */
  if( rc==SQLITE_OK ){
    assert( pConfig->pzErrmsg==0 );
    pConfig->pzErrmsg = pzErr;
    rc = sqlite3Fts5IndexLoadConfig(pTab->p.pIndex);
    sqlite3Fts5IndexRollback(pTab->p.pIndex);
    pConfig->pzErrmsg = 0;
  }

  if( rc==SQLITE_OK && pConfig->eContent==FTS5_CONTENT_NORMAL ){
    rc = sqlite3_vtab_config(db, SQLITE_VTAB_CONSTRAINT_SUPPORT, (int)1);
  }
  if( rc==SQLITE_OK ){
    rc = sqlite3_vtab_config(db, SQLITE_VTAB_INNOCUOUS);
  }


  if( rc!=SQLITE_OK ){
    fts5FreeVtab(pTab);
    pTab = 0;
  }else if( bCreate ){
    fts5CheckTransactionState(pTab, FTS5_BEGIN, 0);
  }
  *ppVTab = (sqlite3_vtab*)pTab;







<
<


<









>







400
401
402
403
404
405
406


407
408

409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
  /* Call sqlite3_declare_vtab() */
  if( rc==SQLITE_OK ){
    rc = sqlite3Fts5ConfigDeclareVtab(pConfig);
  }

  /* Load the initial configuration */
  if( rc==SQLITE_OK ){


    rc = sqlite3Fts5IndexLoadConfig(pTab->p.pIndex);
    sqlite3Fts5IndexRollback(pTab->p.pIndex);

  }

  if( rc==SQLITE_OK && pConfig->eContent==FTS5_CONTENT_NORMAL ){
    rc = sqlite3_vtab_config(db, SQLITE_VTAB_CONSTRAINT_SUPPORT, (int)1);
  }
  if( rc==SQLITE_OK ){
    rc = sqlite3_vtab_config(db, SQLITE_VTAB_INNOCUOUS);
  }

  if( pConfig ) pConfig->pzErrmsg = 0;
  if( rc!=SQLITE_OK ){
    fts5FreeVtab(pTab);
    pTab = 0;
  }else if( bCreate ){
    fts5CheckTransactionState(pTab, FTS5_BEGIN, 0);
  }
  *ppVTab = (sqlite3_vtab*)pTab;
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494

static int fts5UsePatternMatch(
  Fts5Config *pConfig, 
  struct sqlite3_index_constraint *p
){
  assert( FTS5_PATTERN_GLOB==SQLITE_INDEX_CONSTRAINT_GLOB );
  assert( FTS5_PATTERN_LIKE==SQLITE_INDEX_CONSTRAINT_LIKE );
  if( pConfig->ePattern==FTS5_PATTERN_GLOB && p->op==FTS5_PATTERN_GLOB ){
    return 1;
  }
  if( pConfig->ePattern==FTS5_PATTERN_LIKE 
   && (p->op==FTS5_PATTERN_LIKE || p->op==FTS5_PATTERN_GLOB)
  ){
    return 1;
  }
  return 0;
}








|


|







479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496

static int fts5UsePatternMatch(
  Fts5Config *pConfig, 
  struct sqlite3_index_constraint *p
){
  assert( FTS5_PATTERN_GLOB==SQLITE_INDEX_CONSTRAINT_GLOB );
  assert( FTS5_PATTERN_LIKE==SQLITE_INDEX_CONSTRAINT_LIKE );
  if( pConfig->t.ePattern==FTS5_PATTERN_GLOB && p->op==FTS5_PATTERN_GLOB ){
    return 1;
  }
  if( pConfig->t.ePattern==FTS5_PATTERN_LIKE 
   && (p->op==FTS5_PATTERN_LIKE || p->op==FTS5_PATTERN_GLOB)
  ){
    return 1;
  }
  return 0;
}

622
623
624
625
626
627
628

629
630
631
632
633
634
635
      if( iCol>=0 && iCol<nCol && fts5UsePatternMatch(pConfig, p) ){
        assert( p->op==FTS5_PATTERN_LIKE || p->op==FTS5_PATTERN_GLOB );
        idxStr[iIdxStr++] = p->op==FTS5_PATTERN_LIKE ? 'L' : 'G';
        sqlite3_snprintf(6, &idxStr[iIdxStr], "%d", iCol);
        idxStr += strlen(&idxStr[iIdxStr]);
        pInfo->aConstraintUsage[i].argvIndex = ++iCons;
        assert( idxStr[iIdxStr]=='\0' );

      }else if( bSeenEq==0 && p->op==SQLITE_INDEX_CONSTRAINT_EQ && iCol<0 ){
        idxStr[iIdxStr++] = '=';
        bSeenEq = 1;
        pInfo->aConstraintUsage[i].argvIndex = ++iCons;
      }
    }
  }







>







624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
      if( iCol>=0 && iCol<nCol && fts5UsePatternMatch(pConfig, p) ){
        assert( p->op==FTS5_PATTERN_LIKE || p->op==FTS5_PATTERN_GLOB );
        idxStr[iIdxStr++] = p->op==FTS5_PATTERN_LIKE ? 'L' : 'G';
        sqlite3_snprintf(6, &idxStr[iIdxStr], "%d", iCol);
        idxStr += strlen(&idxStr[iIdxStr]);
        pInfo->aConstraintUsage[i].argvIndex = ++iCons;
        assert( idxStr[iIdxStr]=='\0' );
        bSeenMatch = 1;
      }else if( bSeenEq==0 && p->op==SQLITE_INDEX_CONSTRAINT_EQ && iCol<0 ){
        idxStr[iIdxStr++] = '=';
        bSeenEq = 1;
        pInfo->aConstraintUsage[i].argvIndex = ++iCons;
      }
    }
  }
2856
2857
2858
2859
2860
2861
2862
2863
2864
2865
2866
2867
2868
2869
2870
2871
2872
2873
2874
2875
2876
2877
2878
2879
2880
2881
2882
2883
2884
2885
2886
2887
2888
2889
2890
2891
2892
2893
2894
2895
2896
2897
2898











2899
2900
2901
2902
2903
2904
2905
    memset(pTokenizer, 0, sizeof(fts5_tokenizer));
    rc = SQLITE_ERROR;
  }

  return rc;
}

int sqlite3Fts5GetTokenizer(
  Fts5Global *pGlobal, 
  const char **azArg,
  int nArg,
  Fts5Config *pConfig,
  char **pzErr
){
  Fts5TokenizerModule *pMod;
  int rc = SQLITE_OK;

  pMod = fts5LocateTokenizer(pGlobal, nArg==0 ? 0 : azArg[0]);
  if( pMod==0 ){
    assert( nArg>0 );
    rc = SQLITE_ERROR;
    *pzErr = sqlite3_mprintf("no such tokenizer: %s", azArg[0]);
  }else{
    rc = pMod->x.xCreate(
        pMod->pUserData, (azArg?&azArg[1]:0), (nArg?nArg-1:0), &pConfig->pTok
    );
    pConfig->pTokApi = &pMod->x;
    if( rc!=SQLITE_OK ){
      if( pzErr ) *pzErr = sqlite3_mprintf("error in tokenizer constructor");
    }else{
      pConfig->ePattern = sqlite3Fts5TokenizerPattern(
          pMod->x.xCreate, pConfig->pTok
      );
    }
  }

  if( rc!=SQLITE_OK ){
    pConfig->pTokApi = 0;
    pConfig->pTok = 0;
  }

  return rc;
}












static void fts5ModuleDestroy(void *pCtx){
  Fts5TokenizerModule *pTok, *pNextTok;
  Fts5Auxiliary *pAux, *pNextAux;
  Fts5Global *pGlobal = (Fts5Global*)pCtx;

  for(pAux=pGlobal->pAux; pAux; pAux=pNextAux){







|
















|

|



|
|





|
|




>
>
>
>
>
>
>
>
>
>
>







2859
2860
2861
2862
2863
2864
2865
2866
2867
2868
2869
2870
2871
2872
2873
2874
2875
2876
2877
2878
2879
2880
2881
2882
2883
2884
2885
2886
2887
2888
2889
2890
2891
2892
2893
2894
2895
2896
2897
2898
2899
2900
2901
2902
2903
2904
2905
2906
2907
2908
2909
2910
2911
2912
2913
2914
2915
2916
2917
2918
2919
    memset(pTokenizer, 0, sizeof(fts5_tokenizer));
    rc = SQLITE_ERROR;
  }

  return rc;
}

int fts5GetTokenizer(
  Fts5Global *pGlobal, 
  const char **azArg,
  int nArg,
  Fts5Config *pConfig,
  char **pzErr
){
  Fts5TokenizerModule *pMod;
  int rc = SQLITE_OK;

  pMod = fts5LocateTokenizer(pGlobal, nArg==0 ? 0 : azArg[0]);
  if( pMod==0 ){
    assert( nArg>0 );
    rc = SQLITE_ERROR;
    *pzErr = sqlite3_mprintf("no such tokenizer: %s", azArg[0]);
  }else{
    rc = pMod->x.xCreate(
        pMod->pUserData, (azArg?&azArg[1]:0), (nArg?nArg-1:0), &pConfig->t.pTok
    );
    pConfig->t.pTokApi = &pMod->x;
    if( rc!=SQLITE_OK ){
      if( pzErr ) *pzErr = sqlite3_mprintf("error in tokenizer constructor");
    }else{
      pConfig->t.ePattern = sqlite3Fts5TokenizerPattern(
          pMod->x.xCreate, pConfig->t.pTok
      );
    }
  }

  if( rc!=SQLITE_OK ){
    pConfig->t.pTokApi = 0;
    pConfig->t.pTok = 0;
  }

  return rc;
}

/*
** Attempt to instantiate the tokenizer.
*/
int sqlite3Fts5LoadTokenizer(Fts5Config *pConfig){
  return fts5GetTokenizer(
      pConfig->pGlobal, pConfig->t.azArg, pConfig->t.nArg, 
      pConfig, pConfig->pzErrmsg
  );
}


static void fts5ModuleDestroy(void *pCtx){
  Fts5TokenizerModule *pTok, *pNextTok;
  Fts5Auxiliary *pAux, *pNextAux;
  Fts5Global *pGlobal = (Fts5Global*)pCtx;

  for(pAux=pGlobal->pAux; pAux; pAux=pNextAux){
Changes to ext/fts5/fts5_tokenize.c.
1423
1424
1425
1426
1427
1428
1429










1430
1431
1432
1433
1434
1435
1436
    TrigramTokenizer *p = (TrigramTokenizer*)pTok;
    if( p->iFoldParam==0 ){
      return p->bFold ? FTS5_PATTERN_LIKE : FTS5_PATTERN_GLOB;
    }
  }
  return FTS5_PATTERN_NONE;
}











/*
** Register all built-in tokenizers with FTS5.
*/
int sqlite3Fts5TokenizerInit(fts5_api *pApi){
  struct BuiltinTokenizer {
    const char *zName;







>
>
>
>
>
>
>
>
>
>







1423
1424
1425
1426
1427
1428
1429
1430
1431
1432
1433
1434
1435
1436
1437
1438
1439
1440
1441
1442
1443
1444
1445
1446
    TrigramTokenizer *p = (TrigramTokenizer*)pTok;
    if( p->iFoldParam==0 ){
      return p->bFold ? FTS5_PATTERN_LIKE : FTS5_PATTERN_GLOB;
    }
  }
  return FTS5_PATTERN_NONE;
}

/*
** Return true if the tokenizer described by p->azArg[] is the trigram
** tokenizer. This tokenizer needs to be loaded before xBestIndex is
** called for the first time in order to correctly handle LIKE/GLOB.
*/
int sqlite3Fts5TokenizerPreload(Fts5TokenizerConfig *p){
  return (p->nArg>=1 && 0==sqlite3_stricmp(p->azArg[0], "trigram"));
}


/*
** Register all built-in tokenizers with FTS5.
*/
int sqlite3Fts5TokenizerInit(fts5_api *pApi){
  struct BuiltinTokenizer {
    const char *zName;
Changes to ext/fts5/fts5_vocab.c.
60
61
62
63
64
65
66

67
68
69
70
71
72
73

  int bEof;                       /* True if this cursor is at EOF */
  Fts5IndexIter *pIter;           /* Term/rowid iterator object */
  void *pStruct;                  /* From sqlite3Fts5StructureRef() */

  int nLeTerm;                    /* Size of zLeTerm in bytes */
  char *zLeTerm;                  /* (term <= $zLeTerm) paramater, or NULL */


  /* These are used by 'col' tables only */
  int iCol;
  i64 *aCnt;
  i64 *aDoc;

  /* Output values used by all tables. */







>







60
61
62
63
64
65
66
67
68
69
70
71
72
73
74

  int bEof;                       /* True if this cursor is at EOF */
  Fts5IndexIter *pIter;           /* Term/rowid iterator object */
  void *pStruct;                  /* From sqlite3Fts5StructureRef() */

  int nLeTerm;                    /* Size of zLeTerm in bytes */
  char *zLeTerm;                  /* (term <= $zLeTerm) paramater, or NULL */
  int colUsed;                    /* Copy of sqlite3_index_info.colUsed */

  /* These are used by 'col' tables only */
  int iCol;
  i64 *aCnt;
  i64 *aDoc;

  /* Output values used by all tables. */
86
87
88
89
90
91
92
93
94
95


96
97
98
99
100
101
102
#define FTS5_VOCAB_COL_SCHEMA  "term, col, doc, cnt"
#define FTS5_VOCAB_ROW_SCHEMA  "term, doc, cnt"
#define FTS5_VOCAB_INST_SCHEMA "term, doc, col, offset"

/*
** Bits for the mask used as the idxNum value by xBestIndex/xFilter.
*/
#define FTS5_VOCAB_TERM_EQ 0x01
#define FTS5_VOCAB_TERM_GE 0x02
#define FTS5_VOCAB_TERM_LE 0x04




/*
** Translate a string containing an fts5vocab table type to an 
** FTS5_VOCAB_XXX constant. If successful, set *peType to the output
** value and return SQLITE_OK. Otherwise, set *pzErr to an error message
** and return SQLITE_ERROR.







|
|
|
>
>







87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
#define FTS5_VOCAB_COL_SCHEMA  "term, col, doc, cnt"
#define FTS5_VOCAB_ROW_SCHEMA  "term, doc, cnt"
#define FTS5_VOCAB_INST_SCHEMA "term, doc, col, offset"

/*
** Bits for the mask used as the idxNum value by xBestIndex/xFilter.
*/
#define FTS5_VOCAB_TERM_EQ 0x0100
#define FTS5_VOCAB_TERM_GE 0x0200
#define FTS5_VOCAB_TERM_LE 0x0400

#define FTS5_VOCAB_COLUSED_MASK 0xFF


/*
** Translate a string containing an fts5vocab table type to an 
** FTS5_VOCAB_XXX constant. If successful, set *peType to the output
** value and return SQLITE_OK. Otherwise, set *pzErr to an error message
** and return SQLITE_ERROR.
265
266
267
268
269
270
271
272
273
274
275


276
277
278
279
280
281
282
  sqlite3_vtab *pUnused,
  sqlite3_index_info *pInfo
){
  int i;
  int iTermEq = -1;
  int iTermGe = -1;
  int iTermLe = -1;
  int idxNum = 0;
  int nArg = 0;

  UNUSED_PARAM(pUnused);



  for(i=0; i<pInfo->nConstraint; i++){
    struct sqlite3_index_constraint *p = &pInfo->aConstraint[i];
    if( p->usable==0 ) continue;
    if( p->iColumn==0 ){          /* term column */
      if( p->op==SQLITE_INDEX_CONSTRAINT_EQ ) iTermEq = i;
      if( p->op==SQLITE_INDEX_CONSTRAINT_LE ) iTermLe = i;







|



>
>







268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
  sqlite3_vtab *pUnused,
  sqlite3_index_info *pInfo
){
  int i;
  int iTermEq = -1;
  int iTermGe = -1;
  int iTermLe = -1;
  int idxNum = (int)pInfo->colUsed;
  int nArg = 0;

  UNUSED_PARAM(pUnused);

  assert( (pInfo->colUsed & FTS5_VOCAB_COLUSED_MASK)==pInfo->colUsed );

  for(i=0; i<pInfo->nConstraint; i++){
    struct sqlite3_index_constraint *p = &pInfo->aConstraint[i];
    if( p->usable==0 ) continue;
    if( p->iColumn==0 ){          /* term column */
      if( p->op==SQLITE_INDEX_CONSTRAINT_EQ ) iTermEq = i;
      if( p->op==SQLITE_INDEX_CONSTRAINT_LE ) iTermLe = i;
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
  if( rc==SQLITE_OK ){
    if( pFts5==0 ){
      rc = sqlite3_finalize(pStmt);
      pStmt = 0;
      if( rc==SQLITE_OK ){
        pVTab->zErrMsg = sqlite3_mprintf(
            "no such fts5 table: %s.%s", pTab->zFts5Db, pTab->zFts5Tbl
            );
        rc = SQLITE_ERROR;
      }
    }else{
      rc = sqlite3Fts5FlushToDisk(pFts5);
    }
  }








|







366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
  if( rc==SQLITE_OK ){
    if( pFts5==0 ){
      rc = sqlite3_finalize(pStmt);
      pStmt = 0;
      if( rc==SQLITE_OK ){
        pVTab->zErrMsg = sqlite3_mprintf(
            "no such fts5 table: %s.%s", pTab->zFts5Db, pTab->zFts5Tbl
        );
        rc = SQLITE_ERROR;
      }
    }else{
      rc = sqlite3Fts5FlushToDisk(pFts5);
    }
  }

521
522
523
524
525
526
527


528
529







530

531
532
533
534
535
536
537
        int iOff = 0;               /* Current offset within position list */

        pPos = pCsr->pIter->pData;
        nPos = pCsr->pIter->nData;

        switch( pTab->eType ){
          case FTS5_VOCAB_ROW:


            if( eDetail==FTS5_DETAIL_FULL ){
              while( 0==sqlite3Fts5PoslistNext64(pPos, nPos, &iOff, &iPos) ){







                pCsr->aCnt[0]++;

              }
            }
            pCsr->aDoc[0]++;
            break;

          case FTS5_VOCAB_COL:
            if( eDetail==FTS5_DETAIL_FULL ){







>
>
|
|
>
>
>
>
>
>
>
|
>







526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
        int iOff = 0;               /* Current offset within position list */

        pPos = pCsr->pIter->pData;
        nPos = pCsr->pIter->nData;

        switch( pTab->eType ){
          case FTS5_VOCAB_ROW:
            /* Do not bother counting the number of instances if the "cnt"
            ** column is not being read (according to colUsed).  */
            if( eDetail==FTS5_DETAIL_FULL && (pCsr->colUsed & 0x04) ){
              while( iPos<nPos ){
                u32 ii;
                fts5FastGetVarint32(pPos, iPos, ii);
                if( ii==1 ){ 
                  /* New column in the position list */
                  fts5FastGetVarint32(pPos, iPos, ii);
                }else{
                  /* An instance - increment pCsr->aCnt[] */
                  pCsr->aCnt[0]++;
                }
              }
            }
            pCsr->aDoc[0]++;
            break;

          case FTS5_VOCAB_COL:
            if( eDetail==FTS5_DETAIL_FULL ){
621
622
623
624
625
626
627

628
629
630
631
632
633
634

  UNUSED_PARAM2(zUnused, nUnused);

  fts5VocabResetCursor(pCsr);
  if( idxNum & FTS5_VOCAB_TERM_EQ ) pEq = apVal[iVal++];
  if( idxNum & FTS5_VOCAB_TERM_GE ) pGe = apVal[iVal++];
  if( idxNum & FTS5_VOCAB_TERM_LE ) pLe = apVal[iVal++];


  if( pEq ){
    zTerm = (const char *)sqlite3_value_text(pEq);
    nTerm = sqlite3_value_bytes(pEq);
    f = FTS5INDEX_QUERY_NOTOKENDATA;
  }else{
    if( pGe ){







>







636
637
638
639
640
641
642
643
644
645
646
647
648
649
650

  UNUSED_PARAM2(zUnused, nUnused);

  fts5VocabResetCursor(pCsr);
  if( idxNum & FTS5_VOCAB_TERM_EQ ) pEq = apVal[iVal++];
  if( idxNum & FTS5_VOCAB_TERM_GE ) pGe = apVal[iVal++];
  if( idxNum & FTS5_VOCAB_TERM_LE ) pLe = apVal[iVal++];
  pCsr->colUsed = (idxNum & FTS5_VOCAB_COLUSED_MASK);

  if( pEq ){
    zTerm = (const char *)sqlite3_value_text(pEq);
    nTerm = sqlite3_value_bytes(pEq);
    f = FTS5INDEX_QUERY_NOTOKENDATA;
  }else{
    if( pGe ){
Changes to ext/fts5/test/fts5tokenizer.test.
296
297
298
299
300
301
302






































































303
304
do_execsql_test 9.4.1 { SELECT * FROM t1('"abc xyz" *'); } {}
do_test 9.4.2 { set ::flags } {prefixquery}

set ::flags [list]
do_execsql_test 9.5.1 { SELECT * FROM t1('"abc xyz*"'); } {}
do_test 9.5.2 { set ::flags } {query}








































































finish_test







>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>


296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
do_execsql_test 9.4.1 { SELECT * FROM t1('"abc xyz" *'); } {}
do_test 9.4.2 { set ::flags } {prefixquery}

set ::flags [list]
do_execsql_test 9.5.1 { SELECT * FROM t1('"abc xyz*"'); } {}
do_test 9.5.2 { set ::flags } {query}

#-------------------------------------------------------------------------
#
reset_db
do_execsql_test 10.1 {
  CREATE VIRTUAL TABLE x1 USING fts5(x, tokenize=unicode61);
  PRAGMA writable_schema = 1;
  UPDATE sqlite_schema 
  SET sql = 'CREATE VIRTUAL TABLE x1 USING fts5(x, tokenize="unicode61 error");'
  WHERE name = 'x1';
}

db close
sqlite3 db test.db

do_catchsql_test 10.2 {
  SELECT * FROM x1('abc');
} {1 {error in tokenizer constructor}}

do_catchsql_test 10.3 {
  INSERT INTO x1 VALUES('abc');
} {1 {error in tokenizer constructor}}

do_execsql_test 10.4 {
  PRAGMA writable_schema = 1;
  UPDATE sqlite_schema 
  SET sql = 'CREATE VIRTUAL TABLE x1 USING fts5(x, tokenize="nosuch error");'
  WHERE name = 'x1';
}

db close
sqlite3 db test.db

do_catchsql_test 10.5 {
  SELECT * FROM x1('abc');
} {1 {no such tokenizer: nosuch}}
do_catchsql_test 10.6 {
  INSERT INTO x1 VALUES('abc');
} {1 {no such tokenizer: nosuch}}

do_execsql_test 10.7 {
  DROP TABLE x1;
  SELECT * FROM sqlite_schema;
}

reset_db
do_execsql_test 10.8 {
  CREATE VIRTUAL TABLE x1 USING fts5(x, tokenize=unicode61);
  INSERT INTO x1 VALUES('a b c'), ('d e f'), ('a b c');
  CREATE VIRTUAL TABLE x1v USING fts5vocab(x1, row);

  PRAGMA writable_schema = 1;
  UPDATE sqlite_schema 
  SET sql = 'CREATE VIRTUAL TABLE x1 USING fts5(x, tokenize=simplify);'
  WHERE name = 'x1';
}

do_execsql_test 10.9 {
  SELECT * FROM x1v
} {
  a 2 2 b 2 2 c 2 2 d 1 1 e 1 1 f 1 1
}

db close
sqlite3 db test.db

do_execsql_test 10.10 {
  SELECT * FROM x1v
} {
  a 2 2 b 2 2 c 2 2 d 1 1 e 1 1 f 1 1
}

finish_test
Changes to ext/fts5/test/fts5trigram2.test.
97
98
99
100
101
102
103
104
105
106

107











108
109
} "\u0303(abc\u0303)"

#-------------------------------------------------------------------------
do_execsql_test 4.0 {
  CREATE VIRTUAL TABLE t4 USING fts5(z, tokenize=trigram);
} {}

breakpoint
do_execsql_test 4.1 {
  INSERT INTO t4 VALUES('ABCD');

} {}












finish_test







<


>

>
>
>
>
>
>
>
>
>
>
>


97
98
99
100
101
102
103

104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
} "\u0303(abc\u0303)"

#-------------------------------------------------------------------------
do_execsql_test 4.0 {
  CREATE VIRTUAL TABLE t4 USING fts5(z, tokenize=trigram);
} {}


do_execsql_test 4.1 {
  INSERT INTO t4 VALUES('ABCD');
  INSERT INTO t4 VALUES('DEFG');
} {}

db close
sqlite3 db test.db

do_eqp_test 4.1 {
  SELECT rowid FROM t4 WHERE z LIKE '%abc%'
} {VIRTUAL TABLE INDEX 0:L0}

do_execsql_test 4.2 {
  SELECT rowid FROM t4 WHERE z LIKE '%abc%'
} {1}

finish_test
Changes to src/expr.c.
5302
5303
5304
5305
5306
5307
5308
5309
5310
5311
5312

5313
5314
5315
5316
5317
5318
5319
5320
5321
5322
5323
5324
        return 0;
      }
      if( pExpr->affExpr==OE_Abort ){
        sqlite3MayAbort(pParse);
      }
      assert( !ExprHasProperty(pExpr, EP_IntValue) );
      if( pExpr->affExpr==OE_Ignore ){
        sqlite3VdbeAddOp4(
            v, OP_Halt, SQLITE_OK, OE_Ignore, 0, pExpr->u.zToken,0);
        VdbeCoverage(v);
      }else{

        sqlite3HaltConstraint(pParse,
             pParse->pTriggerTab ? SQLITE_CONSTRAINT_TRIGGER : SQLITE_ERROR,
             pExpr->affExpr, pExpr->u.zToken, 0, 0);
      }

      break;
    }
#endif
  }
  sqlite3ReleaseTempReg(pParse, regFree1);
  sqlite3ReleaseTempReg(pParse, regFree2);
  return inReg;







|
<


>
|

|

<







5302
5303
5304
5305
5306
5307
5308
5309

5310
5311
5312
5313
5314
5315
5316

5317
5318
5319
5320
5321
5322
5323
        return 0;
      }
      if( pExpr->affExpr==OE_Abort ){
        sqlite3MayAbort(pParse);
      }
      assert( !ExprHasProperty(pExpr, EP_IntValue) );
      if( pExpr->affExpr==OE_Ignore ){
        sqlite3VdbeAddOp2(v, OP_Halt, SQLITE_OK, OE_Ignore);

        VdbeCoverage(v);
      }else{
        r1 = sqlite3ExprCodeTemp(pParse, pExpr->pLeft, &regFree1);
        sqlite3VdbeAddOp3(v, OP_Halt, 
             pParse->pTriggerTab ? SQLITE_CONSTRAINT_TRIGGER : SQLITE_ERROR,
             pExpr->affExpr, r1);
      }

      break;
    }
#endif
  }
  sqlite3ReleaseTempReg(pParse, regFree1);
  sqlite3ReleaseTempReg(pParse, regFree2);
  return inReg;
Changes to src/fkey.c.
1324
1325
1326
1327
1328
1329
1330
1331

1332
1333
1334
1335
1336
1337
1338
    nFrom = sqlite3Strlen30(zFrom);

    if( action==OE_Restrict ){
      int iDb = sqlite3SchemaToIndex(db, pTab->pSchema);
      SrcList *pSrc;
      Expr *pRaise; 

      pRaise = sqlite3Expr(db, TK_RAISE, "FOREIGN KEY constraint failed");

      if( pRaise ){
        pRaise->affExpr = OE_Abort;
      }
      pSrc = sqlite3SrcListAppend(pParse, 0, 0, 0);
      if( pSrc ){
        assert( pSrc->nSrc==1 );
        pSrc->a[0].zName = sqlite3DbStrDup(db, zFrom);







|
>







1324
1325
1326
1327
1328
1329
1330
1331
1332
1333
1334
1335
1336
1337
1338
1339
    nFrom = sqlite3Strlen30(zFrom);

    if( action==OE_Restrict ){
      int iDb = sqlite3SchemaToIndex(db, pTab->pSchema);
      SrcList *pSrc;
      Expr *pRaise; 

      pRaise = sqlite3Expr(db, TK_STRING, "FOREIGN KEY constraint failed"),
      pRaise = sqlite3PExpr(pParse, TK_RAISE, pRaise, 0);
      if( pRaise ){
        pRaise->affExpr = OE_Abort;
      }
      pSrc = sqlite3SrcListAppend(pParse, 0, 0, 0);
      if( pSrc ){
        assert( pSrc->nSrc==1 );
        pSrc->a[0].zName = sqlite3DbStrDup(db, zFrom);
Changes to src/parse.y.
1654
1655
1656
1657
1658
1659
1660
1661
1662
1663
1664
1665
1666
1667
1668
1669
// The special RAISE expression that may occur in trigger programs
expr(A) ::= RAISE LP IGNORE RP.  {
  A = sqlite3PExpr(pParse, TK_RAISE, 0, 0); 
  if( A ){
    A->affExpr = OE_Ignore;
  }
}
expr(A) ::= RAISE LP raisetype(T) COMMA nm(Z) RP.  {
  A = sqlite3ExprAlloc(pParse->db, TK_RAISE, &Z, 1);
  if( A ) {
    A->affExpr = (char)T;
  }
}
%endif  !SQLITE_OMIT_TRIGGER

%type raisetype {int}







|
|







1654
1655
1656
1657
1658
1659
1660
1661
1662
1663
1664
1665
1666
1667
1668
1669
// The special RAISE expression that may occur in trigger programs
expr(A) ::= RAISE LP IGNORE RP.  {
  A = sqlite3PExpr(pParse, TK_RAISE, 0, 0); 
  if( A ){
    A->affExpr = OE_Ignore;
  }
}
expr(A) ::= RAISE LP raisetype(T) COMMA expr(Z) RP.  {
  A = sqlite3PExpr(pParse, TK_RAISE, Z, 0);
  if( A ) {
    A->affExpr = (char)T;
  }
}
%endif  !SQLITE_OMIT_TRIGGER

%type raisetype {int}
Changes to src/treeview.c.
814
815
816
817
818
819
820
821

822
823
824
825
826
827
828
      switch( pExpr->affExpr ){
        case OE_Rollback:   zType = "rollback";  break;
        case OE_Abort:      zType = "abort";     break;
        case OE_Fail:       zType = "fail";      break;
        case OE_Ignore:     zType = "ignore";    break;
      }
      assert( !ExprHasProperty(pExpr, EP_IntValue) );
      sqlite3TreeViewLine(pView, "RAISE %s(%Q)", zType, pExpr->u.zToken);

      break;
    }
#endif
    case TK_MATCH: {
      sqlite3TreeViewLine(pView, "MATCH {%d:%d}%s",
                          pExpr->iTable, pExpr->iColumn, zFlgs);
      sqlite3TreeViewExpr(pView, pExpr->pRight, 0);







|
>







814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
      switch( pExpr->affExpr ){
        case OE_Rollback:   zType = "rollback";  break;
        case OE_Abort:      zType = "abort";     break;
        case OE_Fail:       zType = "fail";      break;
        case OE_Ignore:     zType = "ignore";    break;
      }
      assert( !ExprHasProperty(pExpr, EP_IntValue) );
      sqlite3TreeViewLine(pView, "RAISE %s", zType);
      sqlite3TreeViewExpr(pView, pExpr->pLeft, 0);
      break;
    }
#endif
    case TK_MATCH: {
      sqlite3TreeViewLine(pView, "MATCH {%d:%d}%s",
                          pExpr->iTable, pExpr->iColumn, zFlgs);
      sqlite3TreeViewExpr(pView, pExpr->pRight, 0);
Changes to src/vdbe.c.
1210
1211
1212
1213
1214
1215
1216
1217
1218
1219
1220
1221
1222
1223
1224
1225
1226
1227
1228
1229

1230
1231



1232

1233
1234
1235
1236
1237
1238
1239
1240
1241
1242
1243
1244
1245
1246
1247
1248
1249
1250
1251
1252
1253



1254
1255
1256
1257
1258
1259
1260
  if( pOp->p2==OE_Abort ){ sqlite3VdbeAssertAbortable(p); }
#endif
  if( (pIn3->flags & MEM_Null)==0 ) break;
  /* Fall through into OP_Halt */
  /* no break */ deliberate_fall_through
}

/* Opcode:  Halt P1 P2 * P4 P5
**
** Exit immediately.  All open cursors, etc are closed
** automatically.
**
** P1 is the result code returned by sqlite3_exec(), sqlite3_reset(),
** or sqlite3_finalize().  For a normal halt, this should be SQLITE_OK (0).
** For errors, it can be some other value.  If P1!=0 then P2 will determine
** whether or not to rollback the current transaction.  Do not rollback
** if P2==OE_Fail. Do the rollback if P2==OE_Rollback.  If P2==OE_Abort,
** then back out all changes that have occurred during this execution of the
** VDBE, but do not rollback the transaction.
**

** If P4 is not null then it is an error message string.
**



** P5 is a value between 0 and 4, inclusive, that modifies the P4 string.

**
**    0:  (no change)
**    1:  NOT NULL constraint failed: P4
**    2:  UNIQUE constraint failed: P4
**    3:  CHECK constraint failed: P4
**    4:  FOREIGN KEY constraint failed: P4
**
** If P5 is not zero and P4 is NULL, then everything after the ":" is
** omitted.
**
** There is an implied "Halt 0 0 0" instruction inserted at the very end of
** every program.  So a jump past the last instruction of the program
** is the same as executing Halt.
*/
case OP_Halt: {
  VdbeFrame *pFrame;
  int pcx;

#ifdef SQLITE_DEBUG
  if( pOp->p2==OE_Abort ){ sqlite3VdbeAssertAbortable(p); }
#endif




  /* A deliberately coded "OP_Halt SQLITE_INTERNAL * * * *" opcode indicates
  ** something is wrong with the code generator.  Raise an assertion in order
  ** to bring this to the attention of fuzzers and other testing tools. */
  assert( pOp->p1!=SQLITE_INTERNAL );

  if( p->pFrame && pOp->p1==SQLITE_OK ){







|












>
|

>
>
>
|
>

<





|
|












>
>
>







1210
1211
1212
1213
1214
1215
1216
1217
1218
1219
1220
1221
1222
1223
1224
1225
1226
1227
1228
1229
1230
1231
1232
1233
1234
1235
1236
1237
1238

1239
1240
1241
1242
1243
1244
1245
1246
1247
1248
1249
1250
1251
1252
1253
1254
1255
1256
1257
1258
1259
1260
1261
1262
1263
1264
1265
1266
1267
  if( pOp->p2==OE_Abort ){ sqlite3VdbeAssertAbortable(p); }
#endif
  if( (pIn3->flags & MEM_Null)==0 ) break;
  /* Fall through into OP_Halt */
  /* no break */ deliberate_fall_through
}

/* Opcode:  Halt P1 P2 P3 P4 P5
**
** Exit immediately.  All open cursors, etc are closed
** automatically.
**
** P1 is the result code returned by sqlite3_exec(), sqlite3_reset(),
** or sqlite3_finalize().  For a normal halt, this should be SQLITE_OK (0).
** For errors, it can be some other value.  If P1!=0 then P2 will determine
** whether or not to rollback the current transaction.  Do not rollback
** if P2==OE_Fail. Do the rollback if P2==OE_Rollback.  If P2==OE_Abort,
** then back out all changes that have occurred during this execution of the
** VDBE, but do not rollback the transaction.
**
** If P3 is not zero and P4 is NULL, then P3 is a register that holds the
** text of an error message.
**
** If P3 is zero and P4 is not null then the error message string is held
** in P4.
**
** P5 is a value between 1 and 4, inclusive, then the P4 error message
** string is modified as follows:
**

**    1:  NOT NULL constraint failed: P4
**    2:  UNIQUE constraint failed: P4
**    3:  CHECK constraint failed: P4
**    4:  FOREIGN KEY constraint failed: P4
**
** If P3 is zero and P5 is not zero and P4 is NULL, then everything after
** the ":" is omitted.
**
** There is an implied "Halt 0 0 0" instruction inserted at the very end of
** every program.  So a jump past the last instruction of the program
** is the same as executing Halt.
*/
case OP_Halt: {
  VdbeFrame *pFrame;
  int pcx;

#ifdef SQLITE_DEBUG
  if( pOp->p2==OE_Abort ){ sqlite3VdbeAssertAbortable(p); }
#endif
  assert( pOp->p4type==P4_NOTUSED
       || pOp->p4type==P4_STATIC
       || pOp->p4type==P4_DYNAMIC );

  /* A deliberately coded "OP_Halt SQLITE_INTERNAL * * * *" opcode indicates
  ** something is wrong with the code generator.  Raise an assertion in order
  ** to bring this to the attention of fuzzers and other testing tools. */
  assert( pOp->p1!=SQLITE_INTERNAL );

  if( p->pFrame && pOp->p1==SQLITE_OK ){
1277
1278
1279
1280
1281
1282
1283





1284
1285
1286
1287
1288
1289
1290
1291
    pOp = &aOp[pcx];
    break;
  }
  p->rc = pOp->p1;
  p->errorAction = (u8)pOp->p2;
  assert( pOp->p5<=4 );
  if( p->rc ){





    if( pOp->p5 ){
      static const char * const azType[] = { "NOT NULL", "UNIQUE", "CHECK",
                                             "FOREIGN KEY" };
      testcase( pOp->p5==1 );
      testcase( pOp->p5==2 );
      testcase( pOp->p5==3 );
      testcase( pOp->p5==4 );
      sqlite3VdbeError(p, "%s constraint failed", azType[pOp->p5-1]);







>
>
>
>
>
|







1284
1285
1286
1287
1288
1289
1290
1291
1292
1293
1294
1295
1296
1297
1298
1299
1300
1301
1302
1303
    pOp = &aOp[pcx];
    break;
  }
  p->rc = pOp->p1;
  p->errorAction = (u8)pOp->p2;
  assert( pOp->p5<=4 );
  if( p->rc ){
    if( pOp->p3>0 && pOp->p4type==P4_NOTUSED ){
      const char *zErr;
      assert( pOp->p3<=(p->nMem + 1 - p->nCursor) );
      zErr = sqlite3ValueText(&aMem[pOp->p3], SQLITE_UTF8);
      sqlite3VdbeError(p, "%s", zErr);
    }else if( pOp->p5 ){
      static const char * const azType[] = { "NOT NULL", "UNIQUE", "CHECK",
                                             "FOREIGN KEY" };
      testcase( pOp->p5==1 );
      testcase( pOp->p5==2 );
      testcase( pOp->p5==3 );
      testcase( pOp->p5==4 );
      sqlite3VdbeError(p, "%s constraint failed", azType[pOp->p5-1]);
4320
4321
4322
4323
4324
4325
4326
4327
4328
4329
4330
4331
4332
4333
4334
4335
4336
4337
4338
4339
4340
4341
4342
4343




4344
4345
4346
4347
4348
4349
4350
  if( pOp->opcode==OP_OpenWrite ){
    assert( OPFLAG_FORDELETE==BTREE_FORDELETE );
    wrFlag = BTREE_WRCSR | (pOp->p5 & OPFLAG_FORDELETE);
    assert( sqlite3SchemaMutexHeld(db, iDb, 0) );
    if( pDb->pSchema->file_format < p->minWriteFileFormat ){
      p->minWriteFileFormat = pDb->pSchema->file_format;
    }
  }else{
    wrFlag = 0;
  }
  if( pOp->p5 & OPFLAG_P2ISREG ){
    assert( p2>0 );
    assert( p2<=(u32)(p->nMem+1 - p->nCursor) );
    assert( pOp->opcode==OP_OpenWrite );
    pIn2 = &aMem[p2];
    assert( memIsValid(pIn2) );
    assert( (pIn2->flags & MEM_Int)!=0 );
    sqlite3VdbeMemIntegerify(pIn2);
    p2 = (int)pIn2->u.i;
    /* The p2 value always comes from a prior OP_CreateBtree opcode and
    ** that opcode will always set the p2 value to 2 or more or else fail.
    ** If there were a failure, the prepared statement would have halted
    ** before reaching this instruction. */
    assert( p2>=2 );




  }
  if( pOp->p4type==P4_KEYINFO ){
    pKeyInfo = pOp->p4.pKeyInfo;
    assert( pKeyInfo->enc==ENC(db) );
    assert( pKeyInfo->db==db );
    nField = pKeyInfo->nAllField;
  }else if( pOp->p4type==P4_INT32 ){







<
<
<
|
|
|
<
|
|
|
|
|
|
|
|
|
|
>
>
>
>







4332
4333
4334
4335
4336
4337
4338



4339
4340
4341

4342
4343
4344
4345
4346
4347
4348
4349
4350
4351
4352
4353
4354
4355
4356
4357
4358
4359
4360
4361
4362
  if( pOp->opcode==OP_OpenWrite ){
    assert( OPFLAG_FORDELETE==BTREE_FORDELETE );
    wrFlag = BTREE_WRCSR | (pOp->p5 & OPFLAG_FORDELETE);
    assert( sqlite3SchemaMutexHeld(db, iDb, 0) );
    if( pDb->pSchema->file_format < p->minWriteFileFormat ){
      p->minWriteFileFormat = pDb->pSchema->file_format;
    }



    if( pOp->p5 & OPFLAG_P2ISREG ){
      assert( p2>0 );
      assert( p2<=(u32)(p->nMem+1 - p->nCursor) );

      pIn2 = &aMem[p2];
      assert( memIsValid(pIn2) );
      assert( (pIn2->flags & MEM_Int)!=0 );
      sqlite3VdbeMemIntegerify(pIn2);
      p2 = (int)pIn2->u.i;
      /* The p2 value always comes from a prior OP_CreateBtree opcode and
      ** that opcode will always set the p2 value to 2 or more or else fail.
      ** If there were a failure, the prepared statement would have halted
      ** before reaching this instruction. */
      assert( p2>=2 );
    }
  }else{
    wrFlag = 0;
    assert( (pOp->p5 & OPFLAG_P2ISREG)==0 );
  }
  if( pOp->p4type==P4_KEYINFO ){
    pKeyInfo = pOp->p4.pKeyInfo;
    assert( pKeyInfo->enc==ENC(db) );
    assert( pKeyInfo->db==db );
    nField = pKeyInfo->nAllField;
  }else if( pOp->p4type==P4_INT32 ){
Changes to src/where.c.
3166
3167
3168
3169
3170
3171
3172
3173
3174
3175
3176
3177
3178
3179
3180
      int iCol = pProbe->aiColumn[saved_nEq];
      pNew->wsFlags |= WHERE_COLUMN_EQ;
      assert( saved_nEq==pNew->u.btree.nEq );
      if( iCol==XN_ROWID
       || (iCol>=0 && nInMul==0 && saved_nEq==pProbe->nKeyCol-1)
      ){
        if( iCol==XN_ROWID || pProbe->uniqNotNull
         || (pProbe->nKeyCol==1 && pProbe->onError && eOp==WO_EQ)
        ){
          pNew->wsFlags |= WHERE_ONEROW;
        }else{
          pNew->wsFlags |= WHERE_UNQ_WANTED;
        }
      }
      if( scan.iEquiv>1 ) pNew->wsFlags |= WHERE_TRANSCONS;







|







3166
3167
3168
3169
3170
3171
3172
3173
3174
3175
3176
3177
3178
3179
3180
      int iCol = pProbe->aiColumn[saved_nEq];
      pNew->wsFlags |= WHERE_COLUMN_EQ;
      assert( saved_nEq==pNew->u.btree.nEq );
      if( iCol==XN_ROWID
       || (iCol>=0 && nInMul==0 && saved_nEq==pProbe->nKeyCol-1)
      ){
        if( iCol==XN_ROWID || pProbe->uniqNotNull
         || (pProbe->nKeyCol==1 && pProbe->onError && (eOp & WO_EQ))
        ){
          pNew->wsFlags |= WHERE_ONEROW;
        }else{
          pNew->wsFlags |= WHERE_UNQ_WANTED;
        }
      }
      if( scan.iEquiv>1 ) pNew->wsFlags |= WHERE_TRANSCONS;
Changes to test/colname.test.
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
# also caused by check-in https://sqlite.org/src/info/6b2ff26c25
#
# Prior to being fixed, the following CREATE TABLE caused an
# assertion fault.
#
do_catchsql_test colname-9.410 {
  CREATE TABLE t5 AS SELECT RAISE(abort,a);
} {1 {RAISE() may only be used within a trigger-program}}

# Make sure the quotation marks get removed from the column names
# when constructing a new table from an aggregate SELECT.
# Email from Juergen Palm on 2017-07-11.
#
do_execsql_test colname-10.100 {
  DROP TABLE IF EXISTS t1;







|







420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
# also caused by check-in https://sqlite.org/src/info/6b2ff26c25
#
# Prior to being fixed, the following CREATE TABLE caused an
# assertion fault.
#
do_catchsql_test colname-9.410 {
  CREATE TABLE t5 AS SELECT RAISE(abort,a);
} {1 {no such column: a}}

# Make sure the quotation marks get removed from the column names
# when constructing a new table from an aggregate SELECT.
# Email from Juergen Palm on 2017-07-11.
#
do_execsql_test colname-10.100 {
  DROP TABLE IF EXISTS t1;
Changes to test/trigger1.test.
833
834
835
836
837
838
839













840
841
reset_db
do_catchsql_test trigger1-23.1 {
  CREATE TABLE t1(a INT);
  CREATE TRIGGER r1 AFTER INSERT ON t1 BEGIN
    INSERT INTO t1 SELECT e_master LIMIT 1,#1;
  END;
} {1 {near "#1": syntax error}}














finish_test







>
>
>
>
>
>
>
>
>
>
>
>
>


833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
reset_db
do_catchsql_test trigger1-23.1 {
  CREATE TABLE t1(a INT);
  CREATE TRIGGER r1 AFTER INSERT ON t1 BEGIN
    INSERT INTO t1 SELECT e_master LIMIT 1,#1;
  END;
} {1 {near "#1": syntax error}}

# 2024-05-08 Allow arbitrary expressions as the 2nd argument to RAISE().
#
do_catchsql_test trigger1-24.1 {
  CREATE TRIGGER r1 AFTER INSERT ON t1 BEGIN
    SELECT raise(abort,format('attempt to insert %d where is not a power of 2',new.a))
     WHERE (new.a & (new.a-1))!=0;
  END;
  INSERT INTO t1 VALUES(0),(1),(2),(4),(8),(65536);
} {0 {}}
do_catchsql_test trigger1-24.2 {
  INSERT INTO t1 VALUES(9876);
} {1 {attempt to insert 9876 where is not a power of 2}}

finish_test