SQLite

Check-in [f5732f4caf]
Login

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

Overview
Comment:The fts3_tokenizer() function returns NULL if the SQLITE_DBCONFIG_ENABLE_FTS_TOKENIZER setting is disabled, which is is by default.
Downloads: Tarball | ZIP archive
Timelines: family | ancestors | descendants | both | trunk
Files: files | file ages | folders
SHA3-256: f5732f4caf7a37a6445c61ae0d0ac14cc9deb897376e73aa36a1ead025b92c69
User & Date: drh 2019-03-01 21:33:29.039
Context
2019-03-02
12:17
Remove a redundant parameter extraction call from the percentile.c extension. (check-in: d49c32e6e7 user: drh tags: trunk)
2019-03-01
21:37
Merge recent changes from trunk. (check-in: f270245a7b user: drh tags: apple-osx)
21:33
The fts3_tokenizer() function returns NULL if the SQLITE_DBCONFIG_ENABLE_FTS_TOKENIZER setting is disabled, which is is by default. (check-in: f5732f4caf user: drh tags: trunk)
21:12
Fix a minor comment typo. No code changes. (check-in: 9a0a93c89d user: drh tags: trunk)
Changes
Side-by-Side Diff Ignore Whitespace Patch
Changes to ext/fts3/README.tokenizers.
︙
48
49
50
51
52
53
54
55
56




57
58
59
60
61
62
63

48
49
50
51
52
53
54


55
56
57
58
59
60
61
62
63
64
65








-
-
+
+
+
+








  returned. If only one argument is passed, a pointer to the tokenizer
  implementation currently registered as <tokenizer-name> is returned,
  encoded as a blob. Or, if no such tokenizer exists, an SQL exception
  (error) is raised.

  SECURITY: If the fts3 extension is used in an environment where potentially
    malicious users may execute arbitrary SQL (i.e. gears), they should be
    prevented from invoking the fts3_tokenizer() function, possibly using the
    authorisation callback.
    prevented from invoking the fts3_tokenizer() function.  The
    fts3_tokenizer() function is disabled by default. It is only enabled
    by SQLITE_DBCONFIG_ENABLE_FTS3_TOKENIZER. Do not enable it in
    security sensitive environments.

  See "Sample code" below for an example of calling the fts3_tokenizer()
  function from C code.

3. ICU Library Tokenizers

  If this extension is compiled with the SQLITE_ENABLE_ICU pre-processor
︙
Changes to ext/fts3/fts3_tokenizer.c.
︙
102
103
104
105
106
107
108

109


110
111
112
113
114
115
116

102
103
104
105
106
107
108
109

110
111
112
113
114
115
116
117
118








+
-
+
+








    if( !pPtr ){
      char *zErr = sqlite3_mprintf("unknown tokenizer: %s", zName);
      sqlite3_result_error(context, zErr, -1);
      sqlite3_free(zErr);
      return;
    }
  }
  if( fts3TokenizerEnabled(context) ){
  sqlite3_result_blob(context, (void *)&pPtr, sizeof(pPtr), SQLITE_TRANSIENT);
    sqlite3_result_blob(context, (void *)&pPtr, sizeof(pPtr), SQLITE_TRANSIENT);
  }
}

int sqlite3Fts3IsIdChar(char c){
  static const char isFtsIdChar[] = {
      0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,  /* 0x */
      0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,  /* 1x */
      0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,  /* 2x */
︙
Changes to src/sqlite.h.in.
︙
2082
2083
2084
2085
2086
2087
2088
2089
2090


2091
2092
2093
2094
2095
2096
2097

2082
2083
2084
2085
2086
2087
2088


2089
2090
2091
2092
2093
2094
2095
2096
2097








-
-
+
+








** The second parameter is a pointer to an integer into which
** is written 0 or 1 to indicate whether triggers are disabled or enabled
** following this call.  The second parameter may be a NULL pointer, in
** which case the trigger setting is not reported back. </dd>
**
** [[SQLITE_DBCONFIG_ENABLE_FTS3_TOKENIZER]]
** <dt>SQLITE_DBCONFIG_ENABLE_FTS3_TOKENIZER</dt>
** <dd> ^This option is used to enable or disable the two-argument
** version of the [fts3_tokenizer()] function which is part of the
** <dd> ^This option is used to enable or disable the
** [fts3_tokenizer()] function which is part of the
** [FTS3] full-text search engine extension.
** There should be two additional arguments.
** The first argument is an integer which is 0 to disable fts3_tokenizer() or
** positive to enable fts3_tokenizer() or negative to leave the setting
** unchanged.
** The second parameter is a pointer to an integer into which
** is written 0 or 1 to indicate whether fts3_tokenizer is disabled or enabled
︙
Changes to test/fts3atoken.test.
︙
103
104
105
106
107
108
109

110
111
112
113
114
115
116

103
104
105
106
107
108
109
110
111
112
113
114
115
116
117








+








} {1 {unknown tokenizer: nosuchtokenizer}}

#--------------------------------------------------------------------------
# Test cases fts3atoken-3.* test the three built-in tokenizers with a
# simple input string via the built-in test function. This is as much
# to test the test function as the tokenizer implementations.
#
sqlite3_db_config db SQLITE_DBCONFIG_ENABLE_FTS3_TOKENIZER 1
do_test fts3atoken-3.1 {
  execsql {
    SELECT fts3_tokenizer_test('simple', 'I don''t see how');
  }
} {{0 i I 1 don don 2 t t 3 see see 4 how how}}
do_test fts3atoken-3.2 {
  execsql {
︙