/ History for ext/fts5/fts5_tokenize.c
Login

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

History for ext/fts5/fts5_tokenize.c

2019-04-13
04:38
[2e508c6a] part of check-in [07ee06fd] Use the 64-bit memory allocator interfaces in extensions, whenever possible. (check-in: [07ee06fd] user: drh branch: trunk, size: 34630)
2019-01-08
20:02
[4d904c23] part of check-in [ca67f2ec] Use 64-bit math to compute the sizes of memory allocations in extensions. (check-in: [ca67f2ec] user: drh branch: trunk, size: 34628)
2018-12-31
21:43
[8b7ef00c] part of check-in [b57c545a] Fix harmless compiler warnings. (check-in: [b57c545a] user: drh branch: trunk, size: 34557)
2018-12-28
14:33
[d49f479c] part of check-in [c3a3a111] Avoid an undefined left-shift operation in fts5 caused by malformed utf-8 text. (check-in: [c3a3a111] user: dan branch: trunk, size: 34554)
07:37
[240f849c] part of check-in [c564bf87] Fix problems in fts5 found by ASAN. (check-in: [c564bf87] user: dan branch: trunk, size: 34559)
2018-12-03
16:14
[ca2b6a03] part of check-in [06177f3f] Add the "remove_diacritics=2" option to the unicode61 tokenizer in both FTS5 and FTS3/4. (check-in: [06177f3f] user: dan branch: trunk, size: 34549)
2018-07-13
19:52
[ebd13d03] part of check-in [80d2b9e6] Add the "categories" option to the unicode61 tokenizer in fts5. (check-in: [80d2b9e6] user: dan branch: trunk, size: 34058)
2016-02-11
17:01
[2ce7b441] part of check-in [bc3f7900] Handle parser stack overflow when parsing fts5 query expressions. Fix some compiler warnings in fts5 code. (check-in: [bc3f7900] user: dan branch: trunk, size: 33283)
2016-01-23
18:51
[4d5c4f18] part of check-in [72d53699] Fix an fts5 problem with using both xPhraseFirst() and xPhraseFirstColumn() within a single statement in detail=col mode. (check-in: [72d53699] user: dan branch: fts5-perf, size: 33170)
2015-12-23
16:42
[504984ac] part of check-in [5d44d4a6] Fix some harmless gcc compiler warnings. Mostly in fts5, but also two in the core code. (check-in: [5d44d4a6] user: dan branch: trunk, size: 33175)
2015-12-16
23:30
[618efe03] part of check-in [1d0e6aa1] Fix even more harmless compiler warnings. (check-in: [1d0e6aa1] user: mistachkin branch: msvcWarn, size: 33187)
2015-10-14
20:34
[12c5d925] part of check-in [1c46c194] Fix harmless compiler warnings. (check-in: [1c46c194] user: mistachkin branch: trunk, size: 33172)
2015-09-02
19:48
[f380f46f] part of check-in [bdedd838] Further tests to raise coverage of fts5 synonym code to 100%. Fix a dropped error code in the same. (check-in: [bdedd838] user: dan branch: fts5-incompatible, size: 33167)
2015-08-29
15:44
[71054151] part of check-in [fc718684] Another change to the fts5 tokenizer API. (check-in: [fc718684] user: dan branch: fts5-incompatible, size: 33174)
2015-08-28
19:56
[07a89441] part of check-in [90b85b42] Change the fts5 tokenizer API to allow more than one token to occupy a single position within a document. (check-in: [90b85b42] user: dan branch: fts5-incompatible, size: 33224)
2015-07-31
14:43
[2836f672] part of check-in [c3c672af] Fix a bug in the fts5 porter tokenizer preventing it from passing xCreate() arguments through to its parent tokenizer. (check-in: [c3c672af] user: dan branch: trunk, size: 33071)
2015-07-02
15:52
[30f97a8c] part of check-in [7819002e] Remove "#ifdef SQLITE_ENABLE_FTS5" from individual fts5 source files. Add a single "#if !defined(SQLITE_CORE) || defined(SQLITE_ENABLE_FTS5)" to fts5.c. (check-in: [7819002e] user: dan branch: trunk, size: 32972)
2015-05-30
11:49
[97251d68] part of check-in [e008c3c8] Remove the "#include sqlite3Int.h" from fts5Int.h. (check-in: [e008c3c8] user: dan branch: fts5, size: 33045)
2015-05-22
06:08
[24649425] part of check-in [fea8a4db] Improve test coverage of fts5_unicode2.c. (check-in: [fea8a4db] user: dan branch: fts5, size: 33082)
2015-05-20
09:27
[6f4d2cbe] part of check-in [0e91a6a5] Improve test coverage of fts5_tokenize.c. (check-in: [0e91a6a5] user: dan branch: fts5, size: 33260)
2015-05-19
11:32
[4d9d5047] part of check-in [de9f8ef6] Fix a memory leak that could follow an OOM condition in fts5. (check-in: [de9f8ef6] user: dan branch: fts5, size: 33245)
2015-04-29
20:54
[830eae0d] part of check-in [c1f07a3a] Improve fts5 tests. (check-in: [c1f07a3a] user: dan branch: fts5, size: 33150)
2015-03-11
14:51
[c07f2c2f] part of check-in [f5db4892] Add an optimization to the fts5 unicode tokenizer code. (check-in: [f5db4892] user: dan branch: fts5, size: 33078)
2015-03-04
08:29
[c3fe3091] part of check-in [a5d5468c] Fix a couple of build problems. (check-in: [a5d5468c] user: dan branch: fts5, size: 33019)
2015-02-02
11:32
[0d108148] part of check-in [fb10bbb9] Fix some problems with building fts5 and fts3 together using the amalgamation. (check-in: [fb10bbb9] user: dan branch: fts5, size: 32945)
2015-01-17
17:48
[7c61d5c3] part of check-in [96ea6004] Improve the performance of the fts5 porter tokenizer implementation. (check-in: [96ea6004] user: dan branch: fts5, size: 32932)
2015-01-12
17:58
[bdb6a1f5] part of check-in [f22dbcca] Optimize the unicode61 tokenizer so that it handles ascii text faster. Make it the default tokenizer. Change the name of the simple tokenizer to "ascii". (check-in: [f22dbcca] user: dan branch: fts5, size: 24110)
2015-01-06
19:08
[4c30cf32] part of check-in [65f0262f] Remove the iPos parameter from the tokenizer callback. Fix the "tokenchars" and "separators" options on the simple tokenizer. (check-in: [65f0262f] user: dan branch: fts5, size: 22293)
2015-01-01
16:46
[5a0ad464] part of check-in [d09f7800] Add a version of the unicode61 tokenizer to fts5. (check-in: [d09f7800] user: dan branch: fts5, size: 21276)
2014-12-29
11:24
[5d6e7853] part of check-in [b33fe0dd] Fixes to built-in tokenizers. (check-in: [b33fe0dd] user: dan branch: fts5, size: 12914)
2014-11-15
20:07
[8360c0d1] part of check-in [fba0b5fc] Fix the customization interfaces so that they match the documentation. (check-in: [fba0b5fc] user: dan branch: fts5, size: 3561) Added