/ File History
Login

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

History of ext/fts5/fts5_tokenize.c

2019-04-13
04:38
Use the 64-bit memory allocator interfaces in extensions, whenever possible. file: [2e508c6a] check-in: [07ee06fd] user: drh branch: trunk, size: 34630
2019-01-08
20:02
Use 64-bit math to compute the sizes of memory allocations in extensions. file: [4d904c23] check-in: [ca67f2ec] user: drh branch: trunk, size: 34628
2018-12-31
21:43
Fix harmless compiler warnings. file: [8b7ef00c] check-in: [b57c545a] user: drh branch: trunk, size: 34557
2018-12-28
14:33
Avoid an undefined left-shift operation in fts5 caused by malformed utf-8 text. file: [d49f479c] check-in: [c3a3a111] user: dan branch: trunk, size: 34554
07:37
Fix problems in fts5 found by ASAN. file: [240f849c] check-in: [c564bf87] user: dan branch: trunk, size: 34559
2018-12-03
16:14
Add the "remove_diacritics=2" option to the unicode61 tokenizer in both FTS5 and FTS3/4. file: [ca2b6a03] check-in: [06177f3f] user: dan branch: trunk, size: 34549
2018-07-13
19:52
Add the "categories" option to the unicode61 tokenizer in fts5. file: [ebd13d03] check-in: [80d2b9e6] user: dan branch: trunk, size: 34058
2016-02-11
17:01
Handle parser stack overflow when parsing fts5 query expressions. Fix some compiler warnings in fts5 code. file: [2ce7b441] check-in: [bc3f7900] user: dan branch: trunk, size: 33283
2016-01-23
18:51
Fix an fts5 problem with using both xPhraseFirst() and xPhraseFirstColumn() within a single statement in detail=col mode. file: [4d5c4f18] check-in: [72d53699] user: dan branch: fts5-perf, size: 33170
2015-12-23
16:42
Fix some harmless gcc compiler warnings. Mostly in fts5, but also two in the core code. file: [504984ac] check-in: [5d44d4a6] user: dan branch: trunk, size: 33175
2015-12-16
23:30
Fix even more harmless compiler warnings. file: [618efe03] check-in: [1d0e6aa1] user: mistachkin branch: msvcWarn, size: 33187
2015-10-14
20:34
Fix harmless compiler warnings. file: [12c5d925] check-in: [1c46c194] user: mistachkin branch: trunk, size: 33172
2015-09-02
19:48
Further tests to raise coverage of fts5 synonym code to 100%. Fix a dropped error code in the same. file: [f380f46f] check-in: [bdedd838] user: dan branch: fts5-incompatible, size: 33167
2015-08-29
15:44
Another change to the fts5 tokenizer API. file: [71054151] check-in: [fc718684] user: dan branch: fts5-incompatible, size: 33174
2015-08-28
19:56
Change the fts5 tokenizer API to allow more than one token to occupy a single position within a document. file: [07a89441] check-in: [90b85b42] user: dan branch: fts5-incompatible, size: 33224
2015-07-31
14:43
Fix a bug in the fts5 porter tokenizer preventing it from passing xCreate() arguments through to its parent tokenizer. file: [2836f672] check-in: [c3c672af] user: dan branch: trunk, size: 33071
2015-07-02
15:52
Remove "#ifdef SQLITE_ENABLE_FTS5" from individual fts5 source files. Add a single "#if !defined(SQLITE_CORE) || defined(SQLITE_ENABLE_FTS5)" to fts5.c. file: [30f97a8c] check-in: [7819002e] user: dan branch: trunk, size: 32972
2015-05-30
11:49
Remove the "#include sqlite3Int.h" from fts5Int.h. file: [97251d68] check-in: [e008c3c8] user: dan branch: fts5, size: 33045
2015-05-22
06:08
Improve test coverage of fts5_unicode2.c. file: [24649425] check-in: [fea8a4db] user: dan branch: fts5, size: 33082
2015-05-20
09:27
Improve test coverage of fts5_tokenize.c. file: [6f4d2cbe] check-in: [0e91a6a5] user: dan branch: fts5, size: 33260
2015-05-19
11:32
Fix a memory leak that could follow an OOM condition in fts5. file: [4d9d5047] check-in: [de9f8ef6] user: dan branch: fts5, size: 33245
2015-04-29
20:54
Improve fts5 tests. file: [830eae0d] check-in: [c1f07a3a] user: dan branch: fts5, size: 33150
2015-03-11
14:51
Add an optimization to the fts5 unicode tokenizer code. file: [c07f2c2f] check-in: [f5db4892] user: dan branch: fts5, size: 33078
2015-03-04
08:29
Fix a couple of build problems. file: [c3fe3091] check-in: [a5d5468c] user: dan branch: fts5, size: 33019
2015-02-02
11:32
Fix some problems with building fts5 and fts3 together using the amalgamation. file: [0d108148] check-in: [fb10bbb9] user: dan branch: fts5, size: 32945
2015-01-17
17:48
Improve the performance of the fts5 porter tokenizer implementation. file: [7c61d5c3] check-in: [96ea6004] user: dan branch: fts5, size: 32932
2015-01-12
17:58
Optimize the unicode61 tokenizer so that it handles ascii text faster. Make it the default tokenizer. Change the name of the simple tokenizer to "ascii". file: [bdb6a1f5] check-in: [f22dbcca] user: dan branch: fts5, size: 24110
2015-01-06
19:08
Remove the iPos parameter from the tokenizer callback. Fix the "tokenchars" and "separators" options on the simple tokenizer. file: [4c30cf32] check-in: [65f0262f] user: dan branch: fts5, size: 22293
2015-01-01
16:46
Add a version of the unicode61 tokenizer to fts5. file: [5a0ad464] check-in: [d09f7800] user: dan branch: fts5, size: 21276
2014-12-29
11:24
Fixes to built-in tokenizers. file: [5d6e7853] check-in: [b33fe0dd] user: dan branch: fts5, size: 12914
2014-11-15
20:07
Fix the customization interfaces so that they match the documentation. file: [8360c0d1] check-in: [fba0b5fc] user: dan branch: fts5, size: 3561 Added