/ File History

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

History of ext/fts2/fts2.c

Modify several extensions to use the new exported function naming. Fix some shared library compilation issues. file: [72c816a9] check-in: [f2ab8747] user: mistachkin branch: extRefactor, size: 220043
Many spelling fixes in comments. No changes to code. file: [b48cc0bb] check-in: [6f6e2d50] user: mistachkin branch: trunk, size: 220005
Fix all known instances of 'repeated the' style typos in comments. No changes to code. file: [4ef7d7ec] check-in: [9b19b847] user: mistachkin branch: trunk, size: 220005
Completely remove all trace of ctype.h from FTS2. file: [238e9e19] check-in: [87684566] user: drh branch: trunk, size: 220009
Corrected typos and misspellings. Ticket #3702. (CVS 6336) file: [6cbd0fbd] check-in: [6404afa0] user: shane branch: trunk, size: 219950
Backport http://www.sqlite.org/cvstrac/chngview?cn=5489 from fts3. Re-used prepared statement from fts2 cursor. (CVS 5499) file: [bc78da57] check-in: [02870ed2] user: shess branch: trunk, size: 219948
Be a bit more susicious of invalid results from the tokenizer. Backports check-in (4514) from fts3. (CVS 5459) file: [5f6f8fa8] check-in: [311aeb9c] user: shess branch: trunk, size: 219164
Implement optimize() function. Backports check-in (5417) from fts3. (CVS 5458) file: [ff1d7646] check-in: [c16900dc] user: shess branch: trunk, size: 219060
Delete all fts2 index data the table becomes empty. Backports check-in (5413) from fts3. (CVS 5457) file: [c0d28766] check-in: [4c98179b] user: shess branch: trunk, size: 208065
fts2 functions for testing scripts. Backports (5340) from fts3. (CVS 5456) file: [af6d1136] check-in: [4e47394b] user: shess branch: trunk, size: 206294
Change prefix search from O(N*M) to O(NlogM). Backports (4599) from fts3. (CVS 5455) file: [7a2e88d1] check-in: [3f614453] user: shess branch: trunk, size: 192085
Changes fts2 to use only sqlite3_malloc() and not system malloc. Backports (4554) and (4555) from fts3. (CVS 5454) file: [f50c7fac] check-in: [ecf2dec6] user: shess branch: trunk, size: 188568
Remove all instances of sprintf() from the FTS modules. Ticket #3049. (CVS 4996) file: [015d44a4] check-in: [062bf5d4] user: drh branch: trunk, size: 188071
Change all instances of "it's" in comments to either "its" or "it is", as appropriate, in case the comments are ever again read by a pedantic grammarian. Ticket #2840. (CVS 4629) file: [cdbace1c] check-in: [4e91a267] user: drh branch: trunk, size: 188046
Do not require SQLITE_ENABLE_BROKEN_FTS2 if FTS2 is not enabled. The same for FTS1. Ticket #2777. (CVS 4556) file: [0f978f0c] check-in: [f94cdcfd] user: drh branch: trunk, size: 188040
Don't do anything when input doclists are both empty. Ticket #2774 (CVS 4546) file: [9c7d635a] check-in: [75cb46f8] user: shess branch: trunk, size: 187966
Drop the forced error from fts3.c and add forced errors to fts2.c and fts1.c. (CVS 4427) file: [02720dd6] check-in: [fec6567a] user: shess branch: trunk, size: 187935
Fix memory leak of InteriorReader.term. Comes up when doing queries against large segments. (CVS 4315) file: [9a02a0db] check-in: [6c617bd8] user: shess branch: trunk, size: 186866
Convert fts2 to use sqlite3_prepare_v2() to prevent certain logic errors around SQLITE_SCHEMA handling. This also allows sql_step_statement() and sql_step_leaf_statement() to be replaced with sqlite3_step().    Also fix a logic error in flushPendingTerms() which was clearing the term table in case of error. This was wrong in the face of SQLITE_SCHEMA. Even though the change to sqlite3_prepare_v2() should cause us not to see SQLITE_SCHEMA any longer, it was still a logic error... (CVS 4205) file: [29992419] check-in: [16730cb1] user: shess branch: trunk, size: 186829
Fix some compiler warnings. (CVS 4196) file: [41224229] check-in: [6cc15409] user: drh branch: trunk, size: 189405
Fix ticket #2439: the FTS1 and FTS2 extensions use the non-standard, unportable and highly deprecated <malloc.h> header on all platforms except Apple Mac OS X. The <malloc.h> actually is never required on any OS with an at least partly POSIX-conforming API as the malloc(3) & friends functions officially live in <stdlib.h> since over 10 years. Under some platform like FreeBSD the inclusion of <malloc.h> since a few years even causes an "#error" and this way a build failure. So, just get rid of the bad <malloc.h> usage in FTS1 and FTS2 extensions at all and stick with <stdlib.h> there only. (CVS 4191) file: [6d7f8546] check-in: [3f9a6661] user: rse branch: trunk, size: 189504
Modify handling of SQLITE_SCHEMA in fts2 code. An SQLITE_SCHEMA error may cause SQLite to reload the internal schema, deleting and recreating v-table objects. So the sqlite3_vtab structure can be deleted out from under a v-table implementation. (CVS 4151) file: [41a63f6e] check-in: [dee1a0fd] user: danielk1977 branch: trunk, size: 189555
Implement xRename() for fts2 so that it is possible to rename fts2 tables. (CVS 4143) file: [4a177d5f] check-in: [488474fd] user: danielk1977 branch: trunk, size: 189649
Remove the unused EXTSRC variable from the non-configure makefile. (CVS 4129) file: [1e1b6b6e] check-in: [bbdcf372] user: danielk1977 branch: trunk, size: 189028
Put #ifdefs in fts2_tokenizer so that the build works even when FTS2 is omitted. Add the SQLite blessing to the header comments on all FTS2 source files. (CVS 4120) file: [e6015f3a] check-in: [c795e6fd] user: drh branch: trunk, size: 188960
All the use of MySQL-style quoting in the FTS modules. Ticket #2446. (CVS 4119) file: [d402141b] check-in: [3be2a6d1] user: drh branch: trunk, size: 188642
Extend fts2 so that user defined tokenizers may be added. Add a tokenizer that uses the ICU library if available. Documentation and tests to come. (CVS 4108) file: [841766f2] check-in: [68677e42] user: danielk1977 branch: trunk, size: 188628
Fix snippet generation when the left-most column of an fts2 table is used in the MATCH clause. Fix for ticket #2429. (CVS 4095) file: [8f9bd5fc] check-in: [fec56ad2] user: danielk1977 branch: trunk, size: 185869
Minor comment edits from my prefix development client. No code changes. (CVS 4058) file: [b058569b] check-in: [6953cd09] user: shess branch: trunk, size: 185850
Fix overzealous fts2 assertions WRT rowid 0 or lower. Only check that docids are ascending if there was a prior docid set for the doclist, ignore the initial docid of 0. (CVS 4026) file: [4c68ff4f] check-in: [ed3a131f] user: shess branch: trunk, size: 184751
Enable prefix-search in query-parsing and snippet generation. If the character immediately after the end of a term is '*', that term is marked for prefix matching. Modify term comparison in snippetOffsetsOfColumn() to respect isPrefix. fts2n.test runs prefix searching through some obvious test cases. (CVS 3893) file: [9e1f5942] check-in: [7c4c6592] user: shess branch: trunk, size: 184426
Modify loadSegmentLeavesInt() to correctly handle prefix searching. The new function docListUnion() is used to accumulate a union of the hits for the matching terms, which will be merged across segments using docListMerge(). (CVS 3891) file: [a6762b7a] check-in: [72c79630] user: shess branch: trunk, size: 184206
Propagate prefix flag through implementation of doclist query code. Also implement correct prefix-handling for traversal of interior nodes of segment tree. A given prefix can span multiple children of an interior node, and from there the branches need to be followed in parallel. (CVS 3889) file: [c750b2db] check-in: [cae844a0] user: shess branch: trunk, size: 180043
Lift docListMerge() call out of loadSegmentLeavesInt() for prefix search. Doclists from multiple prefix matches will need a union merge function, which will have to logically happen across a segment before doclists are merged between segments. (CVS 3887) file: [cb7ca4e3] check-in: [7ddb8266] user: shess branch: trunk, size: 178037
Break interior-node and leaf-node readers apart in loadSegment(). Previously, the code looped until the block was a leaf node as indicated by a leading NUL. Now the code loops until it finds a block in the range of leaf nodes for this segment, then reads it using LeavesReader. This will make it easier to traverse a range of leaves when doing a prefix search. (CVS 3884) file: [b45e07a2] check-in: [9466367d] user: shess branch: trunk, size: 177148
Lift code to traverse interior nodes out of loadSegment(). Refactoring towards prefix searching. (CVS 3882) file: [c1e7528d] check-in: [25935db7] user: shess branch: trunk, size: 174858
Refactor fts2 loadSegmentLeaf() in preparation for prefix-searching. Prefix-searching will want to accumulate data across multiple leaves in the segment, using LeavesReader instead of LeafReader is the first step in that direction. (CVS 3881) file: [430ef109] check-in: [22ffdae4] user: shess branch: trunk, size: 174678
Fix bug in fts2 handling of OR queries. When one doclist ends before the other, the code potentially tries to read past the end of the doclist. http://www.sqlite.org/cvstrac/tktview?tn=2309 (CVS 3862) file: [dd35df80] check-in: [dfac6082] user: shess branch: trunk, size: 173978
Fix crash in delete when existing row has null fields. Previous code assumed that the row had values in all columns, sigh. Fixes bug http://www.sqlite.org/cvstrac/tktview?tn=2289 . (CVS 3833) file: [acfce1c9] check-in: [81be7290] user: shess branch: trunk, size: 173824
Buffer updates per-transaction rather than per-update. If lots of updates happen within a single transaction, there was a lot of wasted encode/decode overhead due to segment merges. This code buffers updates in memory and writes out larger level-0 segments. It only works when documents are presented in ascending order by docid. Comparing a test set running 100 documents per transaction, the total runtime is cut almost in half. (CVS 3751) file: [8d69d6e4] check-in: [0229cba6] user: shess branch: trunk, size: 173657
Don't call ctype functions on hi-bit chars. Some platforms raise assertions when this occurs, and it's almost certainly not the right thing to do in the first place. (CVS 3746) file: [2e3cb46d] check-in: [f6c3abdc] user: shess branch: trunk, size: 168737
Refactor PLWriter to remove owned buffer. DLCollector (Document List Collector) now handles the case where PLWriter (Position List Writer) needed a local buffer. Change to using the associated DLWriter (Document List Writer) buffer, which reduces the number of memory copies needed in doclist processing, and brings PLWriter operation in line with DLWriter operation. (CVS 3707) file: [de8321a2] check-in: [d04fa3a1] user: shess branch: trunk, size: 168105
Refactor PLWriter in preparation for buffered-document change. Currently, PLWriter (Position List Writer) creates a locally-owned DataBuffer to write into. This is necessary to support doclist collection during tokenization, where there is no obvious buffer to write output to, but is not necessary for the other users of PLWriter. This change adds a DLCollector (Doc List Collector) structure to handle the tokenization case.    Also fix a potential memory leak in writeZeroSegment(). In case of error from leafWriterStep(), the DataBuffer dl was being leaked. (CVS 3706) file: [aba63e7f] check-in: [1b9918e2] user: shess branch: trunk, size: 166906
http://www.sqlite.org/cvstrac/tktview?tn=2219    When creating fts tables in an attached database, the backing tables are created in database 'main'. This change propagates the appropriate database name to the routines which build sql statements.    Note that I propagate the database name and table name separately. I briefly considered just making the table name be "db.table", but it didn't fit so well in the model used to store the table name and other information, and having the db name passed separately seemed a bit more transparent. (CVS 3631) file: [a49ed729] check-in: [283385d2] user: shess branch: trunk, size: 166020
http://www.sqlite.org/cvstrac/tktview?tn=2166,35    Calling UPDATE against an fts table in a UTF-16 database inserts corrupted data into the database. The UTF-8 data is being inserted directly. This appears to happen because sqlite3_ value_text() destructively coerces a value to UTF-8, and it's never converted back when updating the table. This works around the problem by rearranging things so that the update happens before the coercion. (CVS 3596) file: [5f7247b8] check-in: [4f2ab4b6] user: shess branch: trunk, size: 165480
Drop a couple variables which are no longer used anywhere. (CVS 3524) file: [5424f41f] check-in: [08c2cc0e] user: shess branch: trunk, size: 165480
http://www.sqlite.org/cvstrac/tktview?tn=2046    The virtual table interface allows for a cursor to field multiple xFilter() calls. For instance, if a join is done with a virtual table, there could be a call for each row which potentially matches. Unfortunately, fulltextFilter() assumes that it has a fresh cursor, and overwrites a prepared statement and a malloc'ed pointer, resulting in unfinalized statements and a memory leak.    This change hacks the code to manually clean up offending items in fulltextFilter(), emphasis on "hacks", since it's a fragile fix insofar as future additions to fulltext_cursor could continue to have the problem. (CVS 3521) file: [94b43848] check-in: [18142fdb] user: shess branch: trunk, size: 165500
Delta-encode terms in interior nodes. While experiments have shown that this is of marginal utility when encoding terms resulting from regular English text, it turns out to be very useful when encoding inputs with very large terms. (CVS 3520) file: [6065a73a] check-in: [c8151a99] user: shess branch: trunk, size: 164964
Store minimal terms in interior nodes. Whenever there's a break between leaf nodes, instead of storing the entire leftmost term of the rightmost child, store only that portion of the leftmost term necessary to distinguish it from the rightmost term of the leftmost child. (CVS 3513) file: [74a5db3f] check-in: [f6e0b080] user: shess branch: trunk, size: 162408
Refactoring groundwork for coming work on interior nodes. Change LeafWriter to use empty data buffer (instead of empty term) to detect an empty block. Code to validate interior nodes. Moderate revisions to leaf-node and doclist validation. Recast leafWriterStep() in terms of LeafWriterStepMerge(). (CVS 3512) file: [57d8cd57] check-in: [f30771d5] user: shess branch: trunk, size: 161513
Delta-encode docids. This is good for around 22% reduction in index size with DL_POSITIONS. It improves performance about 5%-6%. (CVS 3511) file: [79093817] check-in: [9b6d413d] user: shess branch: trunk, size: 160797
Require a minimum fanout for interior nodes. This prevents cases where excessively large terms keep the tree from finding a single root. A downside is that this could result in large interior nodes in the presence of large terms, which may be prone to fragmentation, though if the nodes were smaller that would translate into more levels in the tree, which would also have that problem. (CVS 3510) file: [667a93b3] check-in: [64b7e340] user: shess branch: trunk, size: 159745
Allow backing tables to be missing on dropping fts table. Fixes http://www.sqlite.org/cvstrac/tktview?tn=1992,35 . (CVS 3509) file: [9b28f218] check-in: [9628a61a] user: shess branch: trunk, size: 158927
Fix a pair of memory leaks. These were turned up by running valgrind memcheck with various 10k doc insert, update, delete, and query tests. (CVS 3497) file: [afa395ab] check-in: [3cd9b64b] user: shess branch: trunk, size: 158897
Empty queries should get no results. My recent change ( http://www.sqlite.org/cvstrac/chngview?cn=3486 ) broke test fts2a-5.3. This change should make the expected result more obvious. (CVS 3489) file: [10fe8d96] check-in: [cde383eb] user: shess branch: trunk, size: 158030
Make memset() uses less error-prone. http://www.sqlite.org/cvstrac/tktview?tn=2036,35 describes some cases where we were passing memset() a length which was the sizeof a pointer, rather than the structure pointed to. Instead, wrap this idiom up in CLEAR() and SCRAMBLE() macros. (CVS 3488) file: [bee8988d] check-in: [5878add0] user: shess branch: trunk, size: 157895
Replace the DocList and DocListReader structures. The new structures distinguish reading from a static buffer from writing to a dynamic buffer. This allows n-way doclist merging, and in-place merging of segment leaf nodes, which together cut segment merge times in half. (CVS 3486) file: [f3d0b37b] check-in: [af5bfb98] user: shess branch: trunk, size: 158065
Don't store empty segments. When inserting empty strings, the code was writing out a segment made up of a single leaf node containing the \0 header. LeafReader assumed that leaf nodes always contained at least one term, so assertions would fail.    While it would be possible to support reading and merging empty segments, there's no reason to do so. While this change could have been done in writeZeroSegment(), I put it in leafWriterFlush() so that it would work right if segmentMerge() created an empty segment, which could happen with future changes to how deleted documents are handled. (CVS 3484) file: [8f5e5fcc] check-in: [fed79bee] user: shess branch: trunk, size: 143422
Convert fts2 to store data in a way which allows for much faster updates. Groups of documents form segments which are encoded in a btree layered over a table of blocks, with various tricks to make merges fast. This performs 20x-25x faster than fts1 when loading the Enron corpus, and is only slightly slower for queries. (CVS 3474) file: [ddfca6ae] check-in: [85272b2f] user: shess branch: trunk, size: 143308
Fix leaky symbols. With this change, fts1 and fts2 can both be statically linked. (CVS 3472) file: [ba2b9e96] check-in: [5e8bbb85] user: shess branch: trunk, size: 98613
Copy fts1/ to fts2/, changing reference from fts1 to fts2. For future reference, the source versions copied were:    README.txt r1.1 fts1.c r1.37 fts1.h r1.2 fts1_hash.c r1.1 fts1_hash.h r1.1 fts1_porter.c r1.1 fts1_tokenizer.h r1.4 fts1_tokenizer1.c r1.6 (CVS 3471) file: [1052f034] check-in: [d0d1e7cd] user: shess branch: trunk, size: 98536 Added