Many hyperlinks are disabled.
Use anonymous login
to enable hyperlinks.
Overview
Comment: | Merge the latest trunk changes into the sessions branch. |
---|---|
Downloads: | Tarball | ZIP archive |
Timelines: | family | ancestors | descendants | both | sessions |
Files: | files | file ages | folders |
SHA1: |
4c5e276c902e0b93cfc05bf2e1db966e |
User & Date: | drh 2011-06-20 10:44:10.301 |
Context
2011-06-20
| ||
11:17 | Merge trunk changes with sessions branch. (check-in: 699b884383 user: dan tags: sessions) | |
10:44 | Merge the latest trunk changes into the sessions branch. (check-in: 4c5e276c90 user: drh tags: sessions) | |
2011-06-19
| ||
21:17 | Do not run test tkt-2d1a5c67d.test in the inmemory_journal permutation since that test requires WAL mode which does not work with inmemory_journal. (check-in: 228c43c726 user: drh tags: trunk) | |
2011-05-30
| ||
13:39 | Merge the latest trunk changes into the sessions branch. (check-in: 832886b1e6 user: drh tags: sessions) | |
Changes
Changes to Makefile.in.
︙ | ︙ | |||
375 376 377 378 379 380 381 | $(TOP)/src/test_syscall.c \ $(TOP)/src/test_stat.c \ $(TOP)/src/test_tclvar.c \ $(TOP)/src/test_thread.c \ $(TOP)/src/test_vfs.c \ $(TOP)/src/test_wholenumber.c \ $(TOP)/src/test_wsd.c \ | | > | 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 | $(TOP)/src/test_syscall.c \ $(TOP)/src/test_stat.c \ $(TOP)/src/test_tclvar.c \ $(TOP)/src/test_thread.c \ $(TOP)/src/test_vfs.c \ $(TOP)/src/test_wholenumber.c \ $(TOP)/src/test_wsd.c \ $(TOP)/ext/fts3/fts3_term.c \ $(TOP)/ext/fts3/fts3_test.c # Source code to the library files needed by the test fixture # TESTSRC2 = \ $(TOP)/src/attach.c \ $(TOP)/src/backup.c \ $(TOP)/src/bitvec.c \ |
︙ | ︙ |
Added Makefile.msc.
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 | # # nmake Makefile for SQLite # # The toplevel directory of the source tree. This is the directory # that contains this "Makefile.msc". # TOP = . # Set this non-0 to create and use the SQLite amalgamation file. # USE_AMALGAMATION = 1 # Version numbers and release number for the SQLite being compiled. # VERSION = 3.7 VERSION_NUMBER = 3007007 RELEASE = 3.7.7 # C Compiler and options for use in building executables that # will run on the platform that is doing the build. # BCC = cl.exe -O2 # C Compile and options for use in building executables that # will run on the target platform. (BCC and TCC are usually the # same unless your are cross-compiling.) # TCC = cl.exe -W3 -O2 -DSQLITE_OS_WIN=1 -I. -I$(TOP)\src # The mksqlite3c.tcl and mksqlite3h.tcl scripts will pull in # any extension header files by default. For non-amalgamation # builds, we need to make sure the compiler can find these. # !IF $(USE_AMALGAMATION)==0 TCC = $(TCC) -I$(TOP)\ext\fts3 TCC = $(TCC) -I$(TOP)\ext\rtree !ENDIF # Define -DNDEBUG to compile without debugging (i.e., for production usage) # Omitting the define will cause extra debugging code to be inserted and # includes extra comments when "EXPLAIN stmt" is used. # TCC = $(TCC) -DNDEBUG # The library that programs using TCL must link against. # LIBTCL = tcl85.lib TCLINCDIR = c:\tcl\include TCLLIBDIR = c:\tcl\lib # This is the command to use for tclsh - normally just "tclsh", but we may # know the specific version we want to use # TCLSH_CMD = tclsh85 # Compiler options needed for programs that use the readline() library. # READLINE_FLAGS = -DHAVE_READLINE=0 # The library that programs using readline() must link against. # LIBREADLINE = # Should the database engine be compiled threadsafe # TCC = $(TCC) -DSQLITE_THREADSAFE=1 # Do threads override each others locks by default (1), or do we test (-1) # TCC = $(TCC) -DSQLITE_THREAD_OVERRIDE_LOCK=-1 # Any target libraries which libsqlite must be linked against # TLIBS = # Flags controlling use of the in memory btree implementation # # SQLITE_TEMP_STORE is 0 to force temporary tables to be in a file, 1 to # default to file, 2 to default to memory, and 3 to force temporary # tables to always be in memory. # TCC = $(TCC) -DSQLITE_TEMP_STORE=1 # Enable/disable loadable extensions, and other optional features # based on configuration. (-DSQLITE_OMIT*, -DSQLITE_ENABLE*). # The same set of OMIT and ENABLE flags should be passed to the # LEMON parser generator and the mkkeywordhash tool as well. # BEGIN standard options OPT_FEATURE_FLAGS = $(OPT_FEATURE_FLAGS) -DSQLITE_ENABLE_FTS3=1 OPT_FEATURE_FLAGS = $(OPT_FEATURE_FLAGS) -DSQLITE_ENABLE_RTREE=1 OPT_FEATURE_FLAGS = $(OPT_FEATURE_FLAGS) -DSQLITE_ENABLE_COLUMN_METADATA=1 # END standard options # BEGIN required Windows option OPT_FEATURE_FLAGS = $(OPT_FEATURE_FLAGS) -DSQLITE_MAX_TRIGGER_DEPTH=100 # END required Windows option TCC = $(TCC) $(OPT_FEATURE_FLAGS) # Add in any optional parameters specified on the make commane line # ie. make "OPTS=-DSQLITE_ENABLE_FOO=1 -DSQLITE_OMIT_FOO=1". TCC = $(TCC) $(OPTS) # libtool compile/link LTCOMPILE = $(TCC) -Fo$@ LTLINK = $(TCC) -Fe$@ LTLIB = lib.exe # nawk compatible awk. NAWK = .\gawk.exe # You should not have to change anything below this line ############################################################################### # Object files for the SQLite library (non-amalgamation). # LIBOBJS0 = alter.lo analyze.lo attach.lo auth.lo \ backup.lo bitvec.lo btmutex.lo btree.lo build.lo \ callback.lo complete.lo ctime.lo date.lo delete.lo \ expr.lo fault.lo fkey.lo \ fts3.lo fts3_aux.lo fts3_expr.lo fts3_hash.lo fts3_icu.lo fts3_porter.lo \ fts3_snippet.lo fts3_tokenizer.lo fts3_tokenizer1.lo fts3_write.lo \ func.lo global.lo hash.lo \ icu.lo insert.lo journal.lo legacy.lo loadext.lo \ main.lo malloc.lo mem0.lo mem1.lo mem2.lo mem3.lo mem5.lo \ memjournal.lo \ mutex.lo mutex_noop.lo mutex_os2.lo mutex_unix.lo mutex_w32.lo \ notify.lo opcodes.lo os.lo os_os2.lo os_unix.lo os_win.lo \ pager.lo parse.lo pcache.lo pcache1.lo pragma.lo prepare.lo printf.lo \ random.lo resolve.lo rowset.lo rtree.lo select.lo status.lo \ table.lo tokenize.lo trigger.lo \ update.lo util.lo vacuum.lo \ vdbe.lo vdbeapi.lo vdbeaux.lo vdbeblob.lo vdbemem.lo vdbetrace.lo \ wal.lo walker.lo where.lo utf.lo vtab.lo # Object files for the amalgamation. # LIBOBJS1 = sqlite3.lo # Determine the real value of LIBOBJ based on the 'configure' script # !IF $(USE_AMALGAMATION)==0 LIBOBJ = $(LIBOBJS0) !ELSE LIBOBJ = $(LIBOBJS1) !ENDIF # All of the source code files. # SRC = \ $(TOP)\src\alter.c \ $(TOP)\src\analyze.c \ $(TOP)\src\attach.c \ $(TOP)\src\auth.c \ $(TOP)\src\backup.c \ $(TOP)\src\bitvec.c \ $(TOP)\src\btmutex.c \ $(TOP)\src\btree.c \ $(TOP)\src\btree.h \ $(TOP)\src\btreeInt.h \ $(TOP)\src\build.c \ $(TOP)\src\callback.c \ $(TOP)\src\complete.c \ $(TOP)\src\ctime.c \ $(TOP)\src\date.c \ $(TOP)\src\delete.c \ $(TOP)\src\expr.c \ $(TOP)\src\fault.c \ $(TOP)\src\fkey.c \ $(TOP)\src\func.c \ $(TOP)\src\global.c \ $(TOP)\src\hash.c \ $(TOP)\src\hash.h \ $(TOP)\src\hwtime.h \ $(TOP)\src\insert.c \ $(TOP)\src\journal.c \ $(TOP)\src\legacy.c \ $(TOP)\src\loadext.c \ $(TOP)\src\main.c \ $(TOP)\src\malloc.c \ $(TOP)\src\mem0.c \ $(TOP)\src\mem1.c \ $(TOP)\src\mem2.c \ $(TOP)\src\mem3.c \ $(TOP)\src\mem5.c \ $(TOP)\src\memjournal.c \ $(TOP)\src\mutex.c \ $(TOP)\src\mutex.h \ $(TOP)\src\mutex_noop.c \ $(TOP)\src\mutex_os2.c \ $(TOP)\src\mutex_unix.c \ $(TOP)\src\mutex_w32.c \ $(TOP)\src\notify.c \ $(TOP)\src\os.c \ $(TOP)\src\os.h \ $(TOP)\src\os_common.h \ $(TOP)\src\os_os2.c \ $(TOP)\src\os_unix.c \ $(TOP)\src\os_win.c \ $(TOP)\src\pager.c \ $(TOP)\src\pager.h \ $(TOP)\src\parse.y \ $(TOP)\src\pcache.c \ $(TOP)\src\pcache.h \ $(TOP)\src\pcache1.c \ $(TOP)\src\pragma.c \ $(TOP)\src\prepare.c \ $(TOP)\src\printf.c \ $(TOP)\src\random.c \ $(TOP)\src\resolve.c \ $(TOP)\src\rowset.c \ $(TOP)\src\select.c \ $(TOP)\src\status.c \ $(TOP)\src\shell.c \ $(TOP)\src\sqlite.h.in \ $(TOP)\src\sqlite3ext.h \ $(TOP)\src\sqliteInt.h \ $(TOP)\src\sqliteLimit.h \ $(TOP)\src\table.c \ $(TOP)\src\tclsqlite.c \ $(TOP)\src\tokenize.c \ $(TOP)\src\trigger.c \ $(TOP)\src\utf.c \ $(TOP)\src\update.c \ $(TOP)\src\util.c \ $(TOP)\src\vacuum.c \ $(TOP)\src\vdbe.c \ $(TOP)\src\vdbe.h \ $(TOP)\src\vdbeapi.c \ $(TOP)\src\vdbeaux.c \ $(TOP)\src\vdbeblob.c \ $(TOP)\src\vdbemem.c \ $(TOP)\src\vdbetrace.c \ $(TOP)\src\vdbeInt.h \ $(TOP)\src\vtab.c \ $(TOP)\src\wal.c \ $(TOP)\src\wal.h \ $(TOP)\src\walker.c \ $(TOP)\src\where.c # Source code for extensions # SRC = $(SRC) \ $(TOP)\ext\fts1\fts1.c \ $(TOP)\ext\fts1\fts1.h \ $(TOP)\ext\fts1\fts1_hash.c \ $(TOP)\ext\fts1\fts1_hash.h \ $(TOP)\ext\fts1\fts1_porter.c \ $(TOP)\ext\fts1\fts1_tokenizer.h \ $(TOP)\ext\fts1\fts1_tokenizer1.c SRC = $(SRC) \ $(TOP)\ext\fts2\fts2.c \ $(TOP)\ext\fts2\fts2.h \ $(TOP)\ext\fts2\fts2_hash.c \ $(TOP)\ext\fts2\fts2_hash.h \ $(TOP)\ext\fts2\fts2_icu.c \ $(TOP)\ext\fts2\fts2_porter.c \ $(TOP)\ext\fts2\fts2_tokenizer.h \ $(TOP)\ext\fts2\fts2_tokenizer.c \ $(TOP)\ext\fts2\fts2_tokenizer1.c SRC = $(SRC) \ $(TOP)\ext\fts3\fts3.c \ $(TOP)\ext\fts3\fts3.h \ $(TOP)\ext\fts3\fts3Int.h \ $(TOP)\ext\fts3\fts3_aux.c \ $(TOP)\ext\fts3\fts3_expr.c \ $(TOP)\ext\fts3\fts3_hash.c \ $(TOP)\ext\fts3\fts3_hash.h \ $(TOP)\ext\fts3\fts3_icu.c \ $(TOP)\ext\fts3\fts3_porter.c \ $(TOP)\ext\fts3\fts3_snippet.c \ $(TOP)\ext\fts3\fts3_tokenizer.h \ $(TOP)\ext\fts3\fts3_tokenizer.c \ $(TOP)\ext\fts3\fts3_tokenizer1.c \ $(TOP)\ext\fts3\fts3_write.c SRC = $(SRC) \ $(TOP)\ext\icu\sqliteicu.h \ $(TOP)\ext\icu\icu.c SRC = $(SRC) \ $(TOP)\ext\rtree\rtree.h \ $(TOP)\ext\rtree\rtree.c # Generated source code files # SRC = $(SRC) \ keywordhash.h \ opcodes.c \ opcodes.h \ parse.c \ parse.h \ sqlite3.h # Source code to the test files. # TESTSRC = \ $(TOP)\src\test1.c \ $(TOP)\src\test2.c \ $(TOP)\src\test3.c \ $(TOP)\src\test4.c \ $(TOP)\src\test5.c \ $(TOP)\src\test6.c \ $(TOP)\src\test7.c \ $(TOP)\src\test8.c \ $(TOP)\src\test9.c \ $(TOP)\src\test_autoext.c \ $(TOP)\src\test_async.c \ $(TOP)\src\test_backup.c \ $(TOP)\src\test_btree.c \ $(TOP)\src\test_config.c \ $(TOP)\src\test_demovfs.c \ $(TOP)\src\test_devsym.c \ $(TOP)\src\test_func.c \ $(TOP)\src\test_fuzzer.c \ $(TOP)\src\test_hexio.c \ $(TOP)\src\test_init.c \ $(TOP)\src\test_intarray.c \ $(TOP)\src\test_journal.c \ $(TOP)\src\test_malloc.c \ $(TOP)\src\test_multiplex.c \ $(TOP)\src\test_mutex.c \ $(TOP)\src\test_onefile.c \ $(TOP)\src\test_osinst.c \ $(TOP)\src\test_pcache.c \ $(TOP)\src\test_quota.c \ $(TOP)\src\test_rtree.c \ $(TOP)\src\test_schema.c \ $(TOP)\src\test_server.c \ $(TOP)\src\test_superlock.c \ $(TOP)\src\test_syscall.c \ $(TOP)\src\test_stat.c \ $(TOP)\src\test_tclvar.c \ $(TOP)\src\test_thread.c \ $(TOP)\src\test_vfs.c \ $(TOP)\src\test_wholenumber.c \ $(TOP)\src\test_wsd.c \ $(TOP)\ext\fts3\fts3_term.c \ $(TOP)\ext\fts3\fts3_test.c # Source code to the library files needed by the test fixture # TESTSRC2 = \ $(TOP)\src\attach.c \ $(TOP)\src\backup.c \ $(TOP)\src\bitvec.c \ $(TOP)\src\btree.c \ $(TOP)\src\build.c \ $(TOP)\src\ctime.c \ $(TOP)\src\date.c \ $(TOP)\src\expr.c \ $(TOP)\src\func.c \ $(TOP)\src\insert.c \ $(TOP)\src\wal.c \ $(TOP)\src\mem5.c \ $(TOP)\src\os.c \ $(TOP)\src\os_os2.c \ $(TOP)\src\os_unix.c \ $(TOP)\src\os_win.c \ $(TOP)\src\pager.c \ $(TOP)\src\pragma.c \ $(TOP)\src\prepare.c \ $(TOP)\src\printf.c \ $(TOP)\src\random.c \ $(TOP)\src\pcache.c \ $(TOP)\src\pcache1.c \ $(TOP)\src\select.c \ $(TOP)\src\tokenize.c \ $(TOP)\src\utf.c \ $(TOP)\src\util.c \ $(TOP)\src\vdbeapi.c \ $(TOP)\src\vdbeaux.c \ $(TOP)\src\vdbe.c \ $(TOP)\src\vdbemem.c \ $(TOP)\src\vdbetrace.c \ $(TOP)\src\where.c \ parse.c \ $(TOP)\ext\fts3\fts3.c \ $(TOP)\ext\fts3\fts3_aux.c \ $(TOP)\ext\fts3\fts3_expr.c \ $(TOP)\ext\fts3\fts3_tokenizer.c \ $(TOP)\ext\fts3\fts3_write.c \ $(TOP)\ext\async\sqlite3async.c # Header files used by all library source files. # HDR = \ $(TOP)\src\btree.h \ $(TOP)\src\btreeInt.h \ $(TOP)\src\hash.h \ $(TOP)\src\hwtime.h \ keywordhash.h \ $(TOP)\src\mutex.h \ opcodes.h \ $(TOP)\src\os.h \ $(TOP)\src\os_common.h \ $(TOP)\src\pager.h \ $(TOP)\src\pcache.h \ parse.h \ sqlite3.h \ $(TOP)\src\sqlite3ext.h \ $(TOP)\src\sqliteInt.h \ $(TOP)\src\sqliteLimit.h \ $(TOP)\src\vdbe.h \ $(TOP)\src\vdbeInt.h # Header files used by extensions # EXTHDR = $(EXTHDR) \ $(TOP)\ext\fts1\fts1.h \ $(TOP)\ext\fts1\fts1_hash.h \ $(TOP)\ext\fts1\fts1_tokenizer.h EXTHDR = $(EXTHDR) \ $(TOP)\ext\fts2\fts2.h \ $(TOP)\ext\fts2\fts2_hash.h \ $(TOP)\ext\fts2\fts2_tokenizer.h EXTHDR = $(EXTHDR) \ $(TOP)\ext\fts3\fts3.h \ $(TOP)\ext\fts3\fts3Int.h \ $(TOP)\ext\fts3\fts3_hash.h \ $(TOP)\ext\fts3\fts3_tokenizer.h EXTHDR = $(EXTHDR) \ $(TOP)\ext\rtree\rtree.h EXTHDR = $(EXTHDR) \ $(TOP)\ext\icu\sqliteicu.h EXTHDR = $(EXTHDR) \ $(TOP)\ext\rtree\sqlite3rtree.h # This is the default Makefile target. The objects listed here # are what get build when you type just "make" with no arguments. # all: libsqlite3.lib sqlite3.exe libtclsqlite3.lib libsqlite3.lib: $(LIBOBJ) $(LTLIB) -OUT:$@ $(LIBOBJ) $(TLIBS) libtclsqlite3.lib: tclsqlite.lo libsqlite3.lib $(LTLIB) /LIBPATH:$(TCLLIBDIR) -OUT:$@ tclsqlite.lo libsqlite3.lib $(LIBTCL:tcl=tclstub) $(TLIBS) sqlite3.exe: $(TOP)\src\shell.c libsqlite3.lib sqlite3.h $(LTLINK) $(READLINE_FLAGS) \ $(TOP)\src\shell.c libsqlite3.lib \ $(LIBREADLINE) $(TLIBS) # This target creates a directory named "tsrc" and fills it with # copies of all of the C source code and header files needed to # build on the target system. Some of the C source code and header # files are automatically generated. This target takes care of # all that automatic generation. # .target_source: $(SRC) $(TOP)\tool\vdbe-compress.tcl -rmdir /S/Q tsrc -mkdir tsrc for %i in ($(SRC)) do copy /Y %i tsrc del /Q tsrc\sqlite.h.in tsrc\parse.y $(TCLSH_CMD) $(TOP)\tool\vdbe-compress.tcl <tsrc\vdbe.c >vdbe.new move vdbe.new tsrc\vdbe.c echo > .target_source sqlite3.c: .target_source $(TOP)\tool\mksqlite3c.tcl $(TCLSH_CMD) $(TOP)\tool\mksqlite3c.tcl # Rule to build the amalgamation # sqlite3.lo: sqlite3.c $(LTCOMPILE) -c sqlite3.c # Rules to build the LEMON compiler generator # lempar.c: $(TOP)\src\lempar.c copy $(TOP)\src\lempar.c . lemon.exe: $(TOP)\tool\lemon.c lempar.c $(BCC) -Fe$@ $(TOP)\tool\lemon.c # Rules to build individual *.lo files from generated *.c files. This # applies to: # # parse.lo # opcodes.lo # parse.lo: parse.c $(HDR) $(LTCOMPILE) -c parse.c opcodes.lo: opcodes.c $(LTCOMPILE) -c opcodes.c # Rules to build individual *.lo files from files in the src directory. # alter.lo: $(TOP)\src\alter.c $(HDR) $(LTCOMPILE) -c $(TOP)\src\alter.c analyze.lo: $(TOP)\src\analyze.c $(HDR) $(LTCOMPILE) -c $(TOP)\src\analyze.c attach.lo: $(TOP)\src\attach.c $(HDR) $(LTCOMPILE) -c $(TOP)\src\attach.c auth.lo: $(TOP)\src\auth.c $(HDR) $(LTCOMPILE) -c $(TOP)\src\auth.c backup.lo: $(TOP)\src\backup.c $(HDR) $(LTCOMPILE) -c $(TOP)\src\backup.c bitvec.lo: $(TOP)\src\bitvec.c $(HDR) $(LTCOMPILE) -c $(TOP)\src\bitvec.c btmutex.lo: $(TOP)\src\btmutex.c $(HDR) $(LTCOMPILE) -c $(TOP)\src\btmutex.c btree.lo: $(TOP)\src\btree.c $(HDR) $(TOP)\src\pager.h $(LTCOMPILE) -c $(TOP)\src\btree.c build.lo: $(TOP)\src\build.c $(HDR) $(LTCOMPILE) -c $(TOP)\src\build.c callback.lo: $(TOP)\src\callback.c $(HDR) $(LTCOMPILE) -c $(TOP)\src\callback.c complete.lo: $(TOP)\src\complete.c $(HDR) $(LTCOMPILE) -c $(TOP)\src\complete.c ctime.lo: $(TOP)\src\ctime.c $(HDR) $(LTCOMPILE) -c $(TOP)\src\ctime.c date.lo: $(TOP)\src\date.c $(HDR) $(LTCOMPILE) -c $(TOP)\src\date.c delete.lo: $(TOP)\src\delete.c $(HDR) $(LTCOMPILE) -c $(TOP)\src\delete.c expr.lo: $(TOP)\src\expr.c $(HDR) $(LTCOMPILE) -c $(TOP)\src\expr.c fault.lo: $(TOP)\src\fault.c $(HDR) $(LTCOMPILE) -c $(TOP)\src\fault.c fkey.lo: $(TOP)\src\fkey.c $(HDR) $(LTCOMPILE) -c $(TOP)\src\fkey.c func.lo: $(TOP)\src\func.c $(HDR) $(LTCOMPILE) -c $(TOP)\src\func.c global.lo: $(TOP)\src\global.c $(HDR) $(LTCOMPILE) -c $(TOP)\src\global.c hash.lo: $(TOP)\src\hash.c $(HDR) $(LTCOMPILE) -c $(TOP)\src\hash.c insert.lo: $(TOP)\src\insert.c $(HDR) $(LTCOMPILE) -c $(TOP)\src\insert.c journal.lo: $(TOP)\src\journal.c $(HDR) $(LTCOMPILE) -c $(TOP)\src\journal.c legacy.lo: $(TOP)\src\legacy.c $(HDR) $(LTCOMPILE) -c $(TOP)\src\legacy.c loadext.lo: $(TOP)\src\loadext.c $(HDR) $(LTCOMPILE) -c $(TOP)\src\loadext.c main.lo: $(TOP)\src\main.c $(HDR) $(LTCOMPILE) -c $(TOP)\src\main.c malloc.lo: $(TOP)\src\malloc.c $(HDR) $(LTCOMPILE) -c $(TOP)\src\malloc.c mem0.lo: $(TOP)\src\mem0.c $(HDR) $(LTCOMPILE) -c $(TOP)\src\mem0.c mem1.lo: $(TOP)\src\mem1.c $(HDR) $(LTCOMPILE) -c $(TOP)\src\mem1.c mem2.lo: $(TOP)\src\mem2.c $(HDR) $(LTCOMPILE) -c $(TOP)\src\mem2.c mem3.lo: $(TOP)\src\mem3.c $(HDR) $(LTCOMPILE) -c $(TOP)\src\mem3.c mem5.lo: $(TOP)\src\mem5.c $(HDR) $(LTCOMPILE) -c $(TOP)\src\mem5.c memjournal.lo: $(TOP)\src\memjournal.c $(HDR) $(LTCOMPILE) -c $(TOP)\src\memjournal.c mutex.lo: $(TOP)\src\mutex.c $(HDR) $(LTCOMPILE) -c $(TOP)\src\mutex.c mutex_noop.lo: $(TOP)\src\mutex_noop.c $(HDR) $(LTCOMPILE) -c $(TOP)\src\mutex_noop.c mutex_os2.lo: $(TOP)\src\mutex_os2.c $(HDR) $(LTCOMPILE) -c $(TOP)\src\mutex_os2.c mutex_unix.lo: $(TOP)\src\mutex_unix.c $(HDR) $(LTCOMPILE) -c $(TOP)\src\mutex_unix.c mutex_w32.lo: $(TOP)\src\mutex_w32.c $(HDR) $(LTCOMPILE) -c $(TOP)\src\mutex_w32.c notify.lo: $(TOP)\src\notify.c $(HDR) $(LTCOMPILE) -c $(TOP)\src\notify.c pager.lo: $(TOP)\src\pager.c $(HDR) $(TOP)\src\pager.h $(LTCOMPILE) -c $(TOP)\src\pager.c pcache.lo: $(TOP)\src\pcache.c $(HDR) $(TOP)\src\pcache.h $(LTCOMPILE) -c $(TOP)\src\pcache.c pcache1.lo: $(TOP)\src\pcache1.c $(HDR) $(TOP)\src\pcache.h $(LTCOMPILE) -c $(TOP)\src\pcache1.c os.lo: $(TOP)\src\os.c $(HDR) $(LTCOMPILE) -c $(TOP)\src\os.c os_unix.lo: $(TOP)\src\os_unix.c $(HDR) $(LTCOMPILE) -c $(TOP)\src\os_unix.c os_win.lo: $(TOP)\src\os_win.c $(HDR) $(LTCOMPILE) -c $(TOP)\src\os_win.c os_os2.lo: $(TOP)\src\os_os2.c $(HDR) $(LTCOMPILE) -c $(TOP)\src\os_os2.c pragma.lo: $(TOP)\src\pragma.c $(HDR) $(LTCOMPILE) -c $(TOP)\src\pragma.c prepare.lo: $(TOP)\src\prepare.c $(HDR) $(LTCOMPILE) -c $(TOP)\src\prepare.c printf.lo: $(TOP)\src\printf.c $(HDR) $(LTCOMPILE) -c $(TOP)\src\printf.c random.lo: $(TOP)\src\random.c $(HDR) $(LTCOMPILE) -c $(TOP)\src\random.c resolve.lo: $(TOP)\src\resolve.c $(HDR) $(LTCOMPILE) -c $(TOP)\src\resolve.c rowset.lo: $(TOP)\src\rowset.c $(HDR) $(LTCOMPILE) -c $(TOP)\src\rowset.c select.lo: $(TOP)\src\select.c $(HDR) $(LTCOMPILE) -c $(TOP)\src\select.c status.lo: $(TOP)\src\status.c $(HDR) $(LTCOMPILE) -c $(TOP)\src\status.c table.lo: $(TOP)\src\table.c $(HDR) $(LTCOMPILE) -c $(TOP)\src\table.c tokenize.lo: $(TOP)\src\tokenize.c keywordhash.h $(HDR) $(LTCOMPILE) -c $(TOP)\src\tokenize.c trigger.lo: $(TOP)\src\trigger.c $(HDR) $(LTCOMPILE) -c $(TOP)\src\trigger.c update.lo: $(TOP)\src\update.c $(HDR) $(LTCOMPILE) -c $(TOP)\src\update.c utf.lo: $(TOP)\src\utf.c $(HDR) $(LTCOMPILE) -c $(TOP)\src\utf.c util.lo: $(TOP)\src\util.c $(HDR) $(LTCOMPILE) -c $(TOP)\src\util.c vacuum.lo: $(TOP)\src\vacuum.c $(HDR) $(LTCOMPILE) -c $(TOP)\src\vacuum.c vdbe.lo: $(TOP)\src\vdbe.c $(HDR) $(LTCOMPILE) -c $(TOP)\src\vdbe.c vdbeapi.lo: $(TOP)\src\vdbeapi.c $(HDR) $(LTCOMPILE) -c $(TOP)\src\vdbeapi.c vdbeaux.lo: $(TOP)\src\vdbeaux.c $(HDR) $(LTCOMPILE) -c $(TOP)\src\vdbeaux.c vdbeblob.lo: $(TOP)\src\vdbeblob.c $(HDR) $(LTCOMPILE) -c $(TOP)\src\vdbeblob.c vdbemem.lo: $(TOP)\src\vdbemem.c $(HDR) $(LTCOMPILE) -c $(TOP)\src\vdbemem.c vdbetrace.lo: $(TOP)\src\vdbetrace.c $(HDR) $(LTCOMPILE) -c $(TOP)\src\vdbetrace.c vtab.lo: $(TOP)\src\vtab.c $(HDR) $(LTCOMPILE) -c $(TOP)\src\vtab.c wal.lo: $(TOP)\src\wal.c $(HDR) $(LTCOMPILE) -c $(TOP)\src\wal.c walker.lo: $(TOP)\src\walker.c $(HDR) $(LTCOMPILE) -c $(TOP)\src\walker.c where.lo: $(TOP)\src\where.c $(HDR) $(LTCOMPILE) -c $(TOP)\src\where.c tclsqlite.lo: $(TOP)\src\tclsqlite.c $(HDR) $(LTCOMPILE) -DUSE_TCL_STUBS=1 -DBUILD_sqlite -I$(TCLINCDIR) -c $(TOP)\src\tclsqlite.c tclsqlite-shell.lo: $(TOP)\src\tclsqlite.c $(HDR) $(LTCOMPILE) -DTCLSH=1 -DBUILD_sqlite -I$(TCLINCDIR) -c $(TOP)\src\tclsqlite.c tclsqlite3.exe: tclsqlite-shell.lo libsqlite3.lib $(LTLINK) tclsqlite-shell.lo \ /link /LIBPATH:$(TCLLIBDIR) libsqlite3.lib $(LIBTCL) # Rules to build opcodes.c and opcodes.h # opcodes.c: opcodes.h $(TOP)\mkopcodec.awk $(NAWK) "/#define OP_/ { print }" opcodes.h | sort /+45 | $(NAWK) -f $(TOP)\mkopcodec.awk >opcodes.c opcodes.h: parse.h $(TOP)\src\vdbe.c $(TOP)\mkopcodeh.awk type parse.h $(TOP)\src\vdbe.c | $(NAWK) -f $(TOP)\mkopcodeh.awk >opcodes.h # Rules to build parse.c and parse.h - the outputs of lemon. # parse.h: parse.c parse.c: $(TOP)\src\parse.y lemon.exe $(TOP)\addopcodes.awk del /Q parse.y parse.h parse.h.temp copy $(TOP)\src\parse.y . .\lemon.exe $(OPT_FEATURE_FLAGS) $(OPTS) parse.y move parse.h parse.h.temp $(NAWK) -f $(TOP)\addopcodes.awk parse.h.temp >parse.h sqlite3.h: $(TOP)\src\sqlite.h.in $(TOP)\manifest.uuid $(TOP)\VERSION $(TCLSH_CMD) $(TOP)\tool\mksqlite3h.tcl $(TOP) >sqlite3.h mkkeywordhash.exe: $(TOP)\tool\mkkeywordhash.c $(BCC) -Femkkeywordhash.exe $(OPT_FEATURE_FLAGS) $(OPTS) $(TOP)\tool\mkkeywordhash.c keywordhash.h: $(TOP)\tool\mkkeywordhash.c mkkeywordhash.exe .\mkkeywordhash.exe >keywordhash.h # Rules to build the extension objects. # icu.lo: $(TOP)\ext\icu\icu.c $(HDR) $(EXTHDR) $(LTCOMPILE) -DSQLITE_CORE -c $(TOP)\ext\icu\icu.c fts2.lo: $(TOP)\ext\fts2\fts2.c $(HDR) $(EXTHDR) $(LTCOMPILE) -DSQLITE_CORE -c $(TOP)\ext\fts2\fts2.c fts2_hash.lo: $(TOP)\ext\fts2\fts2_hash.c $(HDR) $(EXTHDR) $(LTCOMPILE) -DSQLITE_CORE -c $(TOP)\ext\fts2\fts2_hash.c fts2_icu.lo: $(TOP)\ext\fts2\fts2_icu.c $(HDR) $(EXTHDR) $(LTCOMPILE) -DSQLITE_CORE -c $(TOP)\ext\fts2\fts2_icu.c fts2_porter.lo: $(TOP)\ext\fts2\fts2_porter.c $(HDR) $(EXTHDR) $(LTCOMPILE) -DSQLITE_CORE -c $(TOP)\ext\fts2\fts2_porter.c fts2_tokenizer.lo: $(TOP)\ext\fts2\fts2_tokenizer.c $(HDR) $(EXTHDR) $(LTCOMPILE) -DSQLITE_CORE -c $(TOP)\ext\fts2\fts2_tokenizer.c fts2_tokenizer1.lo: $(TOP)\ext\fts2\fts2_tokenizer1.c $(HDR) $(EXTHDR) $(LTCOMPILE) -DSQLITE_CORE -c $(TOP)\ext\fts2\fts2_tokenizer1.c fts3.lo: $(TOP)\ext\fts3\fts3.c $(HDR) $(EXTHDR) $(LTCOMPILE) -DSQLITE_CORE -c $(TOP)\ext\fts3\fts3.c fts3_aux.lo: $(TOP)\ext\fts3\fts3_aux.c $(HDR) $(EXTHDR) $(LTCOMPILE) -DSQLITE_CORE -c $(TOP)\ext\fts3\fts3_aux.c fts3_expr.lo: $(TOP)\ext\fts3\fts3_expr.c $(HDR) $(EXTHDR) $(LTCOMPILE) -DSQLITE_CORE -c $(TOP)\ext\fts3\fts3_expr.c fts3_hash.lo: $(TOP)\ext\fts3\fts3_hash.c $(HDR) $(EXTHDR) $(LTCOMPILE) -DSQLITE_CORE -c $(TOP)\ext\fts3\fts3_hash.c fts3_icu.lo: $(TOP)\ext\fts3\fts3_icu.c $(HDR) $(EXTHDR) $(LTCOMPILE) -DSQLITE_CORE -c $(TOP)\ext\fts3\fts3_icu.c fts3_snippet.lo: $(TOP)\ext\fts3\fts3_snippet.c $(HDR) $(EXTHDR) $(LTCOMPILE) -DSQLITE_CORE -c $(TOP)\ext\fts3\fts3_snippet.c fts3_porter.lo: $(TOP)\ext\fts3\fts3_porter.c $(HDR) $(EXTHDR) $(LTCOMPILE) -DSQLITE_CORE -c $(TOP)\ext\fts3\fts3_porter.c fts3_tokenizer.lo: $(TOP)\ext\fts3\fts3_tokenizer.c $(HDR) $(EXTHDR) $(LTCOMPILE) -DSQLITE_CORE -c $(TOP)\ext\fts3\fts3_tokenizer.c fts3_tokenizer1.lo: $(TOP)\ext\fts3\fts3_tokenizer1.c $(HDR) $(EXTHDR) $(LTCOMPILE) -DSQLITE_CORE -c $(TOP)\ext\fts3\fts3_tokenizer1.c fts3_write.lo: $(TOP)\ext\fts3\fts3_write.c $(HDR) $(EXTHDR) $(LTCOMPILE) -DSQLITE_CORE -c $(TOP)\ext\fts3\fts3_write.c rtree.lo: $(TOP)\ext\rtree\rtree.c $(HDR) $(EXTHDR) $(LTCOMPILE) -DSQLITE_CORE -c $(TOP)\ext\rtree\rtree.c # Rules to build the 'testfixture' application. # # If using the amalgamation, use sqlite3.c directly to build the test # fixture. Otherwise link against libsqlite3.lib. (This distinction is # necessary because the test fixture requires non-API symbols which are # hidden when the library is built via the amalgamation). # TESTFIXTURE_FLAGS = -DTCLSH=1 -DSQLITE_TEST=1 -DSQLITE_CRASH_TEST=1 TESTFIXTURE_FLAGS = $(TESTFIXTURE_FLAGS) -DSQLITE_SERVER=1 -DSQLITE_PRIVATE="" -DSQLITE_CORE TESTFIXTURE_SRC0 = $(TESTSRC2) libsqlite3.lib TESTFIXTURE_SRC1 = sqlite3.c !IF $(USE_AMALGAMATION)==0 TESTFIXTURE_SRC = $(TESTSRC) $(TOP)\src\tclsqlite.c $(TESTFIXTURE_SRC0) !ELSE TESTFIXTURE_SRC = $(TESTSRC) $(TOP)\src\tclsqlite.c $(TESTFIXTURE_SRC1) !ENDIF testfixture.exe: $(TESTFIXTURE_SRC) $(HDR) $(LTLINK) -DSQLITE_NO_SYNC=1 $(TESTFIXTURE_FLAGS) \ -DBUILD_sqlite -I$(TCLINCDIR) \ $(TESTFIXTURE_SRC) /link /LIBPATH:$(TCLLIBDIR) $(LIBTCL) $(TLIBS) fulltest: testfixture.exe sqlite3.exe .\testfixture.exe $(TOP)\test\all.test soaktest: testfixture.exe sqlite3.exe .\testfixture.exe $(TOP)\test\all.test -soak=1 test: testfixture.exe sqlite3.exe .\testfixture.exe $(TOP)\test\veryquick.test spaceanal_tcl.h: $(TOP)\tool\spaceanal.tcl $(NAWK) "/^[^#]/ { gsub(/\\/,\"\\\\\\\\\");gsub(/\\\"/,\"\\\\\\\"\");gsub(/^/,\"\\\"\");gsub(/$$/,\"\\n\\\"\");print }" \ $(TOP)\tool\spaceanal.tcl >spaceanal_tcl.h sqlite3_analyzer.exe: $(TESTFIXTURE_SRC) spaceanal_tcl.h $(LTLINK) -DTCLSH=2 -DSQLITE_TEST=1 -DSQLITE_CRASH_TEST=1 \ -DSQLITE_SERVER=1 -DSQLITE_PRIVATE="" -DSQLITE_CORE \ -DBUILD_sqlite -I$(TCLINCDIR) \ $(TESTFIXTURE_SRC) /link /LIBPATH:$(TCLLIBDIR) $(LIBTCL) $(TLIBS) clean: del /Q *.lo *.lib *.obj sqlite3.exe libsqlite3.lib del /Q sqlite3.h opcodes.c opcodes.h del /Q lemon.exe lempar.c parse.* del /Q mkkeywordhash.exe keywordhash.h -rmdir /Q/S tsrc del /Q .target_source del /Q testfixture.exe testfixture.exp test.db del /Q sqlite3.dll sqlite3.lib sqlite3.exp sqlite3.def del /Q sqlite3.c del /Q sqlite3_analyzer.exe sqlite3_analyzer.exp spaceanal_tcl.h # # Windows section # dll: sqlite3.dll sqlite3.def: libsqlite3.lib echo EXPORTS >sqlite3.def dumpbin /all libsqlite3.lib \ | $(NAWK) "/ 1 _sqlite3_/ { sub(/^.* _/,\"\");print }" \ | sort >>sqlite3.def sqlite3.dll: $(LIBOBJ) sqlite3.def link /DLL /OUT:$@ /DEF:sqlite3.def $(LIBOBJ) |
Changes to ext/fts3/fts3.c.
︙ | ︙ | |||
288 289 290 291 292 293 294 295 296 297 298 299 300 | ** older data. ** ** TODO(shess) Provide a VACUUM type operation to clear out all ** deletions and duplications. This would basically be a forced merge ** into a single segment. */ #if !defined(SQLITE_CORE) || defined(SQLITE_ENABLE_FTS3) #if defined(SQLITE_ENABLE_FTS3) && !defined(SQLITE_CORE) # define SQLITE_CORE 1 #endif | > < < | 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 | ** older data. ** ** TODO(shess) Provide a VACUUM type operation to clear out all ** deletions and duplications. This would basically be a forced merge ** into a single segment. */ #include "fts3Int.h" #if !defined(SQLITE_CORE) || defined(SQLITE_ENABLE_FTS3) #if defined(SQLITE_ENABLE_FTS3) && !defined(SQLITE_CORE) # define SQLITE_CORE 1 #endif #include <assert.h> #include <stdlib.h> #include <stddef.h> #include <stdio.h> #include <string.h> #include <stdarg.h> |
︙ | ︙ | |||
419 420 421 422 423 424 425 | *pVal += iVal; } /* ** When this function is called, *pp points to the first byte following a ** varint that is part of a doclist (or position-list, or any other list ** of varints). This function moves *pp to point to the start of that varint, | | | | < < < < < < < < < < < < < < | 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 | *pVal += iVal; } /* ** When this function is called, *pp points to the first byte following a ** varint that is part of a doclist (or position-list, or any other list ** of varints). This function moves *pp to point to the start of that varint, ** and sets *pVal by the varint value. ** ** Argument pStart points to the first byte of the doclist that the ** varint is part of. */ static void fts3GetReverseVarint( char **pp, char *pStart, sqlite3_int64 *pVal ){ sqlite3_int64 iVal; char *p = *pp; /* Pointer p now points at the first byte past the varint we are ** interested in. So, unless the doclist is corrupt, the 0x80 bit is ** clear on character p[-1]. */ for(p = (*pp)-2; p>=pStart && *p&0x80; p--); p++; *pp = p; sqlite3Fts3GetVarint(p, &iVal); *pVal = iVal; } /* ** The xDisconnect() virtual table method. */ static int fts3DisconnectMethod(sqlite3_vtab *pVtab){ Fts3Table *p = (Fts3Table *)pVtab; |
︙ | ︙ | |||
827 828 829 830 831 832 833 834 835 836 837 838 839 840 | fts3Appendf(pRc, &zRet, "?"); for(i=0; i<p->nColumn; i++){ fts3Appendf(pRc, &zRet, ",%s(?)", zFunction); } sqlite3_free(zFree); return zRet; } /* ** This function is the implementation of both the xConnect and xCreate ** methods of the FTS3 virtual table. ** ** The argv[] array contains the following: ** | > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > | 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 | fts3Appendf(pRc, &zRet, "?"); for(i=0; i<p->nColumn; i++){ fts3Appendf(pRc, &zRet, ",%s(?)", zFunction); } sqlite3_free(zFree); return zRet; } static int fts3GobbleInt(const char **pp, int *pnOut){ const char *p = *pp; int nInt = 0; for(p=*pp; p[0]>='0' && p[0]<='9'; p++){ nInt = nInt * 10 + (p[0] - '0'); } if( p==*pp ) return SQLITE_ERROR; *pnOut = nInt; *pp = p; return SQLITE_OK; } static int fts3PrefixParameter( const char *zParam, /* ABC in prefix=ABC parameter to parse */ int *pnIndex, /* OUT: size of *apIndex[] array */ struct Fts3Index **apIndex, /* OUT: Array of indexes for this table */ struct Fts3Index **apFree /* OUT: Free this with sqlite3_free() */ ){ struct Fts3Index *aIndex; int nIndex = 1; if( zParam && zParam[0] ){ const char *p; nIndex++; for(p=zParam; *p; p++){ if( *p==',' ) nIndex++; } } aIndex = sqlite3_malloc(sizeof(struct Fts3Index) * nIndex); *apIndex = *apFree = aIndex; *pnIndex = nIndex; if( !aIndex ){ return SQLITE_NOMEM; } memset(aIndex, 0, sizeof(struct Fts3Index) * nIndex); if( zParam ){ const char *p = zParam; int i; for(i=1; i<nIndex; i++){ int nPrefix; if( fts3GobbleInt(&p, &nPrefix) ) return SQLITE_ERROR; aIndex[i].nPrefix = nPrefix; p++; } } return SQLITE_OK; } /* ** This function is the implementation of both the xConnect and xCreate ** methods of the FTS3 virtual table. ** ** The argv[] array contains the following: ** |
︙ | ︙ | |||
860 861 862 863 864 865 866 | int iCol; /* Column index */ int nString = 0; /* Bytes required to hold all column names */ int nCol = 0; /* Number of columns in the FTS table */ char *zCsr; /* Space for holding column names */ int nDb; /* Bytes required to hold database name */ int nName; /* Bytes required to hold table name */ int isFts4 = (argv[0][3]=='4'); /* True for FTS4, false for FTS3 */ | < > > > > > > > > | | | 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 | int iCol; /* Column index */ int nString = 0; /* Bytes required to hold all column names */ int nCol = 0; /* Number of columns in the FTS table */ char *zCsr; /* Space for holding column names */ int nDb; /* Bytes required to hold database name */ int nName; /* Bytes required to hold table name */ int isFts4 = (argv[0][3]=='4'); /* True for FTS4, false for FTS3 */ const char **aCol; /* Array of column names */ sqlite3_tokenizer *pTokenizer = 0; /* Tokenizer for this table */ int nIndex; /* Size of aIndex[] array */ struct Fts3Index *aIndex; /* Array of indexes for this table */ struct Fts3Index *aFree = 0; /* Free this before returning */ /* The results of parsing supported FTS4 key=value options: */ int bNoDocsize = 0; /* True to omit %_docsize table */ int bDescIdx = 0; /* True to store descending indexes */ char *zPrefix = 0; /* Prefix parameter value (or NULL) */ char *zCompress = 0; /* compress=? parameter (or NULL) */ char *zUncompress = 0; /* uncompress=? parameter (or NULL) */ assert( strlen(argv[0])==4 ); assert( (sqlite3_strnicmp(argv[0], "fts4", 4)==0 && isFts4) || (sqlite3_strnicmp(argv[0], "fts3", 4)==0 && !isFts4) ); nDb = (int)strlen(argv[1]) + 1; |
︙ | ︙ | |||
906 907 908 909 910 911 912 913 914 | && 0==sqlite3Fts3IsIdChar(z[8]) ){ rc = sqlite3Fts3InitTokenizer(pHash, &z[9], &pTokenizer, pzErr); } /* Check if it is an FTS4 special argument. */ else if( isFts4 && fts3IsSpecialColumn(z, &nKey, &zVal) ){ if( !zVal ){ rc = SQLITE_NOMEM; | > > > > > > > > > > > > > | > > > > | > > | < | > > > | | | > > | > > > > > > > > | | > | > > | | > | > > > > | | | > > > > | > | 950 951 952 953 954 955 956 957 958 959 960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979 980 981 982 983 984 985 986 987 988 989 990 991 992 993 994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 1008 1009 1010 1011 1012 1013 1014 1015 1016 1017 1018 1019 1020 1021 1022 1023 1024 1025 1026 1027 1028 1029 | && 0==sqlite3Fts3IsIdChar(z[8]) ){ rc = sqlite3Fts3InitTokenizer(pHash, &z[9], &pTokenizer, pzErr); } /* Check if it is an FTS4 special argument. */ else if( isFts4 && fts3IsSpecialColumn(z, &nKey, &zVal) ){ struct Fts4Option { const char *zOpt; int nOpt; char **pzVar; } aFts4Opt[] = { { "matchinfo", 9, 0 }, /* 0 -> MATCHINFO */ { "prefix", 6, 0 }, /* 1 -> PREFIX */ { "compress", 8, 0 }, /* 2 -> COMPRESS */ { "uncompress", 10, 0 }, /* 3 -> UNCOMPRESS */ { "order", 5, 0 } /* 4 -> ORDER */ }; int iOpt; if( !zVal ){ rc = SQLITE_NOMEM; }else{ for(iOpt=0; iOpt<SizeofArray(aFts4Opt); iOpt++){ struct Fts4Option *pOp = &aFts4Opt[iOpt]; if( nKey==pOp->nOpt && !sqlite3_strnicmp(z, pOp->zOpt, pOp->nOpt) ){ break; } } if( iOpt==SizeofArray(aFts4Opt) ){ *pzErr = sqlite3_mprintf("unrecognized parameter: %s", z); rc = SQLITE_ERROR; }else{ switch( iOpt ){ case 0: /* MATCHINFO */ if( strlen(zVal)!=4 || sqlite3_strnicmp(zVal, "fts3", 4) ){ *pzErr = sqlite3_mprintf("unrecognized matchinfo: %s", zVal); rc = SQLITE_ERROR; } bNoDocsize = 1; break; case 1: /* PREFIX */ sqlite3_free(zPrefix); zPrefix = zVal; zVal = 0; break; case 2: /* COMPRESS */ sqlite3_free(zCompress); zCompress = zVal; zVal = 0; break; case 3: /* UNCOMPRESS */ sqlite3_free(zUncompress); zUncompress = zVal; zVal = 0; break; case 4: /* ORDER */ if( (strlen(zVal)!=3 || sqlite3_strnicmp(zVal, "asc", 3)) && (strlen(zVal)!=4 || sqlite3_strnicmp(zVal, "desc", 3)) ){ *pzErr = sqlite3_mprintf("unrecognized order: %s", zVal); rc = SQLITE_ERROR; } bDescIdx = (zVal[0]=='d' || zVal[0]=='D'); break; } } sqlite3_free(zVal); } } /* Otherwise, the argument is a column name. */ else { nString += (int)(strlen(z) + 1); aCol[nCol++] = z; } |
︙ | ︙ | |||
951 952 953 954 955 956 957 | if( pTokenizer==0 ){ rc = sqlite3Fts3InitTokenizer(pHash, "simple", &pTokenizer, pzErr); if( rc!=SQLITE_OK ) goto fts3_init_out; } assert( pTokenizer ); | > > > > | > > | > < > > > > > > | | > | | 1039 1040 1041 1042 1043 1044 1045 1046 1047 1048 1049 1050 1051 1052 1053 1054 1055 1056 1057 1058 1059 1060 1061 1062 1063 1064 1065 1066 1067 1068 1069 1070 1071 1072 1073 1074 1075 1076 1077 1078 1079 1080 1081 1082 1083 1084 1085 1086 1087 1088 1089 1090 1091 1092 1093 | if( pTokenizer==0 ){ rc = sqlite3Fts3InitTokenizer(pHash, "simple", &pTokenizer, pzErr); if( rc!=SQLITE_OK ) goto fts3_init_out; } assert( pTokenizer ); rc = fts3PrefixParameter(zPrefix, &nIndex, &aIndex, &aFree); if( rc==SQLITE_ERROR ){ assert( zPrefix ); *pzErr = sqlite3_mprintf("error parsing prefix parameter: %s", zPrefix); } if( rc!=SQLITE_OK ) goto fts3_init_out; /* Allocate and populate the Fts3Table structure. */ nByte = sizeof(Fts3Table) + /* Fts3Table */ nCol * sizeof(char *) + /* azColumn */ nIndex * sizeof(struct Fts3Index) + /* aIndex */ nName + /* zName */ nDb + /* zDb */ nString; /* Space for azColumn strings */ p = (Fts3Table*)sqlite3_malloc(nByte); if( p==0 ){ rc = SQLITE_NOMEM; goto fts3_init_out; } memset(p, 0, nByte); p->db = db; p->nColumn = nCol; p->nPendingData = 0; p->azColumn = (char **)&p[1]; p->pTokenizer = pTokenizer; p->nMaxPendingData = FTS3_MAX_PENDING_DATA; p->bHasDocsize = (isFts4 && bNoDocsize==0); p->bHasStat = isFts4; p->bDescIdx = bDescIdx; TESTONLY( p->inTransaction = -1 ); TESTONLY( p->mxSavepoint = -1 ); p->aIndex = (struct Fts3Index *)&p->azColumn[nCol]; memcpy(p->aIndex, aIndex, sizeof(struct Fts3Index) * nIndex); p->nIndex = nIndex; for(i=0; i<nIndex; i++){ fts3HashInit(&p->aIndex[i].hPending, FTS3_HASH_STRING, 1); } /* Fill in the zName and zDb fields of the vtab structure. */ zCsr = (char *)&p->aIndex[nIndex]; p->zName = zCsr; memcpy(zCsr, argv[2], nName); zCsr += nName; p->zDb = zCsr; memcpy(zCsr, argv[1], nDb); zCsr += nDb; |
︙ | ︙ | |||
1016 1017 1018 1019 1020 1021 1022 | ** database. TODO: For xConnect(), it could verify that said tables exist. */ if( isCreate ){ rc = fts3CreateTables(p); } /* Figure out the page-size for the database. This is required in order to | | < < > > > > | 1117 1118 1119 1120 1121 1122 1123 1124 1125 1126 1127 1128 1129 1130 1131 1132 1133 1134 1135 1136 1137 1138 1139 1140 1141 1142 1143 1144 1145 1146 1147 1148 1149 1150 1151 | ** database. TODO: For xConnect(), it could verify that said tables exist. */ if( isCreate ){ rc = fts3CreateTables(p); } /* Figure out the page-size for the database. This is required in order to ** estimate the cost of loading large doclists from the database. */ fts3DatabasePageSize(&rc, p); p->nNodeSize = p->nPgsz-35; /* Declare the table schema to SQLite. */ fts3DeclareVtab(&rc, p); fts3_init_out: sqlite3_free(zPrefix); sqlite3_free(aFree); sqlite3_free(zCompress); sqlite3_free(zUncompress); sqlite3_free((void *)aCol); if( rc!=SQLITE_OK ){ if( p ){ fts3DisconnectMethod((sqlite3_vtab *)p); }else if( pTokenizer ){ pTokenizer->pModule->xDestroy(pTokenizer); } }else{ assert( p->pSegments==0 ); *ppVTab = &p->base; } return rc; } /* ** The xConnect() and xCreate() methods for the virtual table. All the |
︙ | ︙ | |||
1132 1133 1134 1135 1136 1137 1138 | struct sqlite3_index_orderby *pOrder = &pInfo->aOrderBy[0]; if( pOrder->iColumn<0 || pOrder->iColumn==p->nColumn+1 ){ if( pOrder->desc ){ pInfo->idxStr = "DESC"; }else{ pInfo->idxStr = "ASC"; } | < | | | > > | 1235 1236 1237 1238 1239 1240 1241 1242 1243 1244 1245 1246 1247 1248 1249 1250 1251 1252 1253 | struct sqlite3_index_orderby *pOrder = &pInfo->aOrderBy[0]; if( pOrder->iColumn<0 || pOrder->iColumn==p->nColumn+1 ){ if( pOrder->desc ){ pInfo->idxStr = "DESC"; }else{ pInfo->idxStr = "ASC"; } pInfo->orderByConsumed = 1; } } assert( p->pSegments==0 ); return SQLITE_OK; } /* ** Implementation of xOpen method. */ static int fts3OpenMethod(sqlite3_vtab *pVTab, sqlite3_vtab_cursor **ppCsr){ |
︙ | ︙ | |||
1171 1172 1173 1174 1175 1176 1177 1178 1179 1180 1181 1182 1183 1184 1185 1186 1187 1188 | Fts3Cursor *pCsr = (Fts3Cursor *)pCursor; assert( ((Fts3Table *)pCsr->base.pVtab)->pSegments==0 ); sqlite3_finalize(pCsr->pStmt); sqlite3Fts3ExprFree(pCsr->pExpr); sqlite3Fts3FreeDeferredTokens(pCsr); sqlite3_free(pCsr->aDoclist); sqlite3_free(pCsr->aMatchinfo); sqlite3_free(pCsr); return SQLITE_OK; } /* ** Position the pCsr->pStmt statement so that it is on the row ** of the %_content table that contains the last match. Return ** SQLITE_OK on success. */ static int fts3CursorSeek(sqlite3_context *pContext, Fts3Cursor *pCsr){ if( pCsr->isRequireSeek ){ | > < > | 1275 1276 1277 1278 1279 1280 1281 1282 1283 1284 1285 1286 1287 1288 1289 1290 1291 1292 1293 1294 1295 1296 1297 1298 1299 1300 1301 1302 | Fts3Cursor *pCsr = (Fts3Cursor *)pCursor; assert( ((Fts3Table *)pCsr->base.pVtab)->pSegments==0 ); sqlite3_finalize(pCsr->pStmt); sqlite3Fts3ExprFree(pCsr->pExpr); sqlite3Fts3FreeDeferredTokens(pCsr); sqlite3_free(pCsr->aDoclist); sqlite3_free(pCsr->aMatchinfo); assert( ((Fts3Table *)pCsr->base.pVtab)->pSegments==0 ); sqlite3_free(pCsr); return SQLITE_OK; } /* ** Position the pCsr->pStmt statement so that it is on the row ** of the %_content table that contains the last match. Return ** SQLITE_OK on success. */ static int fts3CursorSeek(sqlite3_context *pContext, Fts3Cursor *pCsr){ if( pCsr->isRequireSeek ){ sqlite3_bind_int64(pCsr->pStmt, 1, pCsr->iPrevId); pCsr->isRequireSeek = 0; if( SQLITE_ROW==sqlite3_step(pCsr->pStmt) ){ return SQLITE_OK; }else{ int rc = sqlite3_reset(pCsr->pStmt); if( rc==SQLITE_OK ){ /* If no row was found and no error has occured, then the %_content ** table is missing a row that is present in the full-text index. |
︙ | ︙ | |||
1364 1365 1366 1367 1368 1369 1370 | assert( !piLeaf2 || !piLeaf || rc!=SQLITE_OK || (*piLeaf<=*piLeaf2) ); if( rc==SQLITE_OK && iHeight>1 ){ char *zBlob = 0; /* Blob read from %_segments table */ int nBlob; /* Size of zBlob in bytes */ if( piLeaf && piLeaf2 && (*piLeaf!=*piLeaf2) ){ | | | | 1469 1470 1471 1472 1473 1474 1475 1476 1477 1478 1479 1480 1481 1482 1483 1484 1485 1486 1487 1488 1489 1490 1491 1492 1493 | assert( !piLeaf2 || !piLeaf || rc!=SQLITE_OK || (*piLeaf<=*piLeaf2) ); if( rc==SQLITE_OK && iHeight>1 ){ char *zBlob = 0; /* Blob read from %_segments table */ int nBlob; /* Size of zBlob in bytes */ if( piLeaf && piLeaf2 && (*piLeaf!=*piLeaf2) ){ rc = sqlite3Fts3ReadBlock(p, *piLeaf, &zBlob, &nBlob, 0); if( rc==SQLITE_OK ){ rc = fts3SelectLeaf(p, zTerm, nTerm, zBlob, nBlob, piLeaf, 0); } sqlite3_free(zBlob); piLeaf = 0; zBlob = 0; } if( rc==SQLITE_OK ){ rc = sqlite3Fts3ReadBlock(p, piLeaf?*piLeaf:*piLeaf2, &zBlob, &nBlob, 0); } if( rc==SQLITE_OK ){ rc = fts3SelectLeaf(p, zTerm, nTerm, zBlob, nBlob, piLeaf, piLeaf2); } sqlite3_free(zBlob); } |
︙ | ︙ | |||
1750 1751 1752 1753 1754 1755 1756 | } *p++ = 0x00; *pp = p; return 1; } /* | | > > > > > > > > > > > > < < < < < < | | | | | | | | | | | | | | | | | | | < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > | < | | | > > > > | | < < | 1855 1856 1857 1858 1859 1860 1861 1862 1863 1864 1865 1866 1867 1868 1869 1870 1871 1872 1873 1874 1875 1876 1877 1878 1879 1880 1881 1882 1883 1884 1885 1886 1887 1888 1889 1890 1891 1892 1893 1894 1895 1896 1897 1898 1899 1900 1901 1902 1903 1904 1905 1906 1907 1908 1909 1910 1911 1912 1913 1914 1915 1916 1917 1918 1919 1920 1921 1922 1923 1924 1925 1926 1927 1928 1929 1930 1931 1932 1933 1934 1935 1936 1937 1938 1939 1940 1941 1942 1943 1944 1945 1946 1947 1948 1949 1950 1951 1952 1953 1954 1955 1956 1957 1958 1959 1960 1961 1962 1963 1964 1965 1966 1967 1968 1969 1970 1971 1972 1973 1974 1975 1976 1977 1978 1979 1980 1981 1982 1983 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022 2023 2024 2025 2026 2027 2028 2029 2030 2031 2032 2033 2034 2035 2036 2037 2038 2039 2040 2041 2042 2043 2044 2045 2046 2047 2048 2049 2050 2051 2052 2053 2054 2055 2056 2057 2058 2059 2060 2061 2062 2063 2064 2065 2066 2067 2068 2069 2070 2071 2072 2073 2074 2075 2076 2077 2078 2079 2080 2081 2082 2083 2084 2085 2086 2087 2088 2089 2090 2091 2092 2093 2094 2095 2096 2097 2098 2099 2100 2101 2102 2103 2104 | } *p++ = 0x00; *pp = p; return 1; } /* ** Merge two position-lists as required by the NEAR operator. The argument ** position lists correspond to the left and right phrases of an expression ** like: ** ** "phrase 1" NEAR "phrase number 2" ** ** Position list *pp1 corresponds to the left-hand side of the NEAR ** expression and *pp2 to the right. As usual, the indexes in the position ** lists are the offsets of the last token in each phrase (tokens "1" and "2" ** in the example above). ** ** The output position list - written to *pp - is a copy of *pp2 with those ** entries that are not sufficiently NEAR entries in *pp1 removed. */ static int fts3PoslistNearMerge( char **pp, /* Output buffer */ char *aTmp, /* Temporary buffer space */ int nRight, /* Maximum difference in token positions */ int nLeft, /* Maximum difference in token positions */ char **pp1, /* IN/OUT: Left input list */ char **pp2 /* IN/OUT: Right input list */ ){ char *p1 = *pp1; char *p2 = *pp2; char *pTmp1 = aTmp; char *pTmp2; char *aTmp2; int res = 1; fts3PoslistPhraseMerge(&pTmp1, nRight, 0, 0, pp1, pp2); aTmp2 = pTmp2 = pTmp1; *pp1 = p1; *pp2 = p2; fts3PoslistPhraseMerge(&pTmp2, nLeft, 1, 0, pp2, pp1); if( pTmp1!=aTmp && pTmp2!=aTmp2 ){ fts3PoslistMerge(pp, &aTmp, &aTmp2); }else if( pTmp1!=aTmp ){ fts3PoslistCopy(pp, &aTmp); }else if( pTmp2!=aTmp2 ){ fts3PoslistCopy(pp, &aTmp2); }else{ res = 0; } return res; } /* ** A pointer to an instance of this structure is used as the context ** argument to sqlite3Fts3SegReaderIterate() */ typedef struct TermSelect TermSelect; struct TermSelect { int isReqPos; char *aaOutput[16]; /* Malloc'd output buffer */ int anOutput[16]; /* Size of output in bytes */ }; static void fts3GetDeltaVarint3( char **pp, char *pEnd, int bDescIdx, sqlite3_int64 *pVal ){ if( *pp>=pEnd ){ *pp = 0; }else{ sqlite3_int64 iVal; *pp += sqlite3Fts3GetVarint(*pp, &iVal); if( bDescIdx ){ *pVal -= iVal; }else{ *pVal += iVal; } } } static void fts3PutDeltaVarint3( char **pp, /* IN/OUT: Output pointer */ int bDescIdx, /* True for descending docids */ sqlite3_int64 *piPrev, /* IN/OUT: Previous value written to list */ int *pbFirst, /* IN/OUT: True after first int written */ sqlite3_int64 iVal /* Write this value to the list */ ){ sqlite3_int64 iWrite; if( bDescIdx==0 || *pbFirst==0 ){ iWrite = iVal - *piPrev; }else{ iWrite = *piPrev - iVal; } assert( *pbFirst || *piPrev==0 ); assert( *pbFirst==0 || iWrite>0 ); *pp += sqlite3Fts3PutVarint(*pp, iWrite); *piPrev = iVal; *pbFirst = 1; } #define COMPARE_DOCID(i1, i2) ((bDescIdx?-1:1) * (i1-i2)) static int fts3DoclistOrMerge( int bDescIdx, /* True if arguments are desc */ char *a1, int n1, /* First doclist */ char *a2, int n2, /* Second doclist */ char **paOut, int *pnOut /* OUT: Malloc'd doclist */ ){ sqlite3_int64 i1 = 0; sqlite3_int64 i2 = 0; sqlite3_int64 iPrev = 0; char *pEnd1 = &a1[n1]; char *pEnd2 = &a2[n2]; char *p1 = a1; char *p2 = a2; char *p; char *aOut; int bFirstOut = 0; *paOut = 0; *pnOut = 0; aOut = sqlite3_malloc(n1+n2); if( !aOut ) return SQLITE_NOMEM; p = aOut; fts3GetDeltaVarint3(&p1, pEnd1, 0, &i1); fts3GetDeltaVarint3(&p2, pEnd2, 0, &i2); while( p1 || p2 ){ sqlite3_int64 iDiff = COMPARE_DOCID(i1, i2); if( p2 && p1 && iDiff==0 ){ fts3PutDeltaVarint3(&p, bDescIdx, &iPrev, &bFirstOut, i1); fts3PoslistMerge(&p, &p1, &p2); fts3GetDeltaVarint3(&p1, pEnd1, bDescIdx, &i1); fts3GetDeltaVarint3(&p2, pEnd2, bDescIdx, &i2); }else if( !p2 || (p1 && iDiff<0) ){ fts3PutDeltaVarint3(&p, bDescIdx, &iPrev, &bFirstOut, i1); fts3PoslistCopy(&p, &p1); fts3GetDeltaVarint3(&p1, pEnd1, bDescIdx, &i1); }else{ fts3PutDeltaVarint3(&p, bDescIdx, &iPrev, &bFirstOut, i2); fts3PoslistCopy(&p, &p2); fts3GetDeltaVarint3(&p2, pEnd2, bDescIdx, &i2); } } *paOut = aOut; *pnOut = (p-aOut); return SQLITE_OK; } static void fts3DoclistPhraseMerge( int bDescIdx, /* True if arguments are desc */ int nDist, /* Distance from left to right (1=adjacent) */ char *aLeft, int nLeft, /* Left doclist */ char *aRight, int *pnRight /* IN/OUT: Right/output doclist */ ){ sqlite3_int64 i1 = 0; sqlite3_int64 i2 = 0; sqlite3_int64 iPrev = 0; char *pEnd1 = &aLeft[nLeft]; char *pEnd2 = &aRight[*pnRight]; char *p1 = aLeft; char *p2 = aRight; char *p; int bFirstOut = 0; char *aOut = aRight; assert( nDist>0 ); p = aOut; fts3GetDeltaVarint3(&p1, pEnd1, 0, &i1); fts3GetDeltaVarint3(&p2, pEnd2, 0, &i2); while( p1 && p2 ){ sqlite3_int64 iDiff = COMPARE_DOCID(i1, i2); if( iDiff==0 ){ char *pSave = p; sqlite3_int64 iPrevSave = iPrev; int bFirstOutSave = bFirstOut; fts3PutDeltaVarint3(&p, bDescIdx, &iPrev, &bFirstOut, i1); if( 0==fts3PoslistPhraseMerge(&p, nDist, 0, 1, &p1, &p2) ){ p = pSave; iPrev = iPrevSave; bFirstOut = bFirstOutSave; } fts3GetDeltaVarint3(&p1, pEnd1, bDescIdx, &i1); fts3GetDeltaVarint3(&p2, pEnd2, bDescIdx, &i2); }else if( iDiff<0 ){ fts3PoslistCopy(0, &p1); fts3GetDeltaVarint3(&p1, pEnd1, bDescIdx, &i1); }else{ fts3PoslistCopy(0, &p2); fts3GetDeltaVarint3(&p2, pEnd2, bDescIdx, &i2); } } *pnRight = p - aOut; } /* ** Merge all doclists in the TermSelect.aaOutput[] array into a single ** doclist stored in TermSelect.aaOutput[0]. If successful, delete all ** other doclists (except the aaOutput[0] one) and return SQLITE_OK. ** ** If an OOM error occurs, return SQLITE_NOMEM. In this case it is ** the responsibility of the caller to free any doclists left in the ** TermSelect.aaOutput[] array. */ static int fts3TermSelectMerge(Fts3Table *p, TermSelect *pTS){ char *aOut = 0; int nOut = 0; int i; /* Loop through the doclists in the aaOutput[] array. Merge them all ** into a single doclist. */ for(i=0; i<SizeofArray(pTS->aaOutput); i++){ if( pTS->aaOutput[i] ){ if( !aOut ){ aOut = pTS->aaOutput[i]; nOut = pTS->anOutput[i]; pTS->aaOutput[i] = 0; }else{ int nNew; char *aNew; int rc = fts3DoclistOrMerge(p->bDescIdx, pTS->aaOutput[i], pTS->anOutput[i], aOut, nOut, &aNew, &nNew ); if( rc!=SQLITE_OK ){ sqlite3_free(aOut); return rc; } sqlite3_free(pTS->aaOutput[i]); sqlite3_free(aOut); pTS->aaOutput[i] = 0; aOut = aNew; nOut = nNew; } } |
︙ | ︙ | |||
2053 2054 2055 2056 2057 2058 2059 | UNUSED_PARAMETER(p); UNUSED_PARAMETER(zTerm); UNUSED_PARAMETER(nTerm); if( pTS->aaOutput[0]==0 ){ /* If this is the first term selected, copy the doclist to the output | | < < < < < | > > > | | | | < < | | < < < | | | | | | | | | | | | > > > > | > | < < < > | | | < | < < < < < < < < | < < | < | > | > | > | < < < < | < < < < < | | | | < < > | > | | | < < < < < < < | | | < | > > < < < < < < < < < < < < | | | < < > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > | | | | > > | > > > | | | > | > | > > > | > > > > | > > > > > > > > | > | | 2126 2127 2128 2129 2130 2131 2132 2133 2134 2135 2136 2137 2138 2139 2140 2141 2142 2143 2144 2145 2146 2147 2148 2149 2150 2151 2152 2153 2154 2155 2156 2157 2158 2159 2160 2161 2162 2163 2164 2165 2166 2167 2168 2169 2170 2171 2172 2173 2174 2175 2176 2177 2178 2179 2180 2181 2182 2183 2184 2185 2186 2187 2188 2189 2190 2191 2192 2193 2194 2195 2196 2197 2198 2199 2200 2201 2202 2203 2204 2205 2206 2207 2208 2209 2210 2211 2212 2213 2214 2215 2216 2217 2218 2219 2220 2221 2222 2223 2224 2225 2226 2227 2228 2229 2230 2231 2232 2233 2234 2235 2236 2237 2238 2239 2240 2241 2242 2243 2244 2245 2246 2247 2248 2249 2250 2251 2252 2253 2254 2255 2256 2257 2258 2259 2260 2261 2262 2263 2264 2265 2266 2267 2268 2269 2270 2271 2272 2273 2274 2275 2276 2277 2278 2279 2280 2281 2282 2283 2284 2285 2286 2287 2288 2289 2290 2291 2292 2293 2294 2295 2296 2297 2298 2299 2300 2301 2302 2303 2304 2305 2306 2307 2308 2309 2310 2311 2312 2313 2314 2315 2316 2317 2318 2319 2320 2321 2322 2323 2324 2325 2326 2327 2328 2329 2330 2331 2332 2333 2334 2335 2336 2337 2338 2339 2340 2341 2342 2343 2344 2345 2346 2347 2348 2349 2350 2351 2352 2353 2354 2355 2356 2357 2358 2359 2360 2361 2362 2363 2364 2365 2366 2367 2368 2369 2370 | UNUSED_PARAMETER(p); UNUSED_PARAMETER(zTerm); UNUSED_PARAMETER(nTerm); if( pTS->aaOutput[0]==0 ){ /* If this is the first term selected, copy the doclist to the output ** buffer using memcpy(). */ pTS->aaOutput[0] = sqlite3_malloc(nDoclist); pTS->anOutput[0] = nDoclist; if( pTS->aaOutput[0] ){ memcpy(pTS->aaOutput[0], aDoclist, nDoclist); }else{ return SQLITE_NOMEM; } }else{ char *aMerge = aDoclist; int nMerge = nDoclist; int iOut; for(iOut=0; iOut<SizeofArray(pTS->aaOutput); iOut++){ if( pTS->aaOutput[iOut]==0 ){ assert( iOut>0 ); pTS->aaOutput[iOut] = aMerge; pTS->anOutput[iOut] = nMerge; break; }else{ char *aNew; int nNew; int rc = fts3DoclistOrMerge(p->bDescIdx, aMerge, nMerge, pTS->aaOutput[iOut], pTS->anOutput[iOut], &aNew, &nNew ); if( rc!=SQLITE_OK ){ if( aMerge!=aDoclist ) sqlite3_free(aMerge); return rc; } if( aMerge!=aDoclist ) sqlite3_free(aMerge); sqlite3_free(pTS->aaOutput[iOut]); pTS->aaOutput[iOut] = 0; aMerge = aNew; nMerge = nNew; if( (iOut+1)==SizeofArray(pTS->aaOutput) ){ pTS->aaOutput[iOut] = aMerge; pTS->anOutput[iOut] = nMerge; } } } } return SQLITE_OK; } /* ** Append SegReader object pNew to the end of the pCsr->apSegment[] array. */ static int fts3SegReaderCursorAppend( Fts3MultiSegReader *pCsr, Fts3SegReader *pNew ){ if( (pCsr->nSegment%16)==0 ){ Fts3SegReader **apNew; int nByte = (pCsr->nSegment + 16)*sizeof(Fts3SegReader*); apNew = (Fts3SegReader **)sqlite3_realloc(pCsr->apSegment, nByte); if( !apNew ){ sqlite3Fts3SegReaderFree(pNew); return SQLITE_NOMEM; } pCsr->apSegment = apNew; } pCsr->apSegment[pCsr->nSegment++] = pNew; return SQLITE_OK; } static int fts3SegReaderCursor( Fts3Table *p, /* FTS3 table handle */ int iIndex, /* Index to search (from 0 to p->nIndex-1) */ int iLevel, /* Level of segments to scan */ const char *zTerm, /* Term to query for */ int nTerm, /* Size of zTerm in bytes */ int isPrefix, /* True for a prefix search */ int isScan, /* True to scan from zTerm to EOF */ Fts3MultiSegReader *pCsr /* Cursor object to populate */ ){ int rc = SQLITE_OK; int rc2; sqlite3_stmt *pStmt = 0; /* If iLevel is less than 0 and this is not a scan, include a seg-reader ** for the pending-terms. If this is a scan, then this call must be being ** made by an fts4aux module, not an FTS table. In this case calling ** Fts3SegReaderPending might segfault, as the data structures used by ** fts4aux are not completely populated. So it's easiest to filter these ** calls out here. */ if( iLevel<0 && p->aIndex ){ Fts3SegReader *pSeg = 0; rc = sqlite3Fts3SegReaderPending(p, iIndex, zTerm, nTerm, isPrefix, &pSeg); if( rc==SQLITE_OK && pSeg ){ rc = fts3SegReaderCursorAppend(pCsr, pSeg); } } if( iLevel!=FTS3_SEGCURSOR_PENDING ){ if( rc==SQLITE_OK ){ rc = sqlite3Fts3AllSegdirs(p, iIndex, iLevel, &pStmt); } while( rc==SQLITE_OK && SQLITE_ROW==(rc = sqlite3_step(pStmt)) ){ Fts3SegReader *pSeg = 0; /* Read the values returned by the SELECT into local variables. */ sqlite3_int64 iStartBlock = sqlite3_column_int64(pStmt, 1); sqlite3_int64 iLeavesEndBlock = sqlite3_column_int64(pStmt, 2); sqlite3_int64 iEndBlock = sqlite3_column_int64(pStmt, 3); int nRoot = sqlite3_column_bytes(pStmt, 4); char const *zRoot = sqlite3_column_blob(pStmt, 4); /* If zTerm is not NULL, and this segment is not stored entirely on its ** root node, the range of leaves scanned can be reduced. Do this. */ if( iStartBlock && zTerm ){ sqlite3_int64 *pi = (isPrefix ? &iLeavesEndBlock : 0); rc = fts3SelectLeaf(p, zTerm, nTerm, zRoot, nRoot, &iStartBlock, pi); if( rc!=SQLITE_OK ) goto finished; if( isPrefix==0 && isScan==0 ) iLeavesEndBlock = iStartBlock; } rc = sqlite3Fts3SegReaderNew(pCsr->nSegment+1, iStartBlock, iLeavesEndBlock, iEndBlock, zRoot, nRoot, &pSeg ); if( rc!=SQLITE_OK ) goto finished; rc = fts3SegReaderCursorAppend(pCsr, pSeg); } } finished: rc2 = sqlite3_reset(pStmt); if( rc==SQLITE_DONE ) rc = rc2; return rc; } /* ** Set up a cursor object for iterating through a full-text index or a ** single level therein. */ int sqlite3Fts3SegReaderCursor( Fts3Table *p, /* FTS3 table handle */ int iIndex, /* Index to search (from 0 to p->nIndex-1) */ int iLevel, /* Level of segments to scan */ const char *zTerm, /* Term to query for */ int nTerm, /* Size of zTerm in bytes */ int isPrefix, /* True for a prefix search */ int isScan, /* True to scan from zTerm to EOF */ Fts3MultiSegReader *pCsr /* Cursor object to populate */ ){ assert( iIndex>=0 && iIndex<p->nIndex ); assert( iLevel==FTS3_SEGCURSOR_ALL || iLevel==FTS3_SEGCURSOR_PENDING || iLevel>=0 ); assert( iLevel<FTS3_SEGDIR_MAXLEVEL ); assert( FTS3_SEGCURSOR_ALL<0 && FTS3_SEGCURSOR_PENDING<0 ); assert( isPrefix==0 || isScan==0 ); /* "isScan" is only set to true by the ft4aux module, an ordinary ** full-text tables. */ assert( isScan==0 || p->aIndex==0 ); memset(pCsr, 0, sizeof(Fts3MultiSegReader)); return fts3SegReaderCursor( p, iIndex, iLevel, zTerm, nTerm, isPrefix, isScan, pCsr ); } static int fts3SegReaderCursorAddZero( Fts3Table *p, const char *zTerm, int nTerm, Fts3MultiSegReader *pCsr ){ return fts3SegReaderCursor(p, 0, FTS3_SEGCURSOR_ALL, zTerm, nTerm, 0, 0,pCsr); } int sqlite3Fts3TermSegReaderCursor( Fts3Cursor *pCsr, /* Virtual table cursor handle */ const char *zTerm, /* Term to query for */ int nTerm, /* Size of zTerm in bytes */ int isPrefix, /* True for a prefix search */ Fts3MultiSegReader **ppSegcsr /* OUT: Allocated seg-reader cursor */ ){ Fts3MultiSegReader *pSegcsr; /* Object to allocate and return */ int rc = SQLITE_NOMEM; /* Return code */ pSegcsr = sqlite3_malloc(sizeof(Fts3MultiSegReader)); if( pSegcsr ){ int i; int bFound = 0; /* True once an index has been found */ Fts3Table *p = (Fts3Table *)pCsr->base.pVtab; if( isPrefix ){ for(i=1; bFound==0 && i<p->nIndex; i++){ if( p->aIndex[i].nPrefix==nTerm ){ bFound = 1; rc = sqlite3Fts3SegReaderCursor( p, i, FTS3_SEGCURSOR_ALL, zTerm, nTerm, 0, 0, pSegcsr); pSegcsr->bLookup = 1; } } for(i=1; bFound==0 && i<p->nIndex; i++){ if( p->aIndex[i].nPrefix==nTerm+1 ){ bFound = 1; rc = sqlite3Fts3SegReaderCursor( p, i, FTS3_SEGCURSOR_ALL, zTerm, nTerm, 1, 0, pSegcsr ); if( rc==SQLITE_OK ){ rc = fts3SegReaderCursorAddZero(p, zTerm, nTerm, pSegcsr); } } } } if( bFound==0 ){ rc = sqlite3Fts3SegReaderCursor( p, 0, FTS3_SEGCURSOR_ALL, zTerm, nTerm, isPrefix, 0, pSegcsr ); pSegcsr->bLookup = !isPrefix; } } *ppSegcsr = pSegcsr; return rc; } static void fts3SegReaderCursorFree(Fts3MultiSegReader *pSegcsr){ sqlite3Fts3SegReaderFinish(pSegcsr); sqlite3_free(pSegcsr); } /* ** This function retreives the doclist for the specified term (or term ** prefix) from the database. |
︙ | ︙ | |||
2284 2285 2286 2287 2288 2289 2290 | Fts3PhraseToken *pTok, /* Token to query for */ int iColumn, /* Column to query (or -ve for all columns) */ int isReqPos, /* True to include position lists in output */ int *pnOut, /* OUT: Size of buffer at *ppOut */ char **ppOut /* OUT: Malloced result buffer */ ){ int rc; /* Return code */ | | | 2381 2382 2383 2384 2385 2386 2387 2388 2389 2390 2391 2392 2393 2394 2395 | Fts3PhraseToken *pTok, /* Token to query for */ int iColumn, /* Column to query (or -ve for all columns) */ int isReqPos, /* True to include position lists in output */ int *pnOut, /* OUT: Size of buffer at *ppOut */ char **ppOut /* OUT: Malloced result buffer */ ){ int rc; /* Return code */ Fts3MultiSegReader *pSegcsr; /* Seg-reader cursor for this term */ TermSelect tsc; /* Context object for fts3TermSelectCb() */ Fts3SegFilter filter; /* Segment term filter configuration */ pSegcsr = pTok->pSegcsr; memset(&tsc, 0, sizeof(TermSelect)); tsc.isReqPos = isReqPos; |
︙ | ︙ | |||
2310 2311 2312 2313 2314 2315 2316 | ){ rc = fts3TermSelectCb(p, (void *)&tsc, pSegcsr->zTerm, pSegcsr->nTerm, pSegcsr->aDoclist, pSegcsr->nDoclist ); } if( rc==SQLITE_OK ){ | | | 2407 2408 2409 2410 2411 2412 2413 2414 2415 2416 2417 2418 2419 2420 2421 | ){ rc = fts3TermSelectCb(p, (void *)&tsc, pSegcsr->zTerm, pSegcsr->nTerm, pSegcsr->aDoclist, pSegcsr->nDoclist ); } if( rc==SQLITE_OK ){ rc = fts3TermSelectMerge(p, &tsc); } if( rc==SQLITE_OK ){ *ppOut = tsc.aaOutput[0]; *pnOut = tsc.anOutput[0]; }else{ int i; for(i=0; i<SizeofArray(tsc.aaOutput); i++){ |
︙ | ︙ | |||
2360 2361 2362 2363 2364 2365 2366 | } } } return nDoc; } | < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < | < | < < < | | | < | < < < | < | < | < < < < < < | < < | < | | 2457 2458 2459 2460 2461 2462 2463 2464 2465 2466 2467 2468 2469 2470 2471 2472 2473 2474 2475 2476 2477 2478 2479 2480 2481 2482 2483 2484 2485 2486 2487 2488 2489 2490 2491 2492 2493 2494 2495 2496 | } } } return nDoc; } /* ** Advance the cursor to the next row in the %_content table that ** matches the search criteria. For a MATCH search, this will be ** the next row that matches. For a full-table scan, this will be ** simply the next row in the %_content table. For a docid lookup, ** this routine simply sets the EOF flag. ** ** Return SQLITE_OK if nothing goes wrong. SQLITE_OK is returned ** even if we reach end-of-file. The fts3EofMethod() will be called ** subsequently to determine whether or not an EOF was hit. */ static int fts3NextMethod(sqlite3_vtab_cursor *pCursor){ int rc; Fts3Cursor *pCsr = (Fts3Cursor *)pCursor; if( pCsr->eSearch==FTS3_DOCID_SEARCH || pCsr->eSearch==FTS3_FULLSCAN_SEARCH ){ if( SQLITE_ROW!=sqlite3_step(pCsr->pStmt) ){ pCsr->isEof = 1; rc = sqlite3_reset(pCsr->pStmt); }else{ pCsr->iPrevId = sqlite3_column_int64(pCsr->pStmt, 0); rc = SQLITE_OK; } }else{ rc = sqlite3Fts3EvalNext((Fts3Cursor *)pCursor); } assert( ((Fts3Table *)pCsr->base.pVtab)->pSegments==0 ); return rc; } /* ** This is the xFilter interface for the virtual table. See ** the virtual table xFilter method documentation for additional ** information. |
︙ | ︙ | |||
3085 3086 3087 3088 3089 3090 3091 | static int fts3FilterMethod( sqlite3_vtab_cursor *pCursor, /* The cursor used for this query */ int idxNum, /* Strategy index */ const char *idxStr, /* Unused */ int nVal, /* Number of elements in apVal */ sqlite3_value **apVal /* Arguments for the indexing scheme */ ){ | < < < < | > > > > > > > > | < | > | > > | > > | < > | < < | | > | | < < < < | < < < < < | < < < < < < < | < | < < < < < < < < | < < | < > > | 2509 2510 2511 2512 2513 2514 2515 2516 2517 2518 2519 2520 2521 2522 2523 2524 2525 2526 2527 2528 2529 2530 2531 2532 2533 2534 2535 2536 2537 2538 2539 2540 2541 2542 2543 2544 2545 2546 2547 2548 2549 2550 2551 2552 2553 2554 2555 2556 2557 2558 2559 2560 2561 2562 2563 2564 2565 2566 2567 2568 2569 2570 2571 2572 2573 2574 2575 2576 2577 2578 2579 2580 2581 2582 2583 2584 2585 2586 2587 2588 2589 2590 2591 2592 2593 2594 2595 2596 2597 2598 2599 2600 2601 2602 2603 2604 2605 2606 2607 2608 2609 2610 2611 2612 2613 2614 2615 2616 2617 2618 2619 2620 2621 2622 2623 2624 2625 2626 2627 2628 2629 2630 2631 2632 2633 2634 2635 2636 2637 2638 2639 2640 2641 2642 2643 2644 2645 2646 2647 2648 2649 2650 2651 2652 2653 2654 2655 2656 2657 2658 | static int fts3FilterMethod( sqlite3_vtab_cursor *pCursor, /* The cursor used for this query */ int idxNum, /* Strategy index */ const char *idxStr, /* Unused */ int nVal, /* Number of elements in apVal */ sqlite3_value **apVal /* Arguments for the indexing scheme */ ){ int rc; char *zSql; /* SQL statement used to access %_content */ Fts3Table *p = (Fts3Table *)pCursor->pVtab; Fts3Cursor *pCsr = (Fts3Cursor *)pCursor; UNUSED_PARAMETER(idxStr); UNUSED_PARAMETER(nVal); assert( idxNum>=0 && idxNum<=(FTS3_FULLTEXT_SEARCH+p->nColumn) ); assert( nVal==0 || nVal==1 ); assert( (nVal==0)==(idxNum==FTS3_FULLSCAN_SEARCH) ); assert( p->pSegments==0 ); /* In case the cursor has been used before, clear it now. */ sqlite3_finalize(pCsr->pStmt); sqlite3_free(pCsr->aDoclist); sqlite3Fts3ExprFree(pCsr->pExpr); memset(&pCursor[1], 0, sizeof(Fts3Cursor)-sizeof(sqlite3_vtab_cursor)); if( idxStr ){ pCsr->bDesc = (idxStr[0]=='D'); }else{ pCsr->bDesc = p->bDescIdx; } pCsr->eSearch = (i16)idxNum; if( idxNum!=FTS3_DOCID_SEARCH && idxNum!=FTS3_FULLSCAN_SEARCH ){ int iCol = idxNum-FTS3_FULLTEXT_SEARCH; const char *zQuery = (const char *)sqlite3_value_text(apVal[0]); if( zQuery==0 && sqlite3_value_type(apVal[0])!=SQLITE_NULL ){ return SQLITE_NOMEM; } rc = sqlite3Fts3ExprParse(p->pTokenizer, p->azColumn, p->nColumn, iCol, zQuery, -1, &pCsr->pExpr ); if( rc!=SQLITE_OK ){ if( rc==SQLITE_ERROR ){ static const char *zErr = "malformed MATCH expression: [%s]"; p->base.zErrMsg = sqlite3_mprintf(zErr, zQuery); } return rc; } rc = sqlite3Fts3ReadLock(p); if( rc!=SQLITE_OK ) return rc; rc = sqlite3Fts3EvalStart(pCsr, pCsr->pExpr, 1); sqlite3Fts3SegmentsClose(p); if( rc!=SQLITE_OK ) return rc; pCsr->pNextId = pCsr->aDoclist; pCsr->iPrevId = 0; } /* Compile a SELECT statement for this cursor. For a full-table-scan, the ** statement loops through all rows of the %_content table. For a ** full-text query or docid lookup, the statement retrieves a single ** row by docid. */ if( idxNum==FTS3_FULLSCAN_SEARCH ){ const char *zSort = (pCsr->bDesc ? "DESC" : "ASC"); const char *zTmpl = "SELECT %s FROM %Q.'%q_content' AS x ORDER BY docid %s"; zSql = sqlite3_mprintf(zTmpl, p->zReadExprlist, p->zDb, p->zName, zSort); }else{ const char *zTmpl = "SELECT %s FROM %Q.'%q_content' AS x WHERE docid = ?"; zSql = sqlite3_mprintf(zTmpl, p->zReadExprlist, p->zDb, p->zName); } if( !zSql ) return SQLITE_NOMEM; rc = sqlite3_prepare_v2(p->db, zSql, -1, &pCsr->pStmt, 0); sqlite3_free(zSql); if( rc!=SQLITE_OK ) return rc; if( idxNum==FTS3_DOCID_SEARCH ){ rc = sqlite3_bind_value(pCsr->pStmt, 1, apVal[0]); if( rc!=SQLITE_OK ) return rc; } return fts3NextMethod(pCursor); } /* ** This is the xEof method of the virtual table. SQLite calls this ** routine to find out if it has reached the end of a result set. */ static int fts3EofMethod(sqlite3_vtab_cursor *pCursor){ return ((Fts3Cursor *)pCursor)->isEof; } /* ** This is the xRowid method. The SQLite core calls this routine to ** retrieve the rowid for the current row of the result set. fts3 ** exposes %_content.docid as the rowid for the virtual table. The ** rowid should be written to *pRowid. */ static int fts3RowidMethod(sqlite3_vtab_cursor *pCursor, sqlite_int64 *pRowid){ Fts3Cursor *pCsr = (Fts3Cursor *) pCursor; *pRowid = pCsr->iPrevId; return SQLITE_OK; } /* ** This is the xColumn method, called by SQLite to request a value from ** the row that the supplied cursor currently points to. */ static int fts3ColumnMethod( sqlite3_vtab_cursor *pCursor, /* Cursor to retrieve value from */ sqlite3_context *pContext, /* Context for sqlite3_result_xxx() calls */ int iCol /* Index of column to read value from */ ){ int rc = SQLITE_OK; /* Return Code */ Fts3Cursor *pCsr = (Fts3Cursor *) pCursor; Fts3Table *p = (Fts3Table *)pCursor->pVtab; /* The column value supplied by SQLite must be in range. */ assert( iCol>=0 && iCol<=p->nColumn+1 ); if( iCol==p->nColumn+1 ){ /* This call is a request for the "docid" column. Since "docid" is an ** alias for "rowid", use the xRowid() method to obtain the value. */ sqlite3_result_int64(pContext, pCsr->iPrevId); }else if( iCol==p->nColumn ){ /* The extra column whose name is the same as the table. ** Return a blob which is a pointer to the cursor. */ sqlite3_result_blob(pContext, &pCsr, sizeof(pCsr), SQLITE_TRANSIENT); }else{ rc = fts3CursorSeek(0, pCsr); if( rc==SQLITE_OK ){ sqlite3_result_value(pContext, sqlite3_column_value(pCsr->pStmt, iCol+1)); } } assert( ((Fts3Table *)pCsr->base.pVtab)->pSegments==0 ); return rc; } /* ** This function is the implementation of the xUpdate callback used by ** FTS3 virtual tables. It is invoked by SQLite each time a row is to be ** inserted, updated or deleted. |
︙ | ︙ | |||
3273 3274 3275 3276 3277 3278 3279 3280 3281 3282 3283 3284 3285 3286 3287 3288 3289 3290 3291 3292 3293 3294 3295 3296 3297 3298 3299 3300 3301 3302 3303 3304 3305 3306 3307 3308 3309 3310 3311 3312 3313 | /* ** Implementation of xBegin() method. This is a no-op. */ static int fts3BeginMethod(sqlite3_vtab *pVtab){ UNUSED_PARAMETER(pVtab); TESTONLY( Fts3Table *p = (Fts3Table*)pVtab ); assert( p->nPendingData==0 ); assert( p->inTransaction!=1 ); TESTONLY( p->inTransaction = 1 ); TESTONLY( p->mxSavepoint = -1; ); return SQLITE_OK; } /* ** Implementation of xCommit() method. This is a no-op. The contents of ** the pending-terms hash-table have already been flushed into the database ** by fts3SyncMethod(). */ static int fts3CommitMethod(sqlite3_vtab *pVtab){ UNUSED_PARAMETER(pVtab); TESTONLY( Fts3Table *p = (Fts3Table*)pVtab ); assert( p->nPendingData==0 ); assert( p->inTransaction!=0 ); TESTONLY( p->inTransaction = 0 ); TESTONLY( p->mxSavepoint = -1; ); return SQLITE_OK; } /* ** Implementation of xRollback(). Discard the contents of the pending-terms ** hash-table. Any changes made to the database are reverted by SQLite. */ static int fts3RollbackMethod(sqlite3_vtab *pVtab){ Fts3Table *p = (Fts3Table*)pVtab; sqlite3Fts3PendingTermsClear(p); assert( p->inTransaction!=0 ); TESTONLY( p->inTransaction = 0 ); TESTONLY( p->mxSavepoint = -1; ); return SQLITE_OK; } | > > < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < | | > > < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < | 2678 2679 2680 2681 2682 2683 2684 2685 2686 2687 2688 2689 2690 2691 2692 2693 2694 2695 2696 2697 2698 2699 2700 2701 2702 2703 2704 2705 2706 2707 2708 2709 2710 2711 2712 2713 2714 2715 2716 2717 2718 2719 2720 2721 2722 2723 2724 2725 2726 2727 2728 2729 2730 2731 2732 2733 2734 2735 2736 2737 2738 2739 2740 2741 2742 2743 2744 2745 2746 2747 | /* ** Implementation of xBegin() method. This is a no-op. */ static int fts3BeginMethod(sqlite3_vtab *pVtab){ UNUSED_PARAMETER(pVtab); TESTONLY( Fts3Table *p = (Fts3Table*)pVtab ); assert( p->pSegments==0 ); assert( p->nPendingData==0 ); assert( p->inTransaction!=1 ); TESTONLY( p->inTransaction = 1 ); TESTONLY( p->mxSavepoint = -1; ); return SQLITE_OK; } /* ** Implementation of xCommit() method. This is a no-op. The contents of ** the pending-terms hash-table have already been flushed into the database ** by fts3SyncMethod(). */ static int fts3CommitMethod(sqlite3_vtab *pVtab){ UNUSED_PARAMETER(pVtab); TESTONLY( Fts3Table *p = (Fts3Table*)pVtab ); assert( p->nPendingData==0 ); assert( p->inTransaction!=0 ); assert( p->pSegments==0 ); TESTONLY( p->inTransaction = 0 ); TESTONLY( p->mxSavepoint = -1; ); return SQLITE_OK; } /* ** Implementation of xRollback(). Discard the contents of the pending-terms ** hash-table. Any changes made to the database are reverted by SQLite. */ static int fts3RollbackMethod(sqlite3_vtab *pVtab){ Fts3Table *p = (Fts3Table*)pVtab; sqlite3Fts3PendingTermsClear(p); assert( p->inTransaction!=0 ); TESTONLY( p->inTransaction = 0 ); TESTONLY( p->mxSavepoint = -1; ); return SQLITE_OK; } /* ** When called, *ppPoslist must point to the byte immediately following the ** end of a position-list. i.e. ( (*ppPoslist)[-1]==POS_END ). This function ** moves *ppPoslist so that it instead points to the first byte of the ** same position list. */ static void fts3ReversePoslist(char *pStart, char **ppPoslist){ char *p = &(*ppPoslist)[-2]; char c; while( p>pStart && (c=*p--)==0 ); while( p>pStart && (*p & 0x80) | c ){ c = *p--; } if( p>pStart ){ p = &p[2]; } while( *p++&0x80 ); *ppPoslist = p; } /* ** Helper function used by the implementation of the overloaded snippet(), ** offsets() and optimize() SQL functions. ** ** If the value passed as the third argument is a blob of size ** sizeof(Fts3Cursor*), then the blob contents are copied to the ** output variable *ppCsr and SQLITE_OK is returned. Otherwise, an error |
︙ | ︙ | |||
3660 3661 3662 3663 3664 3665 3666 | "ALTER TABLE %Q.'%q_segdir' RENAME TO '%q_segdir';", p->zDb, p->zName, zName ); return rc; } static int fts3SavepointMethod(sqlite3_vtab *pVtab, int iSavepoint){ | < | | | | | 2965 2966 2967 2968 2969 2970 2971 2972 2973 2974 2975 2976 2977 2978 2979 2980 2981 2982 2983 | "ALTER TABLE %Q.'%q_segdir' RENAME TO '%q_segdir';", p->zDb, p->zName, zName ); return rc; } static int fts3SavepointMethod(sqlite3_vtab *pVtab, int iSavepoint){ UNUSED_PARAMETER(iSavepoint); assert( ((Fts3Table *)pVtab)->inTransaction ); assert( ((Fts3Table *)pVtab)->mxSavepoint < iSavepoint ); TESTONLY( ((Fts3Table *)pVtab)->mxSavepoint = iSavepoint ); return fts3SyncMethod(pVtab); } static int fts3ReleaseMethod(sqlite3_vtab *pVtab, int iSavepoint){ TESTONLY( Fts3Table *p = (Fts3Table*)pVtab ); UNUSED_PARAMETER(iSavepoint); UNUSED_PARAMETER(pVtab); assert( p->inTransaction ); assert( p->mxSavepoint >= iSavepoint ); |
︙ | ︙ | |||
3836 3837 3838 3839 3840 3841 3842 3843 | const sqlite3_api_routines *pApi ){ SQLITE_EXTENSION_INIT2(pApi) return sqlite3Fts3Init(db); } #endif #endif | > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > | 3140 3141 3142 3143 3144 3145 3146 3147 3148 3149 3150 3151 3152 3153 3154 3155 3156 3157 3158 3159 3160 3161 3162 3163 3164 3165 3166 3167 3168 3169 3170 3171 3172 3173 3174 3175 3176 3177 3178 3179 3180 3181 3182 3183 3184 3185 3186 3187 3188 3189 3190 3191 3192 3193 3194 3195 3196 3197 3198 3199 3200 3201 3202 3203 3204 3205 3206 3207 3208 3209 3210 3211 3212 3213 3214 3215 3216 3217 3218 3219 3220 3221 3222 3223 3224 3225 3226 3227 3228 3229 3230 3231 3232 3233 3234 3235 3236 3237 3238 3239 3240 3241 3242 3243 3244 3245 3246 3247 3248 3249 3250 3251 3252 3253 3254 3255 3256 3257 3258 3259 3260 3261 3262 3263 3264 3265 3266 3267 3268 3269 3270 3271 3272 3273 3274 3275 3276 3277 3278 3279 3280 3281 3282 3283 3284 3285 3286 3287 3288 3289 3290 3291 3292 3293 3294 3295 3296 3297 3298 3299 3300 3301 3302 3303 3304 3305 3306 3307 3308 3309 3310 3311 3312 3313 3314 3315 3316 3317 3318 3319 3320 3321 3322 3323 3324 3325 3326 3327 3328 3329 3330 3331 3332 3333 3334 3335 3336 3337 3338 3339 3340 3341 3342 3343 3344 3345 3346 3347 3348 3349 3350 3351 3352 3353 3354 3355 3356 3357 3358 3359 3360 3361 3362 3363 3364 3365 3366 3367 3368 3369 3370 3371 3372 3373 3374 3375 3376 3377 3378 3379 3380 3381 3382 3383 3384 3385 3386 3387 3388 3389 3390 3391 3392 3393 3394 3395 3396 3397 3398 3399 3400 3401 3402 3403 3404 3405 3406 3407 3408 3409 3410 3411 3412 3413 3414 3415 3416 3417 3418 3419 3420 3421 3422 3423 3424 3425 3426 3427 3428 3429 3430 3431 3432 3433 3434 3435 3436 3437 3438 3439 3440 3441 3442 3443 3444 3445 3446 3447 3448 3449 3450 3451 3452 3453 3454 3455 3456 3457 3458 3459 3460 3461 3462 3463 3464 3465 3466 3467 3468 3469 3470 3471 3472 3473 3474 3475 3476 3477 3478 3479 3480 3481 3482 3483 3484 3485 3486 3487 3488 3489 3490 3491 3492 3493 3494 3495 3496 3497 3498 3499 3500 3501 3502 3503 3504 3505 3506 3507 3508 3509 3510 3511 3512 3513 3514 3515 3516 3517 3518 3519 3520 3521 3522 3523 3524 3525 3526 3527 3528 3529 3530 3531 3532 3533 3534 3535 3536 3537 3538 3539 3540 3541 3542 3543 3544 3545 3546 3547 3548 3549 3550 3551 3552 3553 3554 3555 3556 3557 3558 3559 3560 3561 3562 3563 3564 3565 3566 3567 3568 3569 3570 3571 3572 3573 3574 3575 3576 3577 3578 3579 3580 3581 3582 3583 3584 3585 3586 3587 3588 3589 3590 3591 3592 3593 3594 3595 3596 3597 3598 3599 3600 3601 3602 3603 3604 3605 3606 3607 3608 3609 3610 3611 3612 3613 3614 3615 3616 3617 3618 3619 3620 3621 3622 3623 3624 3625 3626 3627 3628 3629 3630 3631 3632 3633 3634 3635 3636 3637 3638 3639 3640 3641 3642 3643 3644 3645 3646 3647 3648 3649 3650 3651 3652 3653 3654 3655 3656 3657 3658 3659 3660 3661 3662 3663 3664 3665 3666 3667 3668 3669 3670 3671 3672 3673 3674 3675 3676 3677 3678 3679 3680 3681 3682 3683 3684 3685 3686 3687 3688 3689 3690 3691 3692 3693 3694 3695 3696 3697 3698 3699 3700 3701 3702 3703 3704 3705 3706 3707 3708 3709 3710 3711 3712 3713 3714 3715 3716 3717 3718 3719 3720 3721 3722 3723 3724 3725 3726 3727 3728 3729 3730 3731 3732 3733 3734 3735 3736 3737 3738 3739 3740 3741 3742 3743 3744 3745 3746 3747 3748 3749 3750 3751 3752 3753 3754 3755 3756 3757 3758 3759 3760 3761 3762 3763 3764 3765 3766 3767 3768 3769 3770 3771 3772 3773 3774 3775 3776 3777 3778 3779 3780 3781 3782 3783 3784 3785 3786 3787 3788 3789 3790 3791 3792 3793 3794 3795 3796 3797 3798 3799 3800 3801 3802 3803 3804 3805 3806 3807 3808 3809 3810 3811 3812 3813 3814 3815 3816 3817 3818 3819 3820 3821 3822 3823 3824 3825 3826 3827 3828 3829 3830 3831 3832 3833 3834 3835 3836 3837 3838 3839 3840 3841 3842 3843 3844 3845 3846 3847 3848 3849 3850 3851 3852 3853 3854 3855 3856 3857 3858 3859 3860 3861 3862 3863 3864 3865 3866 3867 3868 3869 3870 3871 3872 3873 3874 3875 3876 3877 3878 3879 3880 3881 3882 3883 3884 3885 3886 3887 3888 3889 3890 3891 3892 3893 3894 3895 3896 3897 3898 3899 3900 3901 3902 3903 3904 3905 3906 3907 3908 3909 3910 3911 3912 3913 3914 3915 3916 3917 3918 3919 3920 3921 3922 3923 3924 3925 3926 3927 3928 3929 3930 3931 3932 3933 3934 3935 3936 3937 3938 3939 3940 3941 3942 3943 3944 3945 3946 3947 3948 3949 3950 3951 3952 3953 3954 3955 3956 3957 3958 3959 3960 3961 3962 3963 3964 3965 3966 3967 3968 3969 3970 3971 3972 3973 3974 3975 3976 3977 3978 3979 3980 3981 3982 3983 3984 3985 3986 3987 3988 3989 3990 3991 3992 3993 3994 3995 3996 3997 3998 3999 4000 4001 4002 4003 4004 4005 4006 4007 4008 4009 4010 4011 4012 4013 4014 4015 4016 4017 4018 4019 4020 4021 4022 4023 4024 4025 4026 4027 4028 4029 4030 4031 4032 4033 4034 4035 4036 4037 4038 4039 4040 4041 4042 4043 4044 4045 4046 4047 4048 4049 4050 4051 4052 4053 4054 4055 4056 4057 4058 4059 4060 4061 4062 4063 4064 4065 4066 4067 4068 4069 4070 4071 4072 4073 4074 4075 4076 4077 4078 4079 4080 4081 4082 4083 4084 4085 4086 4087 4088 4089 4090 4091 4092 4093 4094 4095 4096 4097 4098 4099 4100 4101 4102 4103 4104 4105 4106 4107 4108 4109 4110 4111 4112 4113 4114 4115 4116 4117 4118 4119 4120 4121 4122 4123 4124 4125 4126 4127 4128 4129 4130 4131 4132 4133 4134 4135 4136 4137 4138 4139 4140 4141 4142 4143 4144 4145 4146 4147 4148 4149 4150 4151 4152 4153 4154 4155 4156 4157 4158 4159 4160 4161 4162 4163 4164 4165 4166 4167 4168 4169 4170 4171 4172 4173 4174 4175 4176 4177 4178 4179 4180 4181 4182 4183 4184 4185 4186 4187 4188 4189 4190 4191 4192 4193 4194 4195 4196 4197 4198 4199 4200 4201 4202 4203 4204 4205 4206 4207 4208 4209 4210 4211 4212 4213 4214 4215 4216 4217 4218 4219 4220 4221 4222 4223 4224 4225 4226 4227 4228 4229 4230 4231 4232 4233 4234 4235 4236 4237 4238 4239 4240 4241 4242 4243 4244 4245 4246 4247 4248 4249 4250 4251 4252 4253 4254 4255 4256 4257 4258 4259 4260 4261 4262 4263 4264 4265 4266 4267 4268 4269 4270 4271 4272 4273 4274 4275 4276 4277 4278 4279 4280 4281 4282 4283 4284 4285 4286 4287 4288 4289 4290 4291 4292 4293 4294 4295 4296 4297 4298 4299 4300 4301 4302 4303 4304 4305 4306 4307 4308 4309 4310 4311 4312 4313 4314 4315 4316 4317 4318 4319 4320 4321 4322 4323 4324 4325 4326 4327 4328 4329 4330 4331 4332 4333 4334 4335 4336 4337 4338 4339 4340 4341 4342 4343 4344 4345 4346 4347 4348 4349 4350 4351 4352 4353 4354 4355 4356 4357 4358 4359 4360 4361 4362 4363 4364 4365 4366 4367 4368 4369 4370 4371 4372 4373 4374 4375 4376 4377 4378 4379 4380 4381 4382 4383 4384 4385 4386 4387 4388 4389 4390 4391 4392 4393 4394 4395 4396 4397 4398 4399 4400 4401 4402 4403 4404 4405 4406 4407 4408 4409 4410 4411 4412 4413 4414 4415 4416 4417 4418 4419 4420 4421 4422 4423 4424 4425 4426 4427 4428 4429 4430 4431 4432 4433 4434 4435 4436 4437 4438 4439 4440 4441 4442 4443 4444 4445 4446 4447 4448 4449 | const sqlite3_api_routines *pApi ){ SQLITE_EXTENSION_INIT2(pApi) return sqlite3Fts3Init(db); } #endif /* ** Allocate an Fts3MultiSegReader for each token in the expression headed ** by pExpr. ** ** An Fts3SegReader object is a cursor that can seek or scan a range of ** entries within a single segment b-tree. An Fts3MultiSegReader uses multiple ** Fts3SegReader objects internally to provide an interface to seek or scan ** within the union of all segments of a b-tree. Hence the name. ** ** If the allocated Fts3MultiSegReader just seeks to a single entry in a ** segment b-tree (if the term is not a prefix or it is a prefix for which ** there exists prefix b-tree of the right length) then it may be traversed ** and merged incrementally. Otherwise, it has to be merged into an in-memory ** doclist and then traversed. */ static void fts3EvalAllocateReaders( Fts3Cursor *pCsr, Fts3Expr *pExpr, int *pnToken, /* OUT: Total number of tokens in phrase. */ int *pnOr, /* OUT: Total number of OR nodes in expr. */ int *pRc ){ if( pExpr && SQLITE_OK==*pRc ){ if( pExpr->eType==FTSQUERY_PHRASE ){ int i; int nToken = pExpr->pPhrase->nToken; *pnToken += nToken; for(i=0; i<nToken; i++){ Fts3PhraseToken *pToken = &pExpr->pPhrase->aToken[i]; int rc = sqlite3Fts3TermSegReaderCursor(pCsr, pToken->z, pToken->n, pToken->isPrefix, &pToken->pSegcsr ); if( rc!=SQLITE_OK ){ *pRc = rc; return; } } assert( pExpr->pPhrase->iDoclistToken==0 ); pExpr->pPhrase->iDoclistToken = -1; }else{ *pnOr += (pExpr->eType==FTSQUERY_OR); fts3EvalAllocateReaders(pCsr, pExpr->pLeft, pnToken, pnOr, pRc); fts3EvalAllocateReaders(pCsr, pExpr->pRight, pnToken, pnOr, pRc); } } } static void fts3EvalPhraseMergeToken( Fts3Table *pTab, Fts3Phrase *p, int iToken, char *pList, int nList ){ assert( iToken!=p->iDoclistToken ); if( pList==0 ){ sqlite3_free(p->doclist.aAll); p->doclist.aAll = 0; p->doclist.nAll = 0; } else if( p->iDoclistToken<0 ){ p->doclist.aAll = pList; p->doclist.nAll = nList; } else if( p->doclist.aAll==0 ){ sqlite3_free(pList); } else { char *pLeft; char *pRight; int nLeft; int nRight; int nDiff; if( p->iDoclistToken<iToken ){ pLeft = p->doclist.aAll; nLeft = p->doclist.nAll; pRight = pList; nRight = nList; nDiff = iToken - p->iDoclistToken; }else{ pRight = p->doclist.aAll; nRight = p->doclist.nAll; pLeft = pList; nLeft = nList; nDiff = p->iDoclistToken - iToken; } fts3DoclistPhraseMerge(pTab->bDescIdx, nDiff, pLeft, nLeft, pRight,&nRight); sqlite3_free(pLeft); p->doclist.aAll = pRight; p->doclist.nAll = nRight; } if( iToken>p->iDoclistToken ) p->iDoclistToken = iToken; } static int fts3EvalPhraseLoad( Fts3Cursor *pCsr, Fts3Phrase *p ){ Fts3Table *pTab = (Fts3Table *)pCsr->base.pVtab; int iToken; int rc = SQLITE_OK; for(iToken=0; rc==SQLITE_OK && iToken<p->nToken; iToken++){ Fts3PhraseToken *pToken = &p->aToken[iToken]; assert( pToken->pDeferred==0 || pToken->pSegcsr==0 ); if( pToken->pSegcsr ){ int nThis = 0; char *pThis = 0; rc = fts3TermSelect(pTab, pToken, p->iColumn, 1, &nThis, &pThis); if( rc==SQLITE_OK ){ fts3EvalPhraseMergeToken(pTab, p, iToken, pThis, nThis); } } assert( pToken->pSegcsr==0 ); } return rc; } static int fts3EvalDeferredPhrase(Fts3Cursor *pCsr, Fts3Phrase *pPhrase){ int iToken; int rc = SQLITE_OK; int nMaxUndeferred = pPhrase->iDoclistToken; char *aPoslist = 0; int nPoslist = 0; int iPrev = -1; assert( pPhrase->doclist.bFreeList==0 ); for(iToken=0; rc==SQLITE_OK && iToken<pPhrase->nToken; iToken++){ Fts3PhraseToken *pToken = &pPhrase->aToken[iToken]; Fts3DeferredToken *pDeferred = pToken->pDeferred; if( pDeferred ){ char *pList; int nList; rc = sqlite3Fts3DeferredTokenList(pDeferred, &pList, &nList); if( rc!=SQLITE_OK ) return rc; if( pList==0 ){ sqlite3_free(aPoslist); pPhrase->doclist.pList = 0; pPhrase->doclist.nList = 0; return SQLITE_OK; }else if( aPoslist==0 ){ aPoslist = pList; nPoslist = nList; }else{ char *aOut = pList; char *p1 = aPoslist; char *p2 = aOut; assert( iPrev>=0 ); fts3PoslistPhraseMerge(&aOut, iToken-iPrev, 0, 1, &p1, &p2); sqlite3_free(aPoslist); aPoslist = pList; nPoslist = aOut - aPoslist; if( nPoslist==0 ){ sqlite3_free(aPoslist); pPhrase->doclist.pList = 0; pPhrase->doclist.nList = 0; return SQLITE_OK; } } iPrev = iToken; } } if( iPrev>=0 ){ if( nMaxUndeferred<0 ){ pPhrase->doclist.pList = aPoslist; pPhrase->doclist.nList = nPoslist; pPhrase->doclist.iDocid = pCsr->iPrevId; pPhrase->doclist.bFreeList = 1; }else{ int nDistance; char *p1; char *p2; char *aOut; if( nMaxUndeferred>iPrev ){ p1 = aPoslist; p2 = pPhrase->doclist.pList; nDistance = nMaxUndeferred - iPrev; }else{ p1 = pPhrase->doclist.pList; p2 = aPoslist; nDistance = iPrev - nMaxUndeferred; } aOut = (char *)sqlite3_malloc(nPoslist+8); if( !aOut ){ sqlite3_free(aPoslist); return SQLITE_NOMEM; } pPhrase->doclist.pList = aOut; if( fts3PoslistPhraseMerge(&aOut, nDistance, 0, 1, &p1, &p2) ){ pPhrase->doclist.bFreeList = 1; pPhrase->doclist.nList = (aOut - pPhrase->doclist.pList); }else{ sqlite3_free(aOut); pPhrase->doclist.pList = 0; pPhrase->doclist.nList = 0; } sqlite3_free(aPoslist); } } return SQLITE_OK; } /* ** This function is called for each Fts3Phrase in a full-text query ** expression to initialize the mechanism for returning rows. Once this ** function has been called successfully on an Fts3Phrase, it may be ** used with fts3EvalPhraseNext() to iterate through the matching docids. */ static int fts3EvalPhraseStart(Fts3Cursor *pCsr, int bOptOk, Fts3Phrase *p){ int rc; Fts3PhraseToken *pFirst = &p->aToken[0]; Fts3Table *pTab = (Fts3Table *)pCsr->base.pVtab; if( pCsr->bDesc==pTab->bDescIdx && bOptOk==1 && p->nToken==1 && pFirst->pSegcsr && pFirst->pSegcsr->bLookup ){ /* Use the incremental approach. */ int iCol = (p->iColumn >= pTab->nColumn ? -1 : p->iColumn); rc = sqlite3Fts3MsrIncrStart( pTab, pFirst->pSegcsr, iCol, pFirst->z, pFirst->n); p->bIncr = 1; }else{ /* Load the full doclist for the phrase into memory. */ rc = fts3EvalPhraseLoad(pCsr, p); p->bIncr = 0; } assert( rc!=SQLITE_OK || p->nToken<1 || p->aToken[0].pSegcsr==0 || p->bIncr ); return rc; } /* ** This function is used to iterate backwards (from the end to start) ** through doclists. */ void sqlite3Fts3DoclistPrev( int bDescIdx, /* True if the doclist is desc */ char *aDoclist, /* Pointer to entire doclist */ int nDoclist, /* Length of aDoclist in bytes */ char **ppIter, /* IN/OUT: Iterator pointer */ sqlite3_int64 *piDocid, /* IN/OUT: Docid pointer */ int *pnList, /* IN/OUT: List length pointer */ u8 *pbEof /* OUT: End-of-file flag */ ){ char *p = *ppIter; assert( nDoclist>0 ); assert( *pbEof==0 ); assert( p || *piDocid==0 ); assert( !p || (p>aDoclist && p<&aDoclist[nDoclist]) ); if( p==0 ){ sqlite3_int64 iDocid = 0; char *pNext = 0; char *pDocid = aDoclist; char *pEnd = &aDoclist[nDoclist]; int iMul = 1; while( pDocid<pEnd ){ sqlite3_int64 iDelta; pDocid += sqlite3Fts3GetVarint(pDocid, &iDelta); iDocid += (iMul * iDelta); pNext = pDocid; fts3PoslistCopy(0, &pDocid); while( pDocid<pEnd && *pDocid==0 ) pDocid++; iMul = (bDescIdx ? -1 : 1); } *pnList = pEnd - pNext; *ppIter = pNext; *piDocid = iDocid; }else{ int iMul = (bDescIdx ? -1 : 1); sqlite3_int64 iDelta; fts3GetReverseVarint(&p, aDoclist, &iDelta); *piDocid -= (iMul * iDelta); if( p==aDoclist ){ *pbEof = 1; }else{ char *pSave = p; fts3ReversePoslist(aDoclist, &p); *pnList = (pSave - p); } *ppIter = p; } } /* ** Attempt to move the phrase iterator to point to the next matching docid. ** If an error occurs, return an SQLite error code. Otherwise, return ** SQLITE_OK. ** ** If there is no "next" entry and no error occurs, then *pbEof is set to ** 1 before returning. Otherwise, if no error occurs and the iterator is ** successfully advanced, *pbEof is set to 0. */ static int fts3EvalPhraseNext( Fts3Cursor *pCsr, Fts3Phrase *p, u8 *pbEof ){ int rc = SQLITE_OK; Fts3Doclist *pDL = &p->doclist; Fts3Table *pTab = (Fts3Table *)pCsr->base.pVtab; if( p->bIncr ){ assert( p->nToken==1 ); assert( pDL->pNextDocid==0 ); rc = sqlite3Fts3MsrIncrNext(pTab, p->aToken[0].pSegcsr, &pDL->iDocid, &pDL->pList, &pDL->nList ); if( rc==SQLITE_OK && !pDL->pList ){ *pbEof = 1; } }else if( pCsr->bDesc!=pTab->bDescIdx && pDL->nAll ){ sqlite3Fts3DoclistPrev(pTab->bDescIdx, pDL->aAll, pDL->nAll, &pDL->pNextDocid, &pDL->iDocid, &pDL->nList, pbEof ); pDL->pList = pDL->pNextDocid; }else{ char *pIter; /* Used to iterate through aAll */ char *pEnd = &pDL->aAll[pDL->nAll]; /* 1 byte past end of aAll */ if( pDL->pNextDocid ){ pIter = pDL->pNextDocid; }else{ pIter = pDL->aAll; } if( pIter>=pEnd ){ /* We have already reached the end of this doclist. EOF. */ *pbEof = 1; }else{ sqlite3_int64 iDelta; pIter += sqlite3Fts3GetVarint(pIter, &iDelta); if( pTab->bDescIdx==0 || pDL->pNextDocid==0 ){ pDL->iDocid += iDelta; }else{ pDL->iDocid -= iDelta; } pDL->pList = pIter; fts3PoslistCopy(0, &pIter); pDL->nList = (pIter - pDL->pList); /* pIter now points just past the 0x00 that terminates the position- ** list for document pDL->iDocid. However, if this position-list was ** edited in place by fts3EvalNearTrim2(), then pIter may not actually ** point to the start of the next docid value. The following line deals ** with this case by advancing pIter past the zero-padding added by ** fts3EvalNearTrim2(). */ while( pIter<pEnd && *pIter==0 ) pIter++; pDL->pNextDocid = pIter; assert( pIter>=&pDL->aAll[pDL->nAll] || *pIter ); *pbEof = 0; } } return rc; } static void fts3EvalStartReaders( Fts3Cursor *pCsr, Fts3Expr *pExpr, int bOptOk, int *pRc ){ if( pExpr && SQLITE_OK==*pRc ){ if( pExpr->eType==FTSQUERY_PHRASE ){ int i; int nToken = pExpr->pPhrase->nToken; for(i=0; i<nToken; i++){ if( pExpr->pPhrase->aToken[i].pDeferred==0 ) break; } pExpr->bDeferred = (i==nToken); *pRc = fts3EvalPhraseStart(pCsr, bOptOk, pExpr->pPhrase); }else{ fts3EvalStartReaders(pCsr, pExpr->pLeft, bOptOk, pRc); fts3EvalStartReaders(pCsr, pExpr->pRight, bOptOk, pRc); pExpr->bDeferred = (pExpr->pLeft->bDeferred && pExpr->pRight->bDeferred); } } } typedef struct Fts3TokenAndCost Fts3TokenAndCost; struct Fts3TokenAndCost { Fts3Phrase *pPhrase; /* The phrase the token belongs to */ int iToken; /* Position of token in phrase */ Fts3PhraseToken *pToken; /* The token itself */ Fts3Expr *pRoot; int nOvfl; int iCol; /* The column the token must match */ }; static void fts3EvalTokenCosts( Fts3Cursor *pCsr, Fts3Expr *pRoot, Fts3Expr *pExpr, Fts3TokenAndCost **ppTC, Fts3Expr ***ppOr, int *pRc ){ if( *pRc==SQLITE_OK && pExpr ){ if( pExpr->eType==FTSQUERY_PHRASE ){ Fts3Phrase *pPhrase = pExpr->pPhrase; int i; for(i=0; *pRc==SQLITE_OK && i<pPhrase->nToken; i++){ Fts3TokenAndCost *pTC = (*ppTC)++; pTC->pPhrase = pPhrase; pTC->iToken = i; pTC->pRoot = pRoot; pTC->pToken = &pPhrase->aToken[i]; pTC->iCol = pPhrase->iColumn; *pRc = sqlite3Fts3MsrOvfl(pCsr, pTC->pToken->pSegcsr, &pTC->nOvfl); } }else if( pExpr->eType!=FTSQUERY_NOT ){ if( pExpr->eType==FTSQUERY_OR ){ pRoot = pExpr->pLeft; **ppOr = pRoot; (*ppOr)++; } fts3EvalTokenCosts(pCsr, pRoot, pExpr->pLeft, ppTC, ppOr, pRc); if( pExpr->eType==FTSQUERY_OR ){ pRoot = pExpr->pRight; **ppOr = pRoot; (*ppOr)++; } fts3EvalTokenCosts(pCsr, pRoot, pExpr->pRight, ppTC, ppOr, pRc); } } } static int fts3EvalAverageDocsize(Fts3Cursor *pCsr, int *pnPage){ if( pCsr->nRowAvg==0 ){ /* The average document size, which is required to calculate the cost ** of each doclist, has not yet been determined. Read the required ** data from the %_stat table to calculate it. ** ** Entry 0 of the %_stat table is a blob containing (nCol+1) FTS3 ** varints, where nCol is the number of columns in the FTS3 table. ** The first varint is the number of documents currently stored in ** the table. The following nCol varints contain the total amount of ** data stored in all rows of each column of the table, from left ** to right. */ int rc; Fts3Table *p = (Fts3Table*)pCsr->base.pVtab; sqlite3_stmt *pStmt; sqlite3_int64 nDoc = 0; sqlite3_int64 nByte = 0; const char *pEnd; const char *a; rc = sqlite3Fts3SelectDoctotal(p, &pStmt); if( rc!=SQLITE_OK ) return rc; a = sqlite3_column_blob(pStmt, 0); assert( a ); pEnd = &a[sqlite3_column_bytes(pStmt, 0)]; a += sqlite3Fts3GetVarint(a, &nDoc); while( a<pEnd ){ a += sqlite3Fts3GetVarint(a, &nByte); } if( nDoc==0 || nByte==0 ){ sqlite3_reset(pStmt); return SQLITE_CORRUPT_VTAB; } pCsr->nDoc = nDoc; pCsr->nRowAvg = (int)(((nByte / nDoc) + p->nPgsz) / p->nPgsz); assert( pCsr->nRowAvg>0 ); rc = sqlite3_reset(pStmt); if( rc!=SQLITE_OK ) return rc; } *pnPage = pCsr->nRowAvg; return SQLITE_OK; } static int fts3EvalSelectDeferred( Fts3Cursor *pCsr, Fts3Expr *pRoot, Fts3TokenAndCost *aTC, int nTC ){ int nDocSize = 0; int nDocEst = 0; int rc = SQLITE_OK; Fts3Table *pTab = (Fts3Table *)pCsr->base.pVtab; int ii; int nOvfl = 0; int nTerm = 0; for(ii=0; ii<nTC; ii++){ if( aTC[ii].pRoot==pRoot ){ nOvfl += aTC[ii].nOvfl; nTerm++; } } if( nOvfl==0 || nTerm<2 ) return SQLITE_OK; rc = fts3EvalAverageDocsize(pCsr, &nDocSize); for(ii=0; ii<nTerm && rc==SQLITE_OK; ii++){ int jj; Fts3TokenAndCost *pTC = 0; for(jj=0; jj<nTC; jj++){ if( aTC[jj].pToken && aTC[jj].pRoot==pRoot && (!pTC || aTC[jj].nOvfl<pTC->nOvfl) ){ pTC = &aTC[jj]; } } assert( pTC ); /* At this point pTC points to the cheapest remaining token. */ if( ii==0 ){ if( pTC->nOvfl ){ nDocEst = (pTC->nOvfl * pTab->nPgsz + pTab->nPgsz) / 10; }else{ Fts3PhraseToken *pToken = pTC->pToken; int nList = 0; char *pList = 0; rc = fts3TermSelect(pTab, pToken, pTC->iCol, 1, &nList, &pList); assert( rc==SQLITE_OK || pList==0 ); if( rc==SQLITE_OK ){ nDocEst = fts3DoclistCountDocids(1, pList, nList); fts3EvalPhraseMergeToken(pTab, pTC->pPhrase, pTC->iToken,pList,nList); } } }else{ if( pTC->nOvfl>=(nDocEst*nDocSize) ){ Fts3PhraseToken *pToken = pTC->pToken; rc = sqlite3Fts3DeferToken(pCsr, pToken, pTC->iCol); fts3SegReaderCursorFree(pToken->pSegcsr); pToken->pSegcsr = 0; } nDocEst = 1 + (nDocEst/4); } pTC->pToken = 0; } return rc; } int sqlite3Fts3EvalStart(Fts3Cursor *pCsr, Fts3Expr *pExpr, int bOptOk){ Fts3Table *pTab = (Fts3Table *)pCsr->base.pVtab; int rc = SQLITE_OK; int nToken = 0; int nOr = 0; /* Allocate a MultiSegReader for each token in the expression. */ fts3EvalAllocateReaders(pCsr, pExpr, &nToken, &nOr, &rc); /* Call fts3EvalPhraseStart() on all phrases in the expression. TODO: ** This call will eventually also be responsible for determining which ** tokens are 'deferred' until the document text is loaded into memory. ** ** Each token in each phrase is dealt with using one of the following ** three strategies: ** ** 1. Entire doclist loaded into memory as part of the ** fts3EvalStartReaders() call. ** ** 2. Doclist loaded into memory incrementally, as part of each ** sqlite3Fts3EvalNext() call. ** ** 3. Token doclist is never loaded. Instead, documents are loaded into ** memory and scanned for the token as part of the sqlite3Fts3EvalNext() ** call. This is known as a "deferred" token. */ /* If bOptOk is true, check if there are any tokens that should be deferred. */ if( rc==SQLITE_OK && bOptOk && nToken>1 && pTab->bHasStat ){ Fts3TokenAndCost *aTC; Fts3Expr **apOr; aTC = (Fts3TokenAndCost *)sqlite3_malloc( sizeof(Fts3TokenAndCost) * nToken + sizeof(Fts3Expr *) * nOr * 2 ); apOr = (Fts3Expr **)&aTC[nToken]; if( !aTC ){ rc = SQLITE_NOMEM; }else{ int ii; Fts3TokenAndCost *pTC = aTC; Fts3Expr **ppOr = apOr; fts3EvalTokenCosts(pCsr, 0, pExpr, &pTC, &ppOr, &rc); nToken = pTC-aTC; nOr = ppOr-apOr; if( rc==SQLITE_OK ){ rc = fts3EvalSelectDeferred(pCsr, 0, aTC, nToken); for(ii=0; rc==SQLITE_OK && ii<nOr; ii++){ rc = fts3EvalSelectDeferred(pCsr, apOr[ii], aTC, nToken); } } sqlite3_free(aTC); } } fts3EvalStartReaders(pCsr, pExpr, bOptOk, &rc); return rc; } static void fts3EvalZeroPoslist(Fts3Phrase *pPhrase){ if( pPhrase->doclist.bFreeList ){ sqlite3_free(pPhrase->doclist.pList); } pPhrase->doclist.pList = 0; pPhrase->doclist.nList = 0; pPhrase->doclist.bFreeList = 0; } static int fts3EvalNearTrim2( int nNear, char *aTmp, /* Temporary space to use */ char **paPoslist, /* IN/OUT: Position list */ int *pnToken, /* IN/OUT: Tokens in phrase of *paPoslist */ Fts3Phrase *pPhrase /* The phrase object to trim the doclist of */ ){ int nParam1 = nNear + pPhrase->nToken; int nParam2 = nNear + *pnToken; int nNew; char *p2; char *pOut; int res; assert( pPhrase->doclist.pList ); p2 = pOut = pPhrase->doclist.pList; res = fts3PoslistNearMerge( &pOut, aTmp, nParam1, nParam2, paPoslist, &p2 ); if( res ){ nNew = (pOut - pPhrase->doclist.pList) - 1; assert( pPhrase->doclist.pList[nNew]=='\0' ); assert( nNew<=pPhrase->doclist.nList && nNew>0 ); memset(&pPhrase->doclist.pList[nNew], 0, pPhrase->doclist.nList - nNew); pPhrase->doclist.nList = nNew; *paPoslist = pPhrase->doclist.pList; *pnToken = pPhrase->nToken; } return res; } static int fts3EvalNearTest(Fts3Expr *pExpr, int *pRc){ int res = 1; /* The following block runs if pExpr is the root of a NEAR query. ** For example, the query: ** ** "w" NEAR "x" NEAR "y" NEAR "z" ** ** which is represented in tree form as: ** ** | ** +--NEAR--+ <-- root of NEAR query ** | | ** +--NEAR--+ "z" ** | | ** +--NEAR--+ "y" ** | | ** "w" "x" ** ** The right-hand child of a NEAR node is always a phrase. The ** left-hand child may be either a phrase or a NEAR node. There are ** no exceptions to this. */ if( *pRc==SQLITE_OK && pExpr->eType==FTSQUERY_NEAR && pExpr->bEof==0 && (pExpr->pParent==0 || pExpr->pParent->eType!=FTSQUERY_NEAR) ){ Fts3Expr *p; int nTmp = 0; /* Bytes of temp space */ char *aTmp; /* Temp space for PoslistNearMerge() */ /* Allocate temporary working space. */ for(p=pExpr; p->pLeft; p=p->pLeft){ nTmp += p->pRight->pPhrase->doclist.nList; } nTmp += p->pPhrase->doclist.nList; aTmp = sqlite3_malloc(nTmp*2); if( !aTmp ){ *pRc = SQLITE_NOMEM; res = 0; }else{ char *aPoslist = p->pPhrase->doclist.pList; int nToken = p->pPhrase->nToken; for(p=p->pParent;res && p && p->eType==FTSQUERY_NEAR; p=p->pParent){ Fts3Phrase *pPhrase = p->pRight->pPhrase; int nNear = p->nNear; res = fts3EvalNearTrim2(nNear, aTmp, &aPoslist, &nToken, pPhrase); } aPoslist = pExpr->pRight->pPhrase->doclist.pList; nToken = pExpr->pRight->pPhrase->nToken; for(p=pExpr->pLeft; p && res; p=p->pLeft){ int nNear = p->pParent->nNear; Fts3Phrase *pPhrase = ( p->eType==FTSQUERY_NEAR ? p->pRight->pPhrase : p->pPhrase ); res = fts3EvalNearTrim2(nNear, aTmp, &aPoslist, &nToken, pPhrase); } } sqlite3_free(aTmp); } return res; } /* ** This macro is used by the fts3EvalNext() function. The two arguments are ** 64-bit docid values. If the current query is "ORDER BY docid ASC", then ** the macro returns (i1 - i2). Or if it is "ORDER BY docid DESC", then ** it returns (i2 - i1). This allows the same code to be used for merging ** doclists in ascending or descending order. */ #define DOCID_CMP(i1, i2) ((pCsr->bDesc?-1:1) * (i1-i2)) static void fts3EvalNext( Fts3Cursor *pCsr, Fts3Expr *pExpr, int *pRc ){ if( *pRc==SQLITE_OK ){ assert( pExpr->bEof==0 ); pExpr->bStart = 1; switch( pExpr->eType ){ case FTSQUERY_NEAR: case FTSQUERY_AND: { Fts3Expr *pLeft = pExpr->pLeft; Fts3Expr *pRight = pExpr->pRight; assert( !pLeft->bDeferred || !pRight->bDeferred ); if( pLeft->bDeferred ){ fts3EvalNext(pCsr, pRight, pRc); pExpr->iDocid = pRight->iDocid; pExpr->bEof = pRight->bEof; }else if( pRight->bDeferred ){ fts3EvalNext(pCsr, pLeft, pRc); pExpr->iDocid = pLeft->iDocid; pExpr->bEof = pLeft->bEof; }else{ fts3EvalNext(pCsr, pLeft, pRc); fts3EvalNext(pCsr, pRight, pRc); while( !pLeft->bEof && !pRight->bEof && *pRc==SQLITE_OK ){ sqlite3_int64 iDiff = DOCID_CMP(pLeft->iDocid, pRight->iDocid); if( iDiff==0 ) break; if( iDiff<0 ){ fts3EvalNext(pCsr, pLeft, pRc); }else{ fts3EvalNext(pCsr, pRight, pRc); } } pExpr->iDocid = pLeft->iDocid; pExpr->bEof = (pLeft->bEof || pRight->bEof); } break; } case FTSQUERY_OR: { Fts3Expr *pLeft = pExpr->pLeft; Fts3Expr *pRight = pExpr->pRight; sqlite3_int64 iCmp = DOCID_CMP(pLeft->iDocid, pRight->iDocid); assert( pLeft->bStart || pLeft->iDocid==pRight->iDocid ); assert( pRight->bStart || pLeft->iDocid==pRight->iDocid ); if( pRight->bEof || (pLeft->bEof==0 && iCmp<0) ){ fts3EvalNext(pCsr, pLeft, pRc); }else if( pLeft->bEof || (pRight->bEof==0 && iCmp>0) ){ fts3EvalNext(pCsr, pRight, pRc); }else{ fts3EvalNext(pCsr, pLeft, pRc); fts3EvalNext(pCsr, pRight, pRc); } pExpr->bEof = (pLeft->bEof && pRight->bEof); iCmp = DOCID_CMP(pLeft->iDocid, pRight->iDocid); if( pRight->bEof || (pLeft->bEof==0 && iCmp<0) ){ pExpr->iDocid = pLeft->iDocid; }else{ pExpr->iDocid = pRight->iDocid; } break; } case FTSQUERY_NOT: { Fts3Expr *pLeft = pExpr->pLeft; Fts3Expr *pRight = pExpr->pRight; if( pRight->bStart==0 ){ fts3EvalNext(pCsr, pRight, pRc); assert( *pRc!=SQLITE_OK || pRight->bStart ); } fts3EvalNext(pCsr, pLeft, pRc); if( pLeft->bEof==0 ){ while( !*pRc && !pRight->bEof && DOCID_CMP(pLeft->iDocid, pRight->iDocid)>0 ){ fts3EvalNext(pCsr, pRight, pRc); } } pExpr->iDocid = pLeft->iDocid; pExpr->bEof = pLeft->bEof; break; } default: { Fts3Phrase *pPhrase = pExpr->pPhrase; fts3EvalZeroPoslist(pPhrase); *pRc = fts3EvalPhraseNext(pCsr, pPhrase, &pExpr->bEof); pExpr->iDocid = pPhrase->doclist.iDocid; break; } } } } static int fts3EvalDeferredTest(Fts3Cursor *pCsr, Fts3Expr *pExpr, int *pRc){ int bHit = 1; if( *pRc==SQLITE_OK ){ switch( pExpr->eType ){ case FTSQUERY_NEAR: case FTSQUERY_AND: bHit = ( fts3EvalDeferredTest(pCsr, pExpr->pLeft, pRc) && fts3EvalDeferredTest(pCsr, pExpr->pRight, pRc) && fts3EvalNearTest(pExpr, pRc) ); /* If the NEAR expression does not match any rows, zero the doclist for ** all phrases involved in the NEAR. This is because the snippet(), ** offsets() and matchinfo() functions are not supposed to recognize ** any instances of phrases that are part of unmatched NEAR queries. ** For example if this expression: ** ** ... MATCH 'a OR (b NEAR c)' ** ** is matched against a row containing: ** ** 'a b d e' ** ** then any snippet() should ony highlight the "a" term, not the "b" ** (as "b" is part of a non-matching NEAR clause). */ if( bHit==0 && pExpr->eType==FTSQUERY_NEAR && (pExpr->pParent==0 || pExpr->pParent->eType!=FTSQUERY_NEAR) ){ Fts3Expr *p; for(p=pExpr; p->pPhrase==0; p=p->pLeft){ if( p->pRight->iDocid==pCsr->iPrevId ){ fts3EvalZeroPoslist(p->pRight->pPhrase); } } if( p->iDocid==pCsr->iPrevId ){ fts3EvalZeroPoslist(p->pPhrase); } } break; case FTSQUERY_OR: { int bHit1 = fts3EvalDeferredTest(pCsr, pExpr->pLeft, pRc); int bHit2 = fts3EvalDeferredTest(pCsr, pExpr->pRight, pRc); bHit = bHit1 || bHit2; break; } case FTSQUERY_NOT: bHit = ( fts3EvalDeferredTest(pCsr, pExpr->pLeft, pRc) && !fts3EvalDeferredTest(pCsr, pExpr->pRight, pRc) ); break; default: { if( pCsr->pDeferred && (pExpr->iDocid==pCsr->iPrevId || pExpr->bDeferred) ){ Fts3Phrase *pPhrase = pExpr->pPhrase; assert( pExpr->bDeferred || pPhrase->doclist.bFreeList==0 ); if( pExpr->bDeferred ){ fts3EvalZeroPoslist(pPhrase); } *pRc = fts3EvalDeferredPhrase(pCsr, pPhrase); bHit = (pPhrase->doclist.pList!=0); pExpr->iDocid = pCsr->iPrevId; }else{ bHit = (pExpr->bEof==0 && pExpr->iDocid==pCsr->iPrevId); } break; } } } return bHit; } /* ** Return 1 if both of the following are true: ** ** 1. *pRc is SQLITE_OK when this function returns, and ** ** 2. After scanning the current FTS table row for the deferred tokens, ** it is determined that the row does not match the query. ** ** Or, if no error occurs and it seems the current row does match the FTS ** query, return 0. */ static int fts3EvalLoadDeferred(Fts3Cursor *pCsr, int *pRc){ int rc = *pRc; int bMiss = 0; if( rc==SQLITE_OK ){ if( pCsr->pDeferred ){ rc = fts3CursorSeek(0, pCsr); if( rc==SQLITE_OK ){ rc = sqlite3Fts3CacheDeferredDoclists(pCsr); } } bMiss = (0==fts3EvalDeferredTest(pCsr, pCsr->pExpr, &rc)); sqlite3Fts3FreeDeferredDoclists(pCsr); *pRc = rc; } return (rc==SQLITE_OK && bMiss); } /* ** Advance to the next document that matches the FTS expression in ** Fts3Cursor.pExpr. */ int sqlite3Fts3EvalNext(Fts3Cursor *pCsr){ int rc = SQLITE_OK; /* Return Code */ Fts3Expr *pExpr = pCsr->pExpr; assert( pCsr->isEof==0 ); if( pExpr==0 ){ pCsr->isEof = 1; }else{ do { if( pCsr->isRequireSeek==0 ){ sqlite3_reset(pCsr->pStmt); } assert( sqlite3_data_count(pCsr->pStmt)==0 ); fts3EvalNext(pCsr, pExpr, &rc); pCsr->isEof = pExpr->bEof; pCsr->isRequireSeek = 1; pCsr->isMatchinfoNeeded = 1; pCsr->iPrevId = pExpr->iDocid; }while( pCsr->isEof==0 && fts3EvalLoadDeferred(pCsr, &rc) ); } return rc; } /* ** Restart interation for expression pExpr so that the next call to ** sqlite3Fts3EvalNext() visits the first row. Do not allow incremental ** loading or merging of phrase doclists for this iteration. ** ** If *pRc is other than SQLITE_OK when this function is called, it is ** a no-op. If an error occurs within this function, *pRc is set to an ** SQLite error code before returning. */ static void fts3EvalRestart( Fts3Cursor *pCsr, Fts3Expr *pExpr, int *pRc ){ if( pExpr && *pRc==SQLITE_OK ){ Fts3Phrase *pPhrase = pExpr->pPhrase; if( pPhrase ){ fts3EvalZeroPoslist(pPhrase); if( pPhrase->bIncr ){ assert( pPhrase->nToken==1 ); assert( pPhrase->aToken[0].pSegcsr ); sqlite3Fts3MsrIncrRestart(pPhrase->aToken[0].pSegcsr); *pRc = fts3EvalPhraseStart(pCsr, 0, pPhrase); } pPhrase->doclist.pNextDocid = 0; pPhrase->doclist.iDocid = 0; } pExpr->iDocid = 0; pExpr->bEof = 0; pExpr->bStart = 0; fts3EvalRestart(pCsr, pExpr->pLeft, pRc); fts3EvalRestart(pCsr, pExpr->pRight, pRc); } } /* ** After allocating the Fts3Expr.aMI[] array for each phrase in the ** expression rooted at pExpr, the cursor iterates through all rows matched ** by pExpr, calling this function for each row. This function increments ** the values in Fts3Expr.aMI[] according to the position-list currently ** found in Fts3Expr.pPhrase->doclist.pList for each of the phrase ** expression nodes. */ static void fts3EvalUpdateCounts(Fts3Expr *pExpr){ if( pExpr ){ Fts3Phrase *pPhrase = pExpr->pPhrase; if( pPhrase && pPhrase->doclist.pList ){ int iCol = 0; char *p = pPhrase->doclist.pList; assert( *p ); while( 1 ){ u8 c = 0; int iCnt = 0; while( 0xFE & (*p | c) ){ if( (c&0x80)==0 ) iCnt++; c = *p++ & 0x80; } /* aMI[iCol*3 + 1] = Number of occurrences ** aMI[iCol*3 + 2] = Number of rows containing at least one instance */ pExpr->aMI[iCol*3 + 1] += iCnt; pExpr->aMI[iCol*3 + 2] += (iCnt>0); if( *p==0x00 ) break; p++; p += sqlite3Fts3GetVarint32(p, &iCol); } } fts3EvalUpdateCounts(pExpr->pLeft); fts3EvalUpdateCounts(pExpr->pRight); } } /* ** Expression pExpr must be of type FTSQUERY_PHRASE. ** ** If it is not already allocated and populated, this function allocates and ** populates the Fts3Expr.aMI[] array for expression pExpr. If pExpr is part ** of a NEAR expression, then it also allocates and populates the same array ** for all other phrases that are part of the NEAR expression. ** ** SQLITE_OK is returned if the aMI[] array is successfully allocated and ** populated. Otherwise, if an error occurs, an SQLite error code is returned. */ static int fts3EvalGatherStats( Fts3Cursor *pCsr, /* Cursor object */ Fts3Expr *pExpr /* FTSQUERY_PHRASE expression */ ){ int rc = SQLITE_OK; /* Return code */ assert( pExpr->eType==FTSQUERY_PHRASE ); if( pExpr->aMI==0 ){ Fts3Table *pTab = (Fts3Table *)pCsr->base.pVtab; Fts3Expr *pRoot; /* Root of NEAR expression */ Fts3Expr *p; /* Iterator used for several purposes */ sqlite3_int64 iPrevId = pCsr->iPrevId; sqlite3_int64 iDocid; u8 bEof; /* Find the root of the NEAR expression */ pRoot = pExpr; while( pRoot->pParent && pRoot->pParent->eType==FTSQUERY_NEAR ){ pRoot = pRoot->pParent; } iDocid = pRoot->iDocid; bEof = pRoot->bEof; assert( pRoot->bStart ); /* Allocate space for the aMSI[] array of each FTSQUERY_PHRASE node */ for(p=pRoot; p; p=p->pLeft){ Fts3Expr *pE = (p->eType==FTSQUERY_PHRASE?p:p->pRight); assert( pE->aMI==0 ); pE->aMI = (u32 *)sqlite3_malloc(pTab->nColumn * 3 * sizeof(u32)); if( !pE->aMI ) return SQLITE_NOMEM; memset(pE->aMI, 0, pTab->nColumn * 3 * sizeof(u32)); } fts3EvalRestart(pCsr, pRoot, &rc); while( pCsr->isEof==0 && rc==SQLITE_OK ){ do { /* Ensure the %_content statement is reset. */ if( pCsr->isRequireSeek==0 ) sqlite3_reset(pCsr->pStmt); assert( sqlite3_data_count(pCsr->pStmt)==0 ); /* Advance to the next document */ fts3EvalNext(pCsr, pRoot, &rc); pCsr->isEof = pRoot->bEof; pCsr->isRequireSeek = 1; pCsr->isMatchinfoNeeded = 1; pCsr->iPrevId = pRoot->iDocid; }while( pCsr->isEof==0 && pRoot->eType==FTSQUERY_NEAR && fts3EvalLoadDeferred(pCsr, &rc) ); if( rc==SQLITE_OK && pCsr->isEof==0 ){ fts3EvalUpdateCounts(pRoot); } } pCsr->isEof = 0; pCsr->iPrevId = iPrevId; if( bEof ){ pRoot->bEof = bEof; }else{ /* Caution: pRoot may iterate through docids in ascending or descending ** order. For this reason, even though it seems more defensive, the ** do loop can not be written: ** ** do {...} while( pRoot->iDocid<iDocid && rc==SQLITE_OK ); */ fts3EvalRestart(pCsr, pRoot, &rc); do { fts3EvalNext(pCsr, pRoot, &rc); assert( pRoot->bEof==0 ); }while( pRoot->iDocid!=iDocid && rc==SQLITE_OK ); fts3EvalLoadDeferred(pCsr, &rc); } } return rc; } /* ** This function is used by the matchinfo() module to query a phrase ** expression node for the following information: ** ** 1. The total number of occurrences of the phrase in each column of ** the FTS table (considering all rows), and ** ** 2. For each column, the number of rows in the table for which the ** column contains at least one instance of the phrase. ** ** If no error occurs, SQLITE_OK is returned and the values for each column ** written into the array aiOut as follows: ** ** aiOut[iCol*3 + 1] = Number of occurrences ** aiOut[iCol*3 + 2] = Number of rows containing at least one instance ** ** Caveats: ** ** * If a phrase consists entirely of deferred tokens, then all output ** values are set to the number of documents in the table. In other ** words we assume that very common tokens occur exactly once in each ** column of each row of the table. ** ** * If a phrase contains some deferred tokens (and some non-deferred ** tokens), count the potential occurrence identified by considering ** the non-deferred tokens instead of actual phrase occurrences. ** ** * If the phrase is part of a NEAR expression, then only phrase instances ** that meet the NEAR constraint are included in the counts. */ int sqlite3Fts3EvalPhraseStats( Fts3Cursor *pCsr, /* FTS cursor handle */ Fts3Expr *pExpr, /* Phrase expression */ u32 *aiOut /* Array to write results into (see above) */ ){ Fts3Table *pTab = (Fts3Table *)pCsr->base.pVtab; int rc = SQLITE_OK; int iCol; if( pExpr->bDeferred && pExpr->pParent->eType!=FTSQUERY_NEAR ){ assert( pCsr->nDoc>0 ); for(iCol=0; iCol<pTab->nColumn; iCol++){ aiOut[iCol*3 + 1] = pCsr->nDoc; aiOut[iCol*3 + 2] = pCsr->nDoc; } }else{ rc = fts3EvalGatherStats(pCsr, pExpr); if( rc==SQLITE_OK ){ assert( pExpr->aMI ); for(iCol=0; iCol<pTab->nColumn; iCol++){ aiOut[iCol*3 + 1] = pExpr->aMI[iCol*3 + 1]; aiOut[iCol*3 + 2] = pExpr->aMI[iCol*3 + 2]; } } } return rc; } /* ** The expression pExpr passed as the second argument to this function ** must be of type FTSQUERY_PHRASE. ** ** The returned value is either NULL or a pointer to a buffer containing ** a position-list indicating the occurrences of the phrase in column iCol ** of the current row. ** ** More specifically, the returned buffer contains 1 varint for each ** occurence of the phrase in the column, stored using the normal (delta+2) ** compression and is terminated by either an 0x01 or 0x00 byte. For example, ** if the requested column contains "a b X c d X X" and the position-list ** for 'X' is requested, the buffer returned may contain: ** ** 0x04 0x05 0x03 0x01 or 0x04 0x05 0x03 0x00 ** ** This function works regardless of whether or not the phrase is deferred, ** incremental, or neither. */ char *sqlite3Fts3EvalPhrasePoslist( Fts3Cursor *pCsr, /* FTS3 cursor object */ Fts3Expr *pExpr, /* Phrase to return doclist for */ int iCol /* Column to return position list for */ ){ Fts3Phrase *pPhrase = pExpr->pPhrase; Fts3Table *pTab = (Fts3Table *)pCsr->base.pVtab; char *pIter = pPhrase->doclist.pList; int iThis; assert( iCol>=0 && iCol<pTab->nColumn ); if( !pIter || pExpr->bEof || pExpr->iDocid!=pCsr->iPrevId || (pPhrase->iColumn<pTab->nColumn && pPhrase->iColumn!=iCol) ){ return 0; } assert( pPhrase->doclist.nList>0 ); if( *pIter==0x01 ){ pIter++; pIter += sqlite3Fts3GetVarint32(pIter, &iThis); }else{ iThis = 0; } while( iThis<iCol ){ fts3ColumnlistCopy(0, &pIter); if( *pIter==0x00 ) return 0; pIter++; pIter += sqlite3Fts3GetVarint32(pIter, &iThis); } return ((iCol==iThis)?pIter:0); } /* ** Free all components of the Fts3Phrase structure that were allocated by ** the eval module. Specifically, this means to free: ** ** * the contents of pPhrase->doclist, and ** * any Fts3MultiSegReader objects held by phrase tokens. */ void sqlite3Fts3EvalPhraseCleanup(Fts3Phrase *pPhrase){ if( pPhrase ){ int i; sqlite3_free(pPhrase->doclist.aAll); fts3EvalZeroPoslist(pPhrase); memset(&pPhrase->doclist, 0, sizeof(Fts3Doclist)); for(i=0; i<pPhrase->nToken; i++){ fts3SegReaderCursorFree(pPhrase->aToken[i].pSegcsr); pPhrase->aToken[i].pSegcsr = 0; } } } #endif |
Changes to ext/fts3/fts3Int.h.
1 2 3 4 5 6 7 8 9 10 11 12 13 | /* ** 2009 Nov 12 ** ** The author disclaims copyright to this source code. In place of ** a legal notice, here is a blessing: ** ** May you do good and not evil. ** May you find forgiveness for yourself and forgive others. ** May you share freely, never taking more than you give. ** ****************************************************************************** ** */ | < > > > > > > > > > > | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 | /* ** 2009 Nov 12 ** ** The author disclaims copyright to this source code. In place of ** a legal notice, here is a blessing: ** ** May you do good and not evil. ** May you find forgiveness for yourself and forgive others. ** May you share freely, never taking more than you give. ** ****************************************************************************** ** */ #ifndef _FTSINT_H #define _FTSINT_H #if !defined(NDEBUG) && !defined(SQLITE_DEBUG) # define NDEBUG 1 #endif /* ** FTS4 is really an extension for FTS3. It is enabled using the ** SQLITE_ENABLE_FTS3 macro. But to avoid confusion we also all ** the SQLITE_ENABLE_FTS4 macro to serve as an alisse for SQLITE_ENABLE_FTS3. */ #if defined(SQLITE_ENABLE_FTS4) && !defined(SQLITE_ENABLE_FTS3) # define SQLITE_ENABLE_FTS3 #endif #ifdef SQLITE_ENABLE_FTS3 #include "sqlite3.h" #include "fts3_tokenizer.h" #include "fts3_hash.h" /* ** This constant controls how often segments are merged. Once there are ** FTS3_MERGE_COUNT segments of level N, they are merged into a single |
︙ | ︙ | |||
43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 | /* ** Macro to return the number of elements in an array. SQLite has a ** similar macro called ArraySize(). Use a different name to avoid ** a collision when building an amalgamation with built-in FTS3. */ #define SizeofArray(X) ((int)(sizeof(X)/sizeof(X[0]))) /* ** Maximum length of a varint encoded integer. The varint format is different ** from that used by SQLite, so the maximum length is 10, not 9. */ #define FTS3_VARINT_MAX 10 /* ** The testcase() macro is only used by the amalgamation. If undefined, ** make it a no-op. */ #ifndef testcase # define testcase(X) #endif | > > > > > > > > > > > > > > > > > > > > > > > | 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 | /* ** Macro to return the number of elements in an array. SQLite has a ** similar macro called ArraySize(). Use a different name to avoid ** a collision when building an amalgamation with built-in FTS3. */ #define SizeofArray(X) ((int)(sizeof(X)/sizeof(X[0]))) #ifndef MIN # define MIN(x,y) ((x)<(y)?(x):(y)) #endif /* ** Maximum length of a varint encoded integer. The varint format is different ** from that used by SQLite, so the maximum length is 10, not 9. */ #define FTS3_VARINT_MAX 10 /* ** FTS4 virtual tables may maintain multiple indexes - one index of all terms ** in the document set and zero or more prefix indexes. All indexes are stored ** as one or more b+-trees in the %_segments and %_segdir tables. ** ** It is possible to determine which index a b+-tree belongs to based on the ** value stored in the "%_segdir.level" column. Given this value L, the index ** that the b+-tree belongs to is (L<<10). In other words, all b+-trees with ** level values between 0 and 1023 (inclusive) belong to index 0, all levels ** between 1024 and 2047 to index 1, and so on. ** ** It is considered impossible for an index to use more than 1024 levels. In ** theory though this may happen, but only after at least ** (FTS3_MERGE_COUNT^1024) separate flushes of the pending-terms tables. */ #define FTS3_SEGDIR_MAXLEVEL 1024 #define FTS3_SEGDIR_MAXLEVEL_STR "1024" /* ** The testcase() macro is only used by the amalgamation. If undefined, ** make it a no-op. */ #ifndef testcase # define testcase(X) #endif |
︙ | ︙ | |||
120 121 122 123 124 125 126 127 128 129 | typedef struct Fts3Table Fts3Table; typedef struct Fts3Cursor Fts3Cursor; typedef struct Fts3Expr Fts3Expr; typedef struct Fts3Phrase Fts3Phrase; typedef struct Fts3PhraseToken Fts3PhraseToken; typedef struct Fts3SegFilter Fts3SegFilter; typedef struct Fts3DeferredToken Fts3DeferredToken; typedef struct Fts3SegReader Fts3SegReader; | > | | > > > | > > > > > > > > > > | | | < | 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 | typedef struct Fts3Table Fts3Table; typedef struct Fts3Cursor Fts3Cursor; typedef struct Fts3Expr Fts3Expr; typedef struct Fts3Phrase Fts3Phrase; typedef struct Fts3PhraseToken Fts3PhraseToken; typedef struct Fts3Doclist Fts3Doclist; typedef struct Fts3SegFilter Fts3SegFilter; typedef struct Fts3DeferredToken Fts3DeferredToken; typedef struct Fts3SegReader Fts3SegReader; typedef struct Fts3MultiSegReader Fts3MultiSegReader; /* ** A connection to a fulltext index is an instance of the following ** structure. The xCreate and xConnect methods create an instance ** of this structure and xDestroy and xDisconnect free that instance. ** All other methods receive a pointer to the structure as one of their ** arguments. */ struct Fts3Table { sqlite3_vtab base; /* Base class used by SQLite core */ sqlite3 *db; /* The database connection */ const char *zDb; /* logical database name */ const char *zName; /* virtual table name */ int nColumn; /* number of named columns in virtual table */ char **azColumn; /* column names. malloced */ sqlite3_tokenizer *pTokenizer; /* tokenizer for inserts and queries */ /* Precompiled statements used by the implementation. Each of these ** statements is run and reset within a single virtual table API call. */ sqlite3_stmt *aStmt[27]; char *zReadExprlist; char *zWriteExprlist; int nNodeSize; /* Soft limit for node size */ u8 bHasStat; /* True if %_stat table exists */ u8 bHasDocsize; /* True if %_docsize table exists */ u8 bDescIdx; /* True if doclists are in reverse order */ int nPgsz; /* Page size for host database */ char *zSegmentsTbl; /* Name of %_segments table */ sqlite3_blob *pSegments; /* Blob handle open on %_segments table */ /* TODO: Fix the first paragraph of this comment. ** ** The following hash table is used to buffer pending index updates during ** transactions. Variable nPendingData estimates the memory size of the ** pending data, including hash table overhead, but not malloc overhead. ** When nPendingData exceeds nMaxPendingData, the buffer is flushed ** automatically. Variable iPrevDocid is the docid of the most recently ** inserted record. ** ** A single FTS4 table may have multiple full-text indexes. For each index ** there is an entry in the aIndex[] array. Index 0 is an index of all the ** terms that appear in the document set. Each subsequent index in aIndex[] ** is an index of prefixes of a specific length. */ int nIndex; /* Size of aIndex[] */ struct Fts3Index { int nPrefix; /* Prefix length (0 for main terms index) */ Fts3Hash hPending; /* Pending terms table for this index */ } *aIndex; int nMaxPendingData; /* Max pending data before flush to disk */ int nPendingData; /* Current bytes of pending data */ sqlite_int64 iPrevDocid; /* Docid of most recently inserted document */ #if defined(SQLITE_DEBUG) /* State variables used for validating that the transaction control ** methods of the virtual table are called at appropriate times. These ** values do not contribution to the FTS computation; they are used for ** verifying the SQLite core. */ |
︙ | ︙ | |||
197 198 199 200 201 202 203 | Fts3Expr *pExpr; /* Parsed MATCH query string */ int nPhrase; /* Number of matchable phrases in query */ Fts3DeferredToken *pDeferred; /* Deferred search tokens, if any */ sqlite3_int64 iPrevId; /* Previous id read from aDoclist */ char *pNextId; /* Pointer into the body of aDoclist */ char *aDoclist; /* List of docids for full-text queries */ int nDoclist; /* Size of buffer at aDoclist */ | | > | 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 | Fts3Expr *pExpr; /* Parsed MATCH query string */ int nPhrase; /* Number of matchable phrases in query */ Fts3DeferredToken *pDeferred; /* Deferred search tokens, if any */ sqlite3_int64 iPrevId; /* Previous id read from aDoclist */ char *pNextId; /* Pointer into the body of aDoclist */ char *aDoclist; /* List of docids for full-text queries */ int nDoclist; /* Size of buffer at aDoclist */ u8 bDesc; /* True to sort in descending order */ int eEvalmode; /* An FTS3_EVAL_XX constant */ int nRowAvg; /* Average size of database rows, in pages */ int nDoc; /* Documents in table */ int isMatchinfoNeeded; /* True when aMatchinfo[] needs filling in */ u32 *aMatchinfo; /* Information about most recent match */ int nMatchinfo; /* Number of elements in aMatchinfo[] */ char *zMatchinfo; /* Matchinfo specification */ }; |
︙ | ︙ | |||
230 231 232 233 234 235 236 237 238 239 240 241 | ** indicating that all columns should be searched, ** then eSearch would be set to FTS3_FULLTEXT_SEARCH+4. */ #define FTS3_FULLSCAN_SEARCH 0 /* Linear scan of %_content table */ #define FTS3_DOCID_SEARCH 1 /* Lookup by rowid on %_content table */ #define FTS3_FULLTEXT_SEARCH 2 /* Full-text index search */ /* ** A "phrase" is a sequence of one or more tokens that must match in ** sequence. A single token is the base case and the most common case. ** For a sequence of tokens contained in double-quotes (i.e. "one two three") ** nToken will be the number of tokens in the string. | > > > > > > > > > > > > < < < < < < | | > > > > > > > > | > > < | | | | | | | > > > > > > > > | > > | | | < | | 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 | ** indicating that all columns should be searched, ** then eSearch would be set to FTS3_FULLTEXT_SEARCH+4. */ #define FTS3_FULLSCAN_SEARCH 0 /* Linear scan of %_content table */ #define FTS3_DOCID_SEARCH 1 /* Lookup by rowid on %_content table */ #define FTS3_FULLTEXT_SEARCH 2 /* Full-text index search */ struct Fts3Doclist { char *aAll; /* Array containing doclist (or NULL) */ int nAll; /* Size of a[] in bytes */ char *pNextDocid; /* Pointer to next docid */ sqlite3_int64 iDocid; /* Current docid (if pList!=0) */ int bFreeList; /* True if pList should be sqlite3_free()d */ char *pList; /* Pointer to position list following iDocid */ int nList; /* Length of position list */ } doclist; /* ** A "phrase" is a sequence of one or more tokens that must match in ** sequence. A single token is the base case and the most common case. ** For a sequence of tokens contained in double-quotes (i.e. "one two three") ** nToken will be the number of tokens in the string. */ struct Fts3PhraseToken { char *z; /* Text of the token */ int n; /* Number of bytes in buffer z */ int isPrefix; /* True if token ends with a "*" character */ /* Variables above this point are populated when the expression is ** parsed (by code in fts3_expr.c). Below this point the variables are ** used when evaluating the expression. */ Fts3DeferredToken *pDeferred; /* Deferred token object for this token */ Fts3MultiSegReader *pSegcsr; /* Segment-reader for this token */ }; struct Fts3Phrase { /* Cache of doclist for this phrase. */ Fts3Doclist doclist; int bIncr; /* True if doclist is loaded incrementally */ int iDoclistToken; /* Variables below this point are populated by fts3_expr.c when parsing ** a MATCH expression. Everything above is part of the evaluation phase. */ int nToken; /* Number of tokens in the phrase */ int iColumn; /* Index of column this phrase must match */ Fts3PhraseToken aToken[1]; /* One entry for each token in the phrase */ }; /* ** A tree of these objects forms the RHS of a MATCH operator. ** ** If Fts3Expr.eType is FTSQUERY_PHRASE and isLoaded is true, then aDoclist ** points to a malloced buffer, size nDoclist bytes, containing the results ** of this phrase query in FTS3 doclist format. As usual, the initial ** "Length" field found in doclists stored on disk is omitted from this ** buffer. ** ** Variable aMI is used only for FTSQUERY_NEAR nodes to store the global ** matchinfo data. If it is not NULL, it points to an array of size nCol*3, ** where nCol is the number of columns in the queried FTS table. The array ** is populated as follows: ** ** aMI[iCol*3 + 0] = Undefined ** aMI[iCol*3 + 1] = Number of occurrences ** aMI[iCol*3 + 2] = Number of rows containing at least one instance ** ** The aMI array is allocated using sqlite3_malloc(). It should be freed ** when the expression node is. */ struct Fts3Expr { int eType; /* One of the FTSQUERY_XXX values defined below */ int nNear; /* Valid if eType==FTSQUERY_NEAR */ Fts3Expr *pParent; /* pParent->pLeft==this or pParent->pRight==this */ Fts3Expr *pLeft; /* Left operand */ Fts3Expr *pRight; /* Right operand */ Fts3Phrase *pPhrase; /* Valid if eType==FTSQUERY_PHRASE */ /* The following are used by the fts3_eval.c module. */ sqlite3_int64 iDocid; /* Current docid */ u8 bEof; /* True this expression is at EOF already */ u8 bStart; /* True if iDocid is valid */ u8 bDeferred; /* True if this expression is entirely deferred */ u32 *aMI; }; /* ** Candidate values for Fts3Query.eType. Note that the order of the first ** four values is in order of precedence when parsing expressions. For ** example, the following: ** |
︙ | ︙ | |||
313 314 315 316 317 318 319 | /* fts3_write.c */ int sqlite3Fts3UpdateMethod(sqlite3_vtab*,int,sqlite3_value**,sqlite3_int64*); int sqlite3Fts3PendingTermsFlush(Fts3Table *); void sqlite3Fts3PendingTermsClear(Fts3Table *); int sqlite3Fts3Optimize(Fts3Table *); int sqlite3Fts3SegReaderNew(int, sqlite3_int64, sqlite3_int64, sqlite3_int64, const char *, int, Fts3SegReader**); | | > < | | < > | | | | | > | | > > > | | > > < | < < | 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 | /* fts3_write.c */ int sqlite3Fts3UpdateMethod(sqlite3_vtab*,int,sqlite3_value**,sqlite3_int64*); int sqlite3Fts3PendingTermsFlush(Fts3Table *); void sqlite3Fts3PendingTermsClear(Fts3Table *); int sqlite3Fts3Optimize(Fts3Table *); int sqlite3Fts3SegReaderNew(int, sqlite3_int64, sqlite3_int64, sqlite3_int64, const char *, int, Fts3SegReader**); int sqlite3Fts3SegReaderPending( Fts3Table*,int,const char*,int,int,Fts3SegReader**); void sqlite3Fts3SegReaderFree(Fts3SegReader *); int sqlite3Fts3AllSegdirs(Fts3Table*, int, int, sqlite3_stmt **); int sqlite3Fts3ReadLock(Fts3Table *); int sqlite3Fts3ReadBlock(Fts3Table*, sqlite3_int64, char **, int*, int*); int sqlite3Fts3SelectDoctotal(Fts3Table *, sqlite3_stmt **); int sqlite3Fts3SelectDocsize(Fts3Table *, sqlite3_int64, sqlite3_stmt **); void sqlite3Fts3FreeDeferredTokens(Fts3Cursor *); int sqlite3Fts3DeferToken(Fts3Cursor *, Fts3PhraseToken *, int); int sqlite3Fts3CacheDeferredDoclists(Fts3Cursor *); void sqlite3Fts3FreeDeferredDoclists(Fts3Cursor *); void sqlite3Fts3SegmentsClose(Fts3Table *); /* Special values interpreted by sqlite3SegReaderCursor() */ #define FTS3_SEGCURSOR_PENDING -1 #define FTS3_SEGCURSOR_ALL -2 int sqlite3Fts3SegReaderStart(Fts3Table*, Fts3MultiSegReader*, Fts3SegFilter*); int sqlite3Fts3SegReaderStep(Fts3Table *, Fts3MultiSegReader *); void sqlite3Fts3SegReaderFinish(Fts3MultiSegReader *); int sqlite3Fts3SegReaderCursor( Fts3Table *, int, int, const char *, int, int, int, Fts3MultiSegReader *); /* Flags allowed as part of the 4th argument to SegmentReaderIterate() */ #define FTS3_SEGMENT_REQUIRE_POS 0x00000001 #define FTS3_SEGMENT_IGNORE_EMPTY 0x00000002 #define FTS3_SEGMENT_COLUMN_FILTER 0x00000004 #define FTS3_SEGMENT_PREFIX 0x00000008 #define FTS3_SEGMENT_SCAN 0x00000010 /* Type passed as 4th argument to SegmentReaderIterate() */ struct Fts3SegFilter { const char *zTerm; int nTerm; int iCol; int flags; }; struct Fts3MultiSegReader { /* Used internally by sqlite3Fts3SegReaderXXX() calls */ Fts3SegReader **apSegment; /* Array of Fts3SegReader objects */ int nSegment; /* Size of apSegment array */ int nAdvance; /* How many seg-readers to advance */ Fts3SegFilter *pFilter; /* Pointer to filter object */ char *aBuffer; /* Buffer to merge doclists in */ int nBuffer; /* Allocated size of aBuffer[] in bytes */ int iColFilter; /* If >=0, filter for this column */ int bRestart; /* Used by fts3.c only. */ int nCost; /* Cost of running iterator */ int bLookup; /* True if a lookup of a single entry. */ /* Output values. Valid only after Fts3SegReaderStep() returns SQLITE_ROW. */ char *zTerm; /* Pointer to term buffer */ int nTerm; /* Size of zTerm in bytes */ char *aDoclist; /* Pointer to doclist buffer */ int nDoclist; /* Size of aDoclist[] in bytes */ }; /* fts3.c */ int sqlite3Fts3PutVarint(char *, sqlite3_int64); int sqlite3Fts3GetVarint(const char *, sqlite_int64 *); int sqlite3Fts3GetVarint32(const char *, int *); int sqlite3Fts3VarintLen(sqlite3_uint64); void sqlite3Fts3Dequote(char *); void sqlite3Fts3DoclistPrev(int,char*,int,char**,sqlite3_int64*,int*,u8*); int sqlite3Fts3EvalPhraseStats(Fts3Cursor *, Fts3Expr *, u32 *); /* fts3_tokenizer.c */ const char *sqlite3Fts3NextToken(const char *, int *); int sqlite3Fts3InitHashTable(sqlite3 *, Fts3Hash *, const char *); int sqlite3Fts3InitTokenizer(Fts3Hash *pHash, const char *, sqlite3_tokenizer **, char ** ); |
︙ | ︙ | |||
413 414 415 416 417 418 419 420 | int sqlite3Fts3ExprInitTestInterface(sqlite3 *db); int sqlite3Fts3InitTerm(sqlite3 *db); #endif /* fts3_aux.c */ int sqlite3Fts3InitAux(sqlite3 *db); #endif /* _FTSINT_H */ | > > > > > > > > > > > > > > > > > > > > > > > > | 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 | int sqlite3Fts3ExprInitTestInterface(sqlite3 *db); int sqlite3Fts3InitTerm(sqlite3 *db); #endif /* fts3_aux.c */ int sqlite3Fts3InitAux(sqlite3 *db); int sqlite3Fts3TermSegReaderCursor( Fts3Cursor *pCsr, /* Virtual table cursor handle */ const char *zTerm, /* Term to query for */ int nTerm, /* Size of zTerm in bytes */ int isPrefix, /* True for a prefix search */ Fts3MultiSegReader **ppSegcsr /* OUT: Allocated seg-reader cursor */ ); void sqlite3Fts3EvalPhraseCleanup(Fts3Phrase *); int sqlite3Fts3EvalStart(Fts3Cursor *, Fts3Expr *, int); int sqlite3Fts3EvalNext(Fts3Cursor *pCsr); int sqlite3Fts3MsrIncrStart( Fts3Table*, Fts3MultiSegReader*, int, const char*, int); int sqlite3Fts3MsrIncrNext( Fts3Table *, Fts3MultiSegReader *, sqlite3_int64 *, char **, int *); char *sqlite3Fts3EvalPhrasePoslist(Fts3Cursor *, Fts3Expr *, int iCol); int sqlite3Fts3MsrOvfl(Fts3Cursor *, Fts3MultiSegReader *, int *); int sqlite3Fts3MsrIncrRestart(Fts3MultiSegReader *pCsr); int sqlite3Fts3DeferredTokenList(Fts3DeferredToken *, char **, int *); #endif /* SQLITE_ENABLE_FTS3 */ #endif /* _FTSINT_H */ |
Changes to ext/fts3/fts3_aux.c.
1 2 3 4 5 6 7 8 9 10 11 12 13 | /* ** 2011 Jan 27 ** ** The author disclaims copyright to this source code. In place of ** a legal notice, here is a blessing: ** ** May you do good and not evil. ** May you find forgiveness for yourself and forgive others. ** May you share freely, never taking more than you give. ** ****************************************************************************** ** */ | | < | | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 | /* ** 2011 Jan 27 ** ** The author disclaims copyright to this source code. In place of ** a legal notice, here is a blessing: ** ** May you do good and not evil. ** May you find forgiveness for yourself and forgive others. ** May you share freely, never taking more than you give. ** ****************************************************************************** ** */ #include "fts3Int.h" #if !defined(SQLITE_CORE) || defined(SQLITE_ENABLE_FTS3) #include <string.h> #include <assert.h> typedef struct Fts3auxTable Fts3auxTable; typedef struct Fts3auxCursor Fts3auxCursor; struct Fts3auxTable { sqlite3_vtab base; /* Base class used by SQLite core */ Fts3Table *pFts3Tab; }; struct Fts3auxCursor { sqlite3_vtab_cursor base; /* Base class used by SQLite core */ Fts3MultiSegReader csr; /* Must be right after "base" */ Fts3SegFilter filter; char *zStop; int nStop; /* Byte-length of string zStop */ int isEof; /* True if cursor is at EOF */ sqlite3_int64 iRowid; /* Current rowid */ int iCol; /* Current value of 'col' column */ |
︙ | ︙ | |||
92 93 94 95 96 97 98 99 100 101 102 103 104 105 | if( !p ) return SQLITE_NOMEM; memset(p, 0, nByte); p->pFts3Tab = (Fts3Table *)&p[1]; p->pFts3Tab->zDb = (char *)&p->pFts3Tab[1]; p->pFts3Tab->zName = &p->pFts3Tab->zDb[nDb+1]; p->pFts3Tab->db = db; memcpy((char *)p->pFts3Tab->zDb, zDb, nDb); memcpy((char *)p->pFts3Tab->zName, zFts3, nFts3); sqlite3Fts3Dequote((char *)p->pFts3Tab->zName); *ppVtab = (sqlite3_vtab *)p; return SQLITE_OK; | > | 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 | if( !p ) return SQLITE_NOMEM; memset(p, 0, nByte); p->pFts3Tab = (Fts3Table *)&p[1]; p->pFts3Tab->zDb = (char *)&p->pFts3Tab[1]; p->pFts3Tab->zName = &p->pFts3Tab->zDb[nDb+1]; p->pFts3Tab->db = db; p->pFts3Tab->nIndex = 1; memcpy((char *)p->pFts3Tab->zDb, zDb, nDb); memcpy((char *)p->pFts3Tab->zName, zFts3, nFts3); sqlite3Fts3Dequote((char *)p->pFts3Tab->zName); *ppVtab = (sqlite3_vtab *)p; return SQLITE_OK; |
︙ | ︙ | |||
372 373 374 375 376 377 378 | if( idxNum&FTS4AUX_LE_CONSTRAINT ){ int iIdx = (idxNum&FTS4AUX_GE_CONSTRAINT) ? 1 : 0; pCsr->zStop = sqlite3_mprintf("%s", sqlite3_value_text(apVal[iIdx])); pCsr->nStop = sqlite3_value_bytes(apVal[iIdx]); if( pCsr->zStop==0 ) return SQLITE_NOMEM; } | | | 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 | if( idxNum&FTS4AUX_LE_CONSTRAINT ){ int iIdx = (idxNum&FTS4AUX_GE_CONSTRAINT) ? 1 : 0; pCsr->zStop = sqlite3_mprintf("%s", sqlite3_value_text(apVal[iIdx])); pCsr->nStop = sqlite3_value_bytes(apVal[iIdx]); if( pCsr->zStop==0 ) return SQLITE_NOMEM; } rc = sqlite3Fts3SegReaderCursor(pFts3, 0, FTS3_SEGCURSOR_ALL, pCsr->filter.zTerm, pCsr->filter.nTerm, 0, isScan, &pCsr->csr ); if( rc==SQLITE_OK ){ rc = sqlite3Fts3SegReaderStart(pFts3, &pCsr->csr, &pCsr->filter); } if( rc==SQLITE_OK ) rc = fts3auxNextMethod(pCursor); |
︙ | ︙ |
Changes to ext/fts3/fts3_expr.c.
︙ | ︙ | |||
11 12 13 14 15 16 17 18 19 20 21 22 23 24 | ****************************************************************************** ** ** This module contains code that implements a parser for fts3 query strings ** (the right-hand argument to the MATCH operator). Because the supported ** syntax is relatively simple, the whole tokenizer/parser system is ** hand-coded. */ #if !defined(SQLITE_CORE) || defined(SQLITE_ENABLE_FTS3) /* ** By default, this module parses the legacy syntax that has been ** traditionally used by fts3. Or, if SQLITE_ENABLE_FTS3_PARENTHESIS ** is defined, then it uses the new syntax. The differences between ** the new and the old syntaxes are: | > | 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 | ****************************************************************************** ** ** This module contains code that implements a parser for fts3 query strings ** (the right-hand argument to the MATCH operator). Because the supported ** syntax is relatively simple, the whole tokenizer/parser system is ** hand-coded. */ #include "fts3Int.h" #if !defined(SQLITE_CORE) || defined(SQLITE_ENABLE_FTS3) /* ** By default, this module parses the legacy syntax that has been ** traditionally used by fts3. Or, if SQLITE_ENABLE_FTS3_PARENTHESIS ** is defined, then it uses the new syntax. The differences between ** the new and the old syntaxes are: |
︙ | ︙ | |||
73 74 75 76 77 78 79 | #endif /* ** Default span for NEAR operators. */ #define SQLITE_FTS3_DEFAULT_NEAR_PARAM 10 | < > > > > > > > > > | 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 | #endif /* ** Default span for NEAR operators. */ #define SQLITE_FTS3_DEFAULT_NEAR_PARAM 10 #include <string.h> #include <assert.h> /* ** isNot: ** This variable is used by function getNextNode(). When getNextNode() is ** called, it sets ParseContext.isNot to true if the 'next node' is a ** FTSQUERY_PHRASE with a unary "-" attached to it. i.e. "mysql" in the ** FTS3 query "sqlite -mysql". Otherwise, ParseContext.isNot is set to ** zero. */ typedef struct ParseContext ParseContext; struct ParseContext { sqlite3_tokenizer *pTokenizer; /* Tokenizer module */ const char **azCol; /* Array of column names for fts3 table */ int nCol; /* Number of entries in azCol[] */ int iDefaultCol; /* Default column to query */ int isNot; /* True if getNextNode() sees a unary - */ sqlite3_context *pCtx; /* Write error message here */ int nNest; /* Number of nested brackets */ }; /* ** This function is equivalent to the standard isspace() function. ** |
︙ | ︙ | |||
168 169 170 171 172 173 174 | memcpy(pRet->pPhrase->aToken[0].z, zToken, nToken); if( iEnd<n && z[iEnd]=='*' ){ pRet->pPhrase->aToken[0].isPrefix = 1; iEnd++; } if( !sqlite3_fts3_enable_parentheses && iStart>0 && z[iStart-1]=='-' ){ | | | 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 | memcpy(pRet->pPhrase->aToken[0].z, zToken, nToken); if( iEnd<n && z[iEnd]=='*' ){ pRet->pPhrase->aToken[0].isPrefix = 1; iEnd++; } if( !sqlite3_fts3_enable_parentheses && iStart>0 && z[iStart-1]=='-' ){ pParse->isNot = 1; } } nConsumed = iEnd; } pModule->xClose(pCursor); } |
︙ | ︙ | |||
220 221 222 223 224 225 226 227 228 229 230 231 | sqlite3_tokenizer_module const *pModule = pTokenizer->pModule; int rc; Fts3Expr *p = 0; sqlite3_tokenizer_cursor *pCursor = 0; char *zTemp = 0; int nTemp = 0; rc = pModule->xOpen(pTokenizer, zInput, nInput, &pCursor); if( rc==SQLITE_OK ){ int ii; pCursor->pTokenizer = pTokenizer; for(ii=0; rc==SQLITE_OK; ii++){ | > > > > > > > > > > > > > > > > > > > > > > | | | | | | < | | > | < < | > | | | < | | < < < < | > > > | | < < | | > > > > | | < | | < | | < | | < < < | 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 | sqlite3_tokenizer_module const *pModule = pTokenizer->pModule; int rc; Fts3Expr *p = 0; sqlite3_tokenizer_cursor *pCursor = 0; char *zTemp = 0; int nTemp = 0; const int nSpace = sizeof(Fts3Expr) + sizeof(Fts3Phrase); int nToken = 0; /* The final Fts3Expr data structure, including the Fts3Phrase, ** Fts3PhraseToken structures token buffers are all stored as a single ** allocation so that the expression can be freed with a single call to ** sqlite3_free(). Setting this up requires a two pass approach. ** ** The first pass, in the block below, uses a tokenizer cursor to iterate ** through the tokens in the expression. This pass uses fts3ReallocOrFree() ** to assemble data in two dynamic buffers: ** ** Buffer p: Points to the Fts3Expr structure, followed by the Fts3Phrase ** structure, followed by the array of Fts3PhraseToken ** structures. This pass only populates the Fts3PhraseToken array. ** ** Buffer zTemp: Contains copies of all tokens. ** ** The second pass, in the block that begins "if( rc==SQLITE_DONE )" below, ** appends buffer zTemp to buffer p, and fills in the Fts3Expr and Fts3Phrase ** structures. */ rc = pModule->xOpen(pTokenizer, zInput, nInput, &pCursor); if( rc==SQLITE_OK ){ int ii; pCursor->pTokenizer = pTokenizer; for(ii=0; rc==SQLITE_OK; ii++){ const char *zByte; int nByte, iBegin, iEnd, iPos; rc = pModule->xNext(pCursor, &zByte, &nByte, &iBegin, &iEnd, &iPos); if( rc==SQLITE_OK ){ Fts3PhraseToken *pToken; p = fts3ReallocOrFree(p, nSpace + ii*sizeof(Fts3PhraseToken)); if( !p ) goto no_mem; zTemp = fts3ReallocOrFree(zTemp, nTemp + nByte); if( !zTemp ) goto no_mem; assert( nToken==ii ); pToken = &((Fts3Phrase *)(&p[1]))->aToken[ii]; memset(pToken, 0, sizeof(Fts3PhraseToken)); memcpy(&zTemp[nTemp], zByte, nByte); nTemp += nByte; pToken->n = nByte; pToken->isPrefix = (iEnd<nInput && zInput[iEnd]=='*'); nToken = ii+1; } } pModule->xClose(pCursor); pCursor = 0; } if( rc==SQLITE_DONE ){ int jj; char *zBuf = 0; p = fts3ReallocOrFree(p, nSpace + nToken*sizeof(Fts3PhraseToken) + nTemp); if( !p ) goto no_mem; memset(p, 0, (char *)&(((Fts3Phrase *)&p[1])->aToken[0])-(char *)p); p->eType = FTSQUERY_PHRASE; p->pPhrase = (Fts3Phrase *)&p[1]; p->pPhrase->iColumn = pParse->iDefaultCol; p->pPhrase->nToken = nToken; zBuf = (char *)&p->pPhrase->aToken[nToken]; memcpy(zBuf, zTemp, nTemp); sqlite3_free(zTemp); for(jj=0; jj<p->pPhrase->nToken; jj++){ p->pPhrase->aToken[jj].z = zBuf; zBuf += p->pPhrase->aToken[jj].n; } rc = SQLITE_OK; } *ppExpr = p; return rc; no_mem: |
︙ | ︙ | |||
336 337 338 339 340 341 342 343 344 345 346 347 348 349 | int iCol; int iColLen; int rc; Fts3Expr *pRet = 0; const char *zInput = z; int nInput = n; /* Skip over any whitespace before checking for a keyword, an open or ** close bracket, or a quoted string. */ while( nInput>0 && fts3isspace(*zInput) ){ nInput--; zInput++; | > > | 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 | int iCol; int iColLen; int rc; Fts3Expr *pRet = 0; const char *zInput = z; int nInput = n; pParse->isNot = 0; /* Skip over any whitespace before checking for a keyword, an open or ** close bracket, or a quoted string. */ while( nInput>0 && fts3isspace(*zInput) ){ nInput--; zInput++; |
︙ | ︙ | |||
555 556 557 558 559 560 561 | Fts3Expr *p = 0; int nByte = 0; rc = getNextNode(pParse, zIn, nIn, &p, &nByte); if( rc==SQLITE_OK ){ int isPhrase; if( !sqlite3_fts3_enable_parentheses | | < | 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 | Fts3Expr *p = 0; int nByte = 0; rc = getNextNode(pParse, zIn, nIn, &p, &nByte); if( rc==SQLITE_OK ){ int isPhrase; if( !sqlite3_fts3_enable_parentheses && p->eType==FTSQUERY_PHRASE && pParse->isNot ){ /* Create an implicit NOT operator. */ Fts3Expr *pNot = fts3MallocZero(sizeof(Fts3Expr)); if( !pNot ){ sqlite3Fts3ExprFree(p); rc = SQLITE_NOMEM; goto exprparse_out; } pNot->eType = FTSQUERY_NOT; pNot->pRight = p; if( pNotBranch ){ pNot->pLeft = pNotBranch; } pNotBranch = pNot; p = pPrev; }else{ int eType = p->eType; isPhrase = (eType==FTSQUERY_PHRASE || p->pLeft); /* The isRequirePhrase variable is set to true if a phrase or ** an expression contained in parenthesis is required. If a ** binary operator (AND, OR, NOT or NEAR) is encounted when ** isRequirePhrase is set, this is a syntax error. */ |
︙ | ︙ | |||
736 737 738 739 740 741 742 743 744 | } /* ** Free a parsed fts3 query expression allocated by sqlite3Fts3ExprParse(). */ void sqlite3Fts3ExprFree(Fts3Expr *p){ if( p ){ sqlite3Fts3ExprFree(p->pLeft); sqlite3Fts3ExprFree(p->pRight); | > > | | 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 | } /* ** Free a parsed fts3 query expression allocated by sqlite3Fts3ExprParse(). */ void sqlite3Fts3ExprFree(Fts3Expr *p){ if( p ){ assert( p->eType==FTSQUERY_PHRASE || p->pPhrase==0 ); sqlite3Fts3ExprFree(p->pLeft); sqlite3Fts3ExprFree(p->pRight); sqlite3Fts3EvalPhraseCleanup(p->pPhrase); sqlite3_free(p->aMI); sqlite3_free(p); } } /**************************************************************************** ***************************************************************************** ** Everything after this point is just test code. |
︙ | ︙ | |||
796 797 798 799 800 801 802 | */ static char *exprToString(Fts3Expr *pExpr, char *zBuf){ switch( pExpr->eType ){ case FTSQUERY_PHRASE: { Fts3Phrase *pPhrase = pExpr->pPhrase; int i; zBuf = sqlite3_mprintf( | | | 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 | */ static char *exprToString(Fts3Expr *pExpr, char *zBuf){ switch( pExpr->eType ){ case FTSQUERY_PHRASE: { Fts3Phrase *pPhrase = pExpr->pPhrase; int i; zBuf = sqlite3_mprintf( "%zPHRASE %d 0", zBuf, pPhrase->iColumn); for(i=0; zBuf && i<pPhrase->nToken; i++){ zBuf = sqlite3_mprintf("%z %.*s%s", zBuf, pPhrase->aToken[i].n, pPhrase->aToken[i].z, (pPhrase->aToken[i].isPrefix?"+":"") ); } return zBuf; |
︙ | ︙ |
Changes to ext/fts3/fts3_hash.c.
︙ | ︙ | |||
19 20 21 22 23 24 25 26 27 28 29 30 31 32 | ** ** * The FTS3 module is being built as an extension ** (in which case SQLITE_CORE is not defined), or ** ** * The FTS3 module is being built into the core of ** SQLite (in which case SQLITE_ENABLE_FTS3 is defined). */ #if !defined(SQLITE_CORE) || defined(SQLITE_ENABLE_FTS3) #include <assert.h> #include <stdlib.h> #include <string.h> #include "sqlite3.h" | > | 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 | ** ** * The FTS3 module is being built as an extension ** (in which case SQLITE_CORE is not defined), or ** ** * The FTS3 module is being built into the core of ** SQLite (in which case SQLITE_ENABLE_FTS3 is defined). */ #include "fts3Int.h" #if !defined(SQLITE_CORE) || defined(SQLITE_ENABLE_FTS3) #include <assert.h> #include <stdlib.h> #include <string.h> #include "sqlite3.h" |
︙ | ︙ |
Changes to ext/fts3/fts3_icu.c.
1 2 3 4 5 6 7 8 9 10 11 12 | /* ** 2007 June 22 ** ** The author disclaims copyright to this source code. In place of ** a legal notice, here is a blessing: ** ** May you do good and not evil. ** May you find forgiveness for yourself and forgive others. ** May you share freely, never taking more than you give. ** ************************************************************************* ** This file implements a tokenizer for fts3 based on the ICU library. | < < | | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 | /* ** 2007 June 22 ** ** The author disclaims copyright to this source code. In place of ** a legal notice, here is a blessing: ** ** May you do good and not evil. ** May you find forgiveness for yourself and forgive others. ** May you share freely, never taking more than you give. ** ************************************************************************* ** This file implements a tokenizer for fts3 based on the ICU library. */ #include "fts3Int.h" #if !defined(SQLITE_CORE) || defined(SQLITE_ENABLE_FTS3) #ifdef SQLITE_ENABLE_ICU #include <assert.h> #include <string.h> #include "fts3_tokenizer.h" |
︙ | ︙ |
Changes to ext/fts3/fts3_porter.c.
︙ | ︙ | |||
18 19 20 21 22 23 24 | ** ** * The FTS3 module is being built as an extension ** (in which case SQLITE_CORE is not defined), or ** ** * The FTS3 module is being built into the core of ** SQLite (in which case SQLITE_ENABLE_FTS3 is defined). */ | < < > | 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 | ** ** * The FTS3 module is being built as an extension ** (in which case SQLITE_CORE is not defined), or ** ** * The FTS3 module is being built into the core of ** SQLite (in which case SQLITE_ENABLE_FTS3 is defined). */ #include "fts3Int.h" #if !defined(SQLITE_CORE) || defined(SQLITE_ENABLE_FTS3) #include <assert.h> #include <stdlib.h> #include <stdio.h> #include <string.h> #include "fts3_tokenizer.h" |
︙ | ︙ |
Changes to ext/fts3/fts3_snippet.c.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 | /* ** 2009 Oct 23 ** ** The author disclaims copyright to this source code. In place of ** a legal notice, here is a blessing: ** ** May you do good and not evil. ** May you find forgiveness for yourself and forgive others. ** May you share freely, never taking more than you give. ** ****************************************************************************** */ #if !defined(SQLITE_CORE) || defined(SQLITE_ENABLE_FTS3) | > < | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 | /* ** 2009 Oct 23 ** ** The author disclaims copyright to this source code. In place of ** a legal notice, here is a blessing: ** ** May you do good and not evil. ** May you find forgiveness for yourself and forgive others. ** May you share freely, never taking more than you give. ** ****************************************************************************** */ #include "fts3Int.h" #if !defined(SQLITE_CORE) || defined(SQLITE_ENABLE_FTS3) #include <string.h> #include <assert.h> /* ** Characters that may appear in the second argument to matchinfo(). */ #define FTS3_MATCHINFO_NPHRASE 'p' /* 1 value */ |
︙ | ︙ | |||
172 173 174 175 176 177 178 | int (*x)(Fts3Expr*,int,void*), /* Callback function to invoke for phrases */ void *pCtx /* Second argument to pass to callback */ ){ int iPhrase = 0; /* Variable used as the phrase counter */ return fts3ExprIterate2(pExpr, &iPhrase, x, pCtx); } | < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < > | < < < < < < < < | 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 | int (*x)(Fts3Expr*,int,void*), /* Callback function to invoke for phrases */ void *pCtx /* Second argument to pass to callback */ ){ int iPhrase = 0; /* Variable used as the phrase counter */ return fts3ExprIterate2(pExpr, &iPhrase, x, pCtx); } /* ** This is an fts3ExprIterate() callback used while loading the doclists ** for each phrase into Fts3Expr.aDoclist[]/nDoclist. See also ** fts3ExprLoadDoclists(). */ static int fts3ExprLoadDoclistsCb(Fts3Expr *pExpr, int iPhrase, void *ctx){ int rc = SQLITE_OK; Fts3Phrase *pPhrase = pExpr->pPhrase; LoadDoclistCtx *p = (LoadDoclistCtx *)ctx; UNUSED_PARAMETER(iPhrase); p->nPhrase++; p->nToken += pPhrase->nToken; return rc; } /* ** Load the doclists for each phrase in the query associated with FTS3 cursor ** pCsr. |
︙ | ︙ | |||
411 412 413 414 415 416 417 | static int fts3SnippetFindPositions(Fts3Expr *pExpr, int iPhrase, void *ctx){ SnippetIter *p = (SnippetIter *)ctx; SnippetPhrase *pPhrase = &p->aPhrase[iPhrase]; char *pCsr; pPhrase->nToken = pExpr->pPhrase->nToken; | | | 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 | static int fts3SnippetFindPositions(Fts3Expr *pExpr, int iPhrase, void *ctx){ SnippetIter *p = (SnippetIter *)ctx; SnippetPhrase *pPhrase = &p->aPhrase[iPhrase]; char *pCsr; pPhrase->nToken = pExpr->pPhrase->nToken; pCsr = sqlite3Fts3EvalPhrasePoslist(p->pCsr, pExpr, p->iCol); if( pCsr ){ int iFirst = 0; pPhrase->pList = pCsr; fts3GetDeltaPosition(&pCsr, &iFirst); pPhrase->pHead = pCsr; pPhrase->pTail = pCsr; pPhrase->iHead = iFirst; |
︙ | ︙ | |||
768 769 770 771 772 773 774 | if( !c ) nEntry++; } *ppCollist = pEnd; return nEntry; } | < < < < < < < < < < < < < < < < < < < < | 716 717 718 719 720 721 722 723 724 725 726 727 728 729 | if( !c ) nEntry++; } *ppCollist = pEnd; return nEntry; } /* ** fts3ExprIterate() callback used to collect the "global" matchinfo stats ** for a single query. ** ** fts3ExprIterate() callback to load the 'global' elements of a ** FTS3_MATCHINFO_HITS matchinfo array. The global stats are those elements ** of the matchinfo array that are constant for all rows returned by the |
︙ | ︙ | |||
821 822 823 824 825 826 827 | */ static int fts3ExprGlobalHitsCb( Fts3Expr *pExpr, /* Phrase expression node */ int iPhrase, /* Phrase number (numbered from zero) */ void *pCtx /* Pointer to MatchInfo structure */ ){ MatchInfo *p = (MatchInfo *)pCtx; | | < < < | | < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < | < < | < > > | | 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 | */ static int fts3ExprGlobalHitsCb( Fts3Expr *pExpr, /* Phrase expression node */ int iPhrase, /* Phrase number (numbered from zero) */ void *pCtx /* Pointer to MatchInfo structure */ ){ MatchInfo *p = (MatchInfo *)pCtx; return sqlite3Fts3EvalPhraseStats( p->pCursor, pExpr, &p->aMatchinfo[3*iPhrase*p->nCol] ); } /* ** fts3ExprIterate() callback used to collect the "local" part of the ** FTS3_MATCHINFO_HITS array. The local stats are those elements of the ** array that are different for each row returned by the query. */ static int fts3ExprLocalHitsCb( Fts3Expr *pExpr, /* Phrase expression node */ int iPhrase, /* Phrase number */ void *pCtx /* Pointer to MatchInfo structure */ ){ MatchInfo *p = (MatchInfo *)pCtx; int iStart = iPhrase * p->nCol * 3; int i; for(i=0; i<p->nCol; i++){ char *pCsr; pCsr = sqlite3Fts3EvalPhrasePoslist(p->pCursor, pExpr, i); if( pCsr ){ p->aMatchinfo[iStart+i*3] = fts3ColumnlistCount(&pCsr); }else{ p->aMatchinfo[iStart+i*3] = 0; } } return SQLITE_OK; } static int fts3MatchinfoCheck( |
︙ | ︙ | |||
972 973 974 975 976 977 978 | ** iterating through a multi-column position-list corresponding to the ** hits for a single phrase on a single row in order to calculate the ** values for a matchinfo() FTS3_MATCHINFO_LCS request. */ typedef struct LcsIterator LcsIterator; struct LcsIterator { Fts3Expr *pExpr; /* Pointer to phrase expression */ | < | | 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 | ** iterating through a multi-column position-list corresponding to the ** hits for a single phrase on a single row in order to calculate the ** values for a matchinfo() FTS3_MATCHINFO_LCS request. */ typedef struct LcsIterator LcsIterator; struct LcsIterator { Fts3Expr *pExpr; /* Pointer to phrase expression */ int iPosOffset; /* Tokens count up to end of this phrase */ char *pRead; /* Cursor used to iterate through aDoclist */ int iPos; /* Current position */ }; /* ** If LcsIterator.iCol is set to the following value, the iterator has ** finished iterating through all offsets for all columns. */ |
︙ | ︙ | |||
1005 1006 1007 1008 1009 1010 1011 | */ static int fts3LcsIteratorAdvance(LcsIterator *pIter){ char *pRead = pIter->pRead; sqlite3_int64 iRead; int rc = 0; pRead += sqlite3Fts3GetVarint(pRead, &iRead); | | | < < < < < < < | 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 | */ static int fts3LcsIteratorAdvance(LcsIterator *pIter){ char *pRead = pIter->pRead; sqlite3_int64 iRead; int rc = 0; pRead += sqlite3Fts3GetVarint(pRead, &iRead); if( iRead==0 || iRead==1 ){ pRead = 0; rc = 1; }else{ pIter->iPos += (int)(iRead-2); } pIter->pRead = pRead; return rc; } |
︙ | ︙ | |||
1047 1048 1049 1050 1051 1052 1053 1054 1055 1056 1057 | /* Allocate and populate the array of LcsIterator objects. The array ** contains one element for each matchable phrase in the query. **/ aIter = sqlite3_malloc(sizeof(LcsIterator) * pCsr->nPhrase); if( !aIter ) return SQLITE_NOMEM; memset(aIter, 0, sizeof(LcsIterator) * pCsr->nPhrase); (void)fts3ExprIterate(pCsr->pExpr, fts3MatchinfoLcsCb, (void*)aIter); for(i=0; i<pInfo->nPhrase; i++){ LcsIterator *pIter = &aIter[i]; nToken -= pIter->pExpr->pPhrase->nToken; pIter->iPosOffset = nToken; | > < < < < < < < < < < > > > > | | | | | < < < | | 927 928 929 930 931 932 933 934 935 936 937 938 939 940 941 942 943 944 945 946 947 948 949 950 951 952 953 954 955 956 957 958 959 960 961 962 963 964 965 966 967 968 | /* Allocate and populate the array of LcsIterator objects. The array ** contains one element for each matchable phrase in the query. **/ aIter = sqlite3_malloc(sizeof(LcsIterator) * pCsr->nPhrase); if( !aIter ) return SQLITE_NOMEM; memset(aIter, 0, sizeof(LcsIterator) * pCsr->nPhrase); (void)fts3ExprIterate(pCsr->pExpr, fts3MatchinfoLcsCb, (void*)aIter); for(i=0; i<pInfo->nPhrase; i++){ LcsIterator *pIter = &aIter[i]; nToken -= pIter->pExpr->pPhrase->nToken; pIter->iPosOffset = nToken; } for(iCol=0; iCol<pInfo->nCol; iCol++){ int nLcs = 0; /* LCS value for this column */ int nLive = 0; /* Number of iterators in aIter not at EOF */ for(i=0; i<pInfo->nPhrase; i++){ LcsIterator *pIt = &aIter[i]; pIt->pRead = sqlite3Fts3EvalPhrasePoslist(pCsr, pIt->pExpr, iCol); if( pIt->pRead ){ pIt->iPos = pIt->iPosOffset; fts3LcsIteratorAdvance(&aIter[i]); nLive++; } } while( nLive>0 ){ LcsIterator *pAdv = 0; /* The iterator to advance by one position */ int nThisLcs = 0; /* LCS for the current iterator positions */ for(i=0; i<pInfo->nPhrase; i++){ LcsIterator *pIter = &aIter[i]; if( pIter->pRead==0 ){ /* This iterator is already at EOF for this column. */ nThisLcs = 0; }else{ if( pAdv==0 || pIter->iPos<pAdv->iPos ){ pAdv = pIter; } if( nThisLcs==0 || pIter->iPos==pIter[-1].iPos ){ |
︙ | ︙ | |||
1422 1423 1424 1425 1426 1427 1428 | TermOffsetCtx *p = (TermOffsetCtx *)ctx; int nTerm; /* Number of tokens in phrase */ int iTerm; /* For looping through nTerm phrase terms */ char *pList; /* Pointer to position list for phrase */ int iPos = 0; /* First position in position-list */ UNUSED_PARAMETER(iPhrase); | | | 1294 1295 1296 1297 1298 1299 1300 1301 1302 1303 1304 1305 1306 1307 1308 | TermOffsetCtx *p = (TermOffsetCtx *)ctx; int nTerm; /* Number of tokens in phrase */ int iTerm; /* For looping through nTerm phrase terms */ char *pList; /* Pointer to position list for phrase */ int iPos = 0; /* First position in position-list */ UNUSED_PARAMETER(iPhrase); pList = sqlite3Fts3EvalPhrasePoslist(p->pCsr, pExpr, p->iCol); nTerm = pExpr->pPhrase->nToken; if( pList ){ fts3GetDeltaPosition(&pList, &iPos); assert( iPos>=0 ); } for(iTerm=0; iTerm<nTerm; iTerm++){ |
︙ | ︙ |
Changes to ext/fts3/fts3_term.c.
︙ | ︙ | |||
11 12 13 14 15 16 17 18 19 20 | ****************************************************************************** ** ** This file is not part of the production FTS code. It is only used for ** testing. It contains a virtual table implementation that provides direct ** access to the full-text index of an FTS table. */ #if !defined(SQLITE_CORE) || defined(SQLITE_ENABLE_FTS3) #ifdef SQLITE_TEST | > < > | | 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 | ****************************************************************************** ** ** This file is not part of the production FTS code. It is only used for ** testing. It contains a virtual table implementation that provides direct ** access to the full-text index of an FTS table. */ #include "fts3Int.h" #if !defined(SQLITE_CORE) || defined(SQLITE_ENABLE_FTS3) #ifdef SQLITE_TEST #include <string.h> #include <assert.h> typedef struct Fts3termTable Fts3termTable; typedef struct Fts3termCursor Fts3termCursor; struct Fts3termTable { sqlite3_vtab base; /* Base class used by SQLite core */ int iIndex; /* Index for Fts3Table.aIndex[] */ Fts3Table *pFts3Tab; }; struct Fts3termCursor { sqlite3_vtab_cursor base; /* Base class used by SQLite core */ Fts3MultiSegReader csr; /* Must be right after "base" */ Fts3SegFilter filter; int isEof; /* True if cursor is at EOF */ char *pNext; sqlite3_int64 iRowid; /* Current 'rowid' value */ sqlite3_int64 iDocid; /* Current 'docid' value */ |
︙ | ︙ | |||
52 53 54 55 56 57 58 | /* ** This function does all the work for both the xConnect and xCreate methods. ** These tables have no persistent representation of their own, so xConnect ** and xCreate are identical operations. */ static int fts3termConnectMethod( sqlite3 *db, /* Database connection */ | | > < > > > > | 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 | /* ** This function does all the work for both the xConnect and xCreate methods. ** These tables have no persistent representation of their own, so xConnect ** and xCreate are identical operations. */ static int fts3termConnectMethod( sqlite3 *db, /* Database connection */ void *pCtx, /* Non-zero for an fts4prefix table */ int argc, /* Number of elements in argv array */ const char * const *argv, /* xCreate/xConnect argument array */ sqlite3_vtab **ppVtab, /* OUT: New sqlite3_vtab object */ char **pzErr /* OUT: sqlite3_malloc'd error message */ ){ char const *zDb; /* Name of database (e.g. "main") */ char const *zFts3; /* Name of fts3 table */ int nDb; /* Result of strlen(zDb) */ int nFts3; /* Result of strlen(zFts3) */ int nByte; /* Bytes of space to allocate here */ int rc; /* value returned by declare_vtab() */ Fts3termTable *p; /* Virtual table object to return */ int iIndex = 0; if( argc==5 ){ iIndex = atoi(argv[4]); argc--; } /* The user should specify a single argument - the name of an fts3 table. */ if( argc!=4 ){ *pzErr = sqlite3_mprintf( "wrong number of arguments to fts4term constructor" ); return SQLITE_ERROR; |
︙ | ︙ | |||
93 94 95 96 97 98 99 100 101 102 103 104 105 106 | if( !p ) return SQLITE_NOMEM; memset(p, 0, nByte); p->pFts3Tab = (Fts3Table *)&p[1]; p->pFts3Tab->zDb = (char *)&p->pFts3Tab[1]; p->pFts3Tab->zName = &p->pFts3Tab->zDb[nDb+1]; p->pFts3Tab->db = db; memcpy((char *)p->pFts3Tab->zDb, zDb, nDb); memcpy((char *)p->pFts3Tab->zName, zFts3, nFts3); sqlite3Fts3Dequote((char *)p->pFts3Tab->zName); *ppVtab = (sqlite3_vtab *)p; return SQLITE_OK; | > > | 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 | if( !p ) return SQLITE_NOMEM; memset(p, 0, nByte); p->pFts3Tab = (Fts3Table *)&p[1]; p->pFts3Tab->zDb = (char *)&p->pFts3Tab[1]; p->pFts3Tab->zName = &p->pFts3Tab->zDb[nDb+1]; p->pFts3Tab->db = db; p->pFts3Tab->nIndex = iIndex+1; p->iIndex = iIndex; memcpy((char *)p->pFts3Tab->zDb, zDb, nDb); memcpy((char *)p->pFts3Tab->zName, zFts3, nFts3); sqlite3Fts3Dequote((char *)p->pFts3Tab->zName); *ppVtab = (sqlite3_vtab *)p; return SQLITE_OK; |
︙ | ︙ | |||
240 241 242 243 244 245 246 | sqlite3_vtab_cursor *pCursor, /* The cursor used for this query */ int idxNum, /* Strategy index */ const char *idxStr, /* Unused */ int nVal, /* Number of elements in apVal */ sqlite3_value **apVal /* Arguments for the indexing scheme */ ){ Fts3termCursor *pCsr = (Fts3termCursor *)pCursor; | > | | | 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 | sqlite3_vtab_cursor *pCursor, /* The cursor used for this query */ int idxNum, /* Strategy index */ const char *idxStr, /* Unused */ int nVal, /* Number of elements in apVal */ sqlite3_value **apVal /* Arguments for the indexing scheme */ ){ Fts3termCursor *pCsr = (Fts3termCursor *)pCursor; Fts3termTable *p = (Fts3termTable *)pCursor->pVtab; Fts3Table *pFts3 = p->pFts3Tab; int rc; UNUSED_PARAMETER(nVal); UNUSED_PARAMETER(idxNum); UNUSED_PARAMETER(idxStr); UNUSED_PARAMETER(apVal); assert( idxStr==0 && idxNum==0 ); /* In case this cursor is being reused, close and zero it. */ testcase(pCsr->filter.zTerm); sqlite3Fts3SegReaderFinish(&pCsr->csr); memset(&pCsr->csr, 0, ((u8*)&pCsr[1]) - (u8*)&pCsr->csr); pCsr->filter.flags = FTS3_SEGMENT_REQUIRE_POS|FTS3_SEGMENT_IGNORE_EMPTY; pCsr->filter.flags |= FTS3_SEGMENT_SCAN; rc = sqlite3Fts3SegReaderCursor(pFts3, p->iIndex, FTS3_SEGCURSOR_ALL, pCsr->filter.zTerm, pCsr->filter.nTerm, 0, 1, &pCsr->csr ); if( rc==SQLITE_OK ){ rc = sqlite3Fts3SegReaderStart(pFts3, &pCsr->csr, &pCsr->filter); } if( rc==SQLITE_OK ){ rc = fts3termNextMethod(pCursor); |
︙ | ︙ |
Added ext/fts3/fts3_test.c.
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 | /* ** 2011 Jun 13 ** ** The author disclaims copyright to this source code. In place of ** a legal notice, here is a blessing: ** ** May you do good and not evil. ** May you find forgiveness for yourself and forgive others. ** May you share freely, never taking more than you give. ** ****************************************************************************** ** ** This file is not part of the production FTS code. It is only used for ** testing. It contains a Tcl command that can be used to test if a document ** matches an FTS NEAR expression. */ #include <tcl.h> #include <string.h> #include <assert.h> #define NM_MAX_TOKEN 12 typedef struct NearPhrase NearPhrase; typedef struct NearDocument NearDocument; typedef struct NearToken NearToken; struct NearDocument { int nToken; /* Length of token in bytes */ NearToken *aToken; /* Token array */ }; struct NearToken { int n; /* Length of token in bytes */ const char *z; /* Pointer to token string */ }; struct NearPhrase { int nNear; /* Preceding NEAR value */ int nToken; /* Number of tokens in this phrase */ NearToken aToken[NM_MAX_TOKEN]; /* Array of tokens in this phrase */ }; static int nm_phrase_match( NearPhrase *p, NearToken *aToken ){ int ii; for(ii=0; ii<p->nToken; ii++){ NearToken *pToken = &p->aToken[ii]; if( pToken->n>0 && pToken->z[pToken->n-1]=='*' ){ if( aToken[ii].n<(pToken->n-1) ) return 0; if( memcmp(aToken[ii].z, pToken->z, pToken->n-1) ) return 0; }else{ if( aToken[ii].n!=pToken->n ) return 0; if( memcmp(aToken[ii].z, pToken->z, pToken->n) ) return 0; } } return 1; } static int nm_near_chain( int iDir, /* Direction to iterate through aPhrase[] */ NearDocument *pDoc, /* Document to match against */ int iPos, /* Position at which iPhrase was found */ int nPhrase, /* Size of phrase array */ NearPhrase *aPhrase, /* Phrase array */ int iPhrase /* Index of phrase found */ ){ int iStart; int iStop; int ii; int nNear; int iPhrase2; NearPhrase *p; NearPhrase *pPrev; assert( iDir==1 || iDir==-1 ); if( iDir==1 ){ if( (iPhrase+1)==nPhrase ) return 1; nNear = aPhrase[iPhrase+1].nNear; }else{ if( iPhrase==0 ) return 1; nNear = aPhrase[iPhrase].nNear; } pPrev = &aPhrase[iPhrase]; iPhrase2 = iPhrase+iDir; p = &aPhrase[iPhrase2]; iStart = iPos - nNear - p->nToken; iStop = iPos + nNear + pPrev->nToken; if( iStart<0 ) iStart = 0; if( iStop > pDoc->nToken - p->nToken ) iStop = pDoc->nToken - p->nToken; for(ii=iStart; ii<=iStop; ii++){ if( nm_phrase_match(p, &pDoc->aToken[ii]) ){ if( nm_near_chain(iDir, pDoc, ii, nPhrase, aPhrase, iPhrase2) ) return 1; } } return 0; } static int nm_match_count( NearDocument *pDoc, /* Document to match against */ int nPhrase, /* Size of phrase array */ NearPhrase *aPhrase, /* Phrase array */ int iPhrase /* Index of phrase to count matches for */ ){ int nOcc = 0; int ii; NearPhrase *p = &aPhrase[iPhrase]; for(ii=0; ii<(pDoc->nToken + 1 - p->nToken); ii++){ if( nm_phrase_match(p, &pDoc->aToken[ii]) ){ /* Test forward NEAR chain (i>iPhrase) */ if( 0==nm_near_chain(1, pDoc, ii, nPhrase, aPhrase, iPhrase) ) continue; /* Test reverse NEAR chain (i<iPhrase) */ if( 0==nm_near_chain(-1, pDoc, ii, nPhrase, aPhrase, iPhrase) ) continue; /* This is a real match. Increment the counter. */ nOcc++; } } return nOcc; } /* ** Tclcmd: fts3_near_match DOCUMENT EXPR ?OPTIONS? */ static int fts3_near_match_cmd( ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[] ){ int nTotal = 0; int rc; int ii; int nPhrase; NearPhrase *aPhrase = 0; NearDocument doc = {0, 0}; Tcl_Obj **apDocToken; Tcl_Obj *pRet; Tcl_Obj *pPhrasecount = 0; Tcl_Obj **apExprToken; int nExprToken; /* Must have 3 or more arguments. */ if( objc<3 || (objc%2)==0 ){ Tcl_WrongNumArgs(interp, 1, objv, "DOCUMENT EXPR ?OPTION VALUE?..."); rc = TCL_ERROR; goto near_match_out; } for(ii=3; ii<objc; ii+=2){ enum NM_enum { NM_PHRASECOUNTS }; struct TestnmSubcmd { char *zName; enum NM_enum eOpt; } aOpt[] = { { "-phrasecountvar", NM_PHRASECOUNTS }, { 0, 0 } }; int iOpt; if( Tcl_GetIndexFromObjStruct( interp, objv[ii], aOpt, sizeof(aOpt[0]), "option", 0, &iOpt) ){ return TCL_ERROR; } switch( aOpt[iOpt].eOpt ){ case NM_PHRASECOUNTS: pPhrasecount = objv[ii+1]; break; } } rc = Tcl_ListObjGetElements(interp, objv[1], &doc.nToken, &apDocToken); if( rc!=TCL_OK ) goto near_match_out; doc.aToken = (NearToken *)ckalloc(doc.nToken*sizeof(NearToken)); for(ii=0; ii<doc.nToken; ii++){ doc.aToken[ii].z = Tcl_GetStringFromObj(apDocToken[ii], &doc.aToken[ii].n); } rc = Tcl_ListObjGetElements(interp, objv[2], &nExprToken, &apExprToken); if( rc!=TCL_OK ) goto near_match_out; nPhrase = (nExprToken + 1) / 2; aPhrase = (NearPhrase *)ckalloc(nPhrase * sizeof(NearPhrase)); memset(aPhrase, 0, nPhrase * sizeof(NearPhrase)); for(ii=0; ii<nPhrase; ii++){ Tcl_Obj *pPhrase = apExprToken[ii*2]; Tcl_Obj **apToken; int nToken; int jj; rc = Tcl_ListObjGetElements(interp, pPhrase, &nToken, &apToken); if( rc!=TCL_OK ) goto near_match_out; if( nToken>NM_MAX_TOKEN ){ Tcl_AppendResult(interp, "Too many tokens in phrase", 0); rc = TCL_ERROR; goto near_match_out; } for(jj=0; jj<nToken; jj++){ NearToken *pT = &aPhrase[ii].aToken[jj]; pT->z = Tcl_GetStringFromObj(apToken[jj], &pT->n); } aPhrase[ii].nToken = nToken; } for(ii=1; ii<nPhrase; ii++){ Tcl_Obj *pNear = apExprToken[2*ii-1]; int nNear; rc = Tcl_GetIntFromObj(interp, pNear, &nNear); if( rc!=TCL_OK ) goto near_match_out; aPhrase[ii].nNear = nNear; } pRet = Tcl_NewObj(); Tcl_IncrRefCount(pRet); for(ii=0; ii<nPhrase; ii++){ int nOcc = nm_match_count(&doc, nPhrase, aPhrase, ii); Tcl_ListObjAppendElement(interp, pRet, Tcl_NewIntObj(nOcc)); nTotal += nOcc; } if( pPhrasecount ){ Tcl_ObjSetVar2(interp, pPhrasecount, 0, pRet, 0); } Tcl_DecrRefCount(pRet); Tcl_SetObjResult(interp, Tcl_NewBooleanObj(nTotal>0)); near_match_out: ckfree((char *)aPhrase); ckfree((char *)doc.aToken); return rc; } /* ** Tclcmd: fts3_configure_incr_load ?CHUNKSIZE THRESHOLD? ** ** Normally, FTS uses hard-coded values to determine the minimum doclist ** size eligible for incremental loading, and the size of the chunks loaded ** when a doclist is incrementally loaded. This command allows the built-in ** values to be overridden for testing purposes. ** ** If present, the first argument is the chunksize in bytes to load doclists ** in. The second argument is the minimum doclist size in bytes to use ** incremental loading with. ** ** Whether or not the arguments are present, this command returns a list of ** two integers - the initial chunksize and threshold when the command is ** invoked. This can be used to restore the default behaviour after running ** tests. For example: ** ** # Override incr-load settings for testing: ** set cfg [fts3_configure_incr_load $new_chunksize $new_threshold] ** ** .... run tests .... ** ** # Restore initial incr-load settings: ** eval fts3_configure_incr_load $cfg */ static int fts3_configure_incr_load_cmd( ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[] ){ #ifdef SQLITE_ENABLE_FTS3 extern int test_fts3_node_chunksize; extern int test_fts3_node_chunk_threshold; int iArg1; int iArg2; Tcl_Obj *pRet; if( objc!=1 && objc!=3 ){ Tcl_WrongNumArgs(interp, 1, objv, "?CHUNKSIZE THRESHOLD?"); return TCL_ERROR; } pRet = Tcl_NewObj(); Tcl_IncrRefCount(pRet); Tcl_ListObjAppendElement( interp, pRet, Tcl_NewIntObj(test_fts3_node_chunksize)); Tcl_ListObjAppendElement( interp, pRet, Tcl_NewIntObj(test_fts3_node_chunk_threshold)); if( objc==3 ){ int iArg1; int iArg2; if( Tcl_GetIntFromObj(interp, objv[1], &iArg1) || Tcl_GetIntFromObj(interp, objv[2], &iArg2) ){ Tcl_DecrRefCount(pRet); return TCL_ERROR; } test_fts3_node_chunksize = iArg1; test_fts3_node_chunk_threshold = iArg2; } Tcl_SetObjResult(interp, pRet); Tcl_DecrRefCount(pRet); #endif return TCL_OK; } int Sqlitetestfts3_Init(Tcl_Interp *interp){ Tcl_CreateObjCommand(interp, "fts3_near_match", fts3_near_match_cmd, 0, 0); Tcl_CreateObjCommand(interp, "fts3_configure_incr_load", fts3_configure_incr_load_cmd, 0, 0 ); return TCL_OK; } |
Changes to ext/fts3/fts3_tokenizer.c.
︙ | ︙ | |||
19 20 21 22 23 24 25 | ** ** * The FTS3 module is being built as an extension ** (in which case SQLITE_CORE is not defined), or ** ** * The FTS3 module is being built into the core of ** SQLite (in which case SQLITE_ENABLE_FTS3 is defined). */ | < < > > | | 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 | ** ** * The FTS3 module is being built as an extension ** (in which case SQLITE_CORE is not defined), or ** ** * The FTS3 module is being built into the core of ** SQLite (in which case SQLITE_ENABLE_FTS3 is defined). */ #include "sqlite3ext.h" #ifndef SQLITE_CORE SQLITE_EXTENSION_INIT1 #endif #include "fts3Int.h" #if !defined(SQLITE_CORE) || defined(SQLITE_ENABLE_FTS3) #include <assert.h> #include <string.h> /* ** Implementation of the SQL scalar function for accessing the underlying ** hash table. This function may be called as follows: ** |
︙ | ︙ |
Changes to ext/fts3/fts3_tokenizer1.c.
︙ | ︙ | |||
18 19 20 21 22 23 24 | ** ** * The FTS3 module is being built as an extension ** (in which case SQLITE_CORE is not defined), or ** ** * The FTS3 module is being built into the core of ** SQLite (in which case SQLITE_ENABLE_FTS3 is defined). */ | < < > | 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 | ** ** * The FTS3 module is being built as an extension ** (in which case SQLITE_CORE is not defined), or ** ** * The FTS3 module is being built into the core of ** SQLite (in which case SQLITE_ENABLE_FTS3 is defined). */ #include "fts3Int.h" #if !defined(SQLITE_CORE) || defined(SQLITE_ENABLE_FTS3) #include <assert.h> #include <stdlib.h> #include <stdio.h> #include <string.h> #include "fts3_tokenizer.h" |
︙ | ︙ |
Changes to ext/fts3/fts3_write.c.
︙ | ︙ | |||
13 14 15 16 17 18 19 20 21 | ** This file is part of the SQLite FTS3 extension module. Specifically, ** this file contains code to insert, update and delete rows from FTS3 ** tables. It also contains code to merge FTS3 b-tree segments. Some ** of the sub-routines used to merge segments are also used by the query ** code in fts3.c. */ #if !defined(SQLITE_CORE) || defined(SQLITE_ENABLE_FTS3) | > < > > > > > > > > > > > > > > > > > > > > > > > > > > > < < | > | 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 | ** This file is part of the SQLite FTS3 extension module. Specifically, ** this file contains code to insert, update and delete rows from FTS3 ** tables. It also contains code to merge FTS3 b-tree segments. Some ** of the sub-routines used to merge segments are also used by the query ** code in fts3.c. */ #include "fts3Int.h" #if !defined(SQLITE_CORE) || defined(SQLITE_ENABLE_FTS3) #include <string.h> #include <assert.h> #include <stdlib.h> /* ** When full-text index nodes are loaded from disk, the buffer that they ** are loaded into has the following number of bytes of padding at the end ** of it. i.e. if a full-text index node is 900 bytes in size, then a buffer ** of 920 bytes is allocated for it. ** ** This means that if we have a pointer into a buffer containing node data, ** it is always safe to read up to two varints from it without risking an ** overread, even if the node data is corrupted. */ #define FTS3_NODE_PADDING (FTS3_VARINT_MAX*2) /* ** Under certain circumstances, b-tree nodes (doclists) can be loaded into ** memory incrementally instead of all at once. This can be a big performance ** win (reduced IO and CPU) if SQLite stops calling the virtual table xNext() ** method before retrieving all query results (as may happen, for example, ** if a query has a LIMIT clause). ** ** Incremental loading is used for b-tree nodes FTS3_NODE_CHUNK_THRESHOLD ** bytes and larger. Nodes are loaded in chunks of FTS3_NODE_CHUNKSIZE bytes. ** The code is written so that the hard lower-limit for each of these values ** is 1. Clearly such small values would be inefficient, but can be useful ** for testing purposes. ** ** If this module is built with SQLITE_TEST defined, these constants may ** be overridden at runtime for testing purposes. File fts3_test.c contains ** a Tcl interface to read and write the values. */ #ifdef SQLITE_TEST int test_fts3_node_chunksize = (4*1024); int test_fts3_node_chunk_threshold = (4*1024)*4; # define FTS3_NODE_CHUNKSIZE test_fts3_node_chunksize # define FTS3_NODE_CHUNK_THRESHOLD test_fts3_node_chunk_threshold #else # define FTS3_NODE_CHUNKSIZE (4*1024) # define FTS3_NODE_CHUNK_THRESHOLD (FTS3_NODE_CHUNKSIZE*4) #endif typedef struct PendingList PendingList; typedef struct SegmentNode SegmentNode; typedef struct SegmentWriter SegmentWriter; /* ** An instance of the following data structure is used to build doclists ** incrementally. See function fts3PendingListAppend() for details. */ struct PendingList { int nData; char *aData; int nSpace; sqlite3_int64 iLastDocid; sqlite3_int64 iLastCol; |
︙ | ︙ | |||
70 71 72 73 74 75 76 | ** a contiguous set of segment b-tree leaf nodes. Although the details of ** this structure are only manipulated by code in this file, opaque handles ** of type Fts3SegReader* are also used by code in fts3.c to iterate through ** terms when querying the full-text index. See functions: ** ** sqlite3Fts3SegReaderNew() ** sqlite3Fts3SegReaderFree() | < > > > | > > > | 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 | ** a contiguous set of segment b-tree leaf nodes. Although the details of ** this structure are only manipulated by code in this file, opaque handles ** of type Fts3SegReader* are also used by code in fts3.c to iterate through ** terms when querying the full-text index. See functions: ** ** sqlite3Fts3SegReaderNew() ** sqlite3Fts3SegReaderFree() ** sqlite3Fts3SegReaderIterate() ** ** Methods used to manipulate Fts3SegReader structures: ** ** fts3SegReaderNext() ** fts3SegReaderFirstDocid() ** fts3SegReaderNextDocid() */ struct Fts3SegReader { int iIdx; /* Index within level, or 0x7FFFFFFF for PT */ sqlite3_int64 iStartBlock; /* Rowid of first leaf block to traverse */ sqlite3_int64 iLeafEndBlock; /* Rowid of final leaf block to traverse */ sqlite3_int64 iEndBlock; /* Rowid of final block in segment (or 0) */ sqlite3_int64 iCurrentBlock; /* Current leaf block (or 0) */ char *aNode; /* Pointer to node data (or NULL) */ int nNode; /* Size of buffer at aNode (or 0) */ int nPopulate; /* If >0, bytes of buffer aNode[] loaded */ sqlite3_blob *pBlob; /* If not NULL, blob handle to read node */ Fts3HashElem **ppNextElem; /* Variables set by fts3SegReaderNext(). These may be read directly ** by the caller. They are valid from the time SegmentReaderNew() returns ** until SegmentReaderNext() returns something other than SQLITE_OK ** (i.e. SQLITE_DONE). */ int nTerm; /* Number of bytes in current term */ char *zTerm; /* Pointer to current term */ int nTermAlloc; /* Allocated size of zTerm buffer */ char *aDoclist; /* Pointer to doclist of current entry */ int nDoclist; /* Size of doclist in current entry */ /* The following variables are used by fts3SegReaderNextDocid() to iterate ** through the current doclist (aDoclist/nDoclist). */ char *pOffsetList; int nOffsetList; /* For descending pending seg-readers only */ sqlite3_int64 iDocid; }; #define fts3SegReaderIsPending(p) ((p)->ppNextElem!=0) #define fts3SegReaderIsRootOnly(p) ((p)->aNode==(char *)&(p)[1]) /* |
︙ | ︙ | |||
141 142 143 144 145 146 147 148 149 150 151 152 153 154 | ** the interior part of the segment b+-tree structures (everything except ** the leaf nodes). These functions and type are only ever used by code ** within the fts3SegWriterXXX() family of functions described above. ** ** fts3NodeAddTerm() ** fts3NodeWrite() ** fts3NodeFree() */ struct SegmentNode { SegmentNode *pParent; /* Parent node (or NULL for root node) */ SegmentNode *pRight; /* Pointer to right-sibling */ SegmentNode *pLeftmost; /* Pointer to left-most node of this depth */ int nEntry; /* Number of terms written to node so far */ char *zTerm; /* Pointer to previous term buffer */ | > > > > > > > > | 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 | ** the interior part of the segment b+-tree structures (everything except ** the leaf nodes). These functions and type are only ever used by code ** within the fts3SegWriterXXX() family of functions described above. ** ** fts3NodeAddTerm() ** fts3NodeWrite() ** fts3NodeFree() ** ** When a b+tree is written to the database (either as a result of a merge ** or the pending-terms table being flushed), leaves are written into the ** database file as soon as they are completely populated. The interior of ** the tree is assembled in memory and written out only once all leaves have ** been populated and stored. This is Ok, as the b+-tree fanout is usually ** very large, meaning that the interior of the tree consumes relatively ** little memory. */ struct SegmentNode { SegmentNode *pParent; /* Parent node (or NULL for root node) */ SegmentNode *pRight; /* Pointer to right-sibling */ SegmentNode *pLeftmost; /* Pointer to left-most node of this depth */ int nEntry; /* Number of terms written to node so far */ char *zTerm; /* Pointer to previous term buffer */ |
︙ | ︙ | |||
171 172 173 174 175 176 177 | #define SQL_DELETE_ALL_STAT 6 #define SQL_SELECT_CONTENT_BY_ROWID 7 #define SQL_NEXT_SEGMENT_INDEX 8 #define SQL_INSERT_SEGMENTS 9 #define SQL_NEXT_SEGMENTS_ID 10 #define SQL_INSERT_SEGDIR 11 #define SQL_SELECT_LEVEL 12 | | | | > > > > > | 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 | #define SQL_DELETE_ALL_STAT 6 #define SQL_SELECT_CONTENT_BY_ROWID 7 #define SQL_NEXT_SEGMENT_INDEX 8 #define SQL_INSERT_SEGMENTS 9 #define SQL_NEXT_SEGMENTS_ID 10 #define SQL_INSERT_SEGDIR 11 #define SQL_SELECT_LEVEL 12 #define SQL_SELECT_LEVEL_RANGE 13 #define SQL_SELECT_LEVEL_COUNT 14 #define SQL_SELECT_SEGDIR_MAX_LEVEL 15 #define SQL_DELETE_SEGDIR_LEVEL 16 #define SQL_DELETE_SEGMENTS_RANGE 17 #define SQL_CONTENT_INSERT 18 #define SQL_DELETE_DOCSIZE 19 #define SQL_REPLACE_DOCSIZE 20 #define SQL_SELECT_DOCSIZE 21 #define SQL_SELECT_DOCTOTAL 22 #define SQL_REPLACE_DOCTOTAL 23 #define SQL_SELECT_ALL_PREFIX_LEVEL 24 #define SQL_DELETE_ALL_TERMS_SEGDIR 25 #define SQL_DELETE_SEGDIR_RANGE 26 /* ** This function is used to obtain an SQLite prepared statement handle ** for the statement identified by the second argument. If successful, ** *pp is set to the requested statement handle and SQLITE_OK returned. ** Otherwise, an SQLite error code is returned and *pp is set to 0. ** |
︙ | ︙ | |||
218 219 220 221 222 223 224 | /* 10 */ "SELECT coalesce((SELECT max(blockid) FROM %Q.'%q_segments') + 1, 1)", /* 11 */ "INSERT INTO %Q.'%q_segdir' VALUES(?,?,?,?,?,?)", /* Return segments in order from oldest to newest.*/ /* 12 */ "SELECT idx, start_block, leaves_end_block, end_block, root " "FROM %Q.'%q_segdir' WHERE level = ? ORDER BY idx ASC", /* 13 */ "SELECT idx, start_block, leaves_end_block, end_block, root " | > | | > > > > > | 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 | /* 10 */ "SELECT coalesce((SELECT max(blockid) FROM %Q.'%q_segments') + 1, 1)", /* 11 */ "INSERT INTO %Q.'%q_segdir' VALUES(?,?,?,?,?,?)", /* Return segments in order from oldest to newest.*/ /* 12 */ "SELECT idx, start_block, leaves_end_block, end_block, root " "FROM %Q.'%q_segdir' WHERE level = ? ORDER BY idx ASC", /* 13 */ "SELECT idx, start_block, leaves_end_block, end_block, root " "FROM %Q.'%q_segdir' WHERE level BETWEEN ? AND ?" "ORDER BY level DESC, idx ASC", /* 14 */ "SELECT count(*) FROM %Q.'%q_segdir' WHERE level = ?", /* 15 */ "SELECT max(level) FROM %Q.'%q_segdir' WHERE level BETWEEN ? AND ?", /* 16 */ "DELETE FROM %Q.'%q_segdir' WHERE level = ?", /* 17 */ "DELETE FROM %Q.'%q_segments' WHERE blockid BETWEEN ? AND ?", /* 18 */ "INSERT INTO %Q.'%q_content' VALUES(%s)", /* 19 */ "DELETE FROM %Q.'%q_docsize' WHERE docid = ?", /* 20 */ "REPLACE INTO %Q.'%q_docsize' VALUES(?,?)", /* 21 */ "SELECT size FROM %Q.'%q_docsize' WHERE docid=?", /* 22 */ "SELECT value FROM %Q.'%q_stat' WHERE id=0", /* 23 */ "REPLACE INTO %Q.'%q_stat' VALUES(0,?)", /* 24 */ "", /* 25 */ "", /* 26 */ "DELETE FROM %Q.'%q_segdir' WHERE level BETWEEN ? AND ?", }; int rc = SQLITE_OK; sqlite3_stmt *pStmt; assert( SizeofArray(azSql)==SizeofArray(p->aStmt) ); assert( eStmt<SizeofArray(azSql) && eStmt>=0 ); |
︙ | ︙ | |||
386 387 388 389 390 391 392 | ** ** 0: idx ** 1: start_block ** 2: leaves_end_block ** 3: end_block ** 4: root */ | | > > > > > > > > > > > | > > > > > | > > | 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 | ** ** 0: idx ** 1: start_block ** 2: leaves_end_block ** 3: end_block ** 4: root */ int sqlite3Fts3AllSegdirs( Fts3Table *p, /* FTS3 table */ int iIndex, /* Index for p->aIndex[] */ int iLevel, /* Level to select */ sqlite3_stmt **ppStmt /* OUT: Compiled statement */ ){ int rc; sqlite3_stmt *pStmt = 0; assert( iLevel==FTS3_SEGCURSOR_ALL || iLevel>=0 ); assert( iLevel<FTS3_SEGDIR_MAXLEVEL ); assert( iIndex>=0 && iIndex<p->nIndex ); if( iLevel<0 ){ /* "SELECT * FROM %_segdir WHERE level BETWEEN ? AND ? ORDER BY ..." */ rc = fts3SqlStmt(p, SQL_SELECT_LEVEL_RANGE, &pStmt, 0); if( rc==SQLITE_OK ){ sqlite3_bind_int(pStmt, 1, iIndex*FTS3_SEGDIR_MAXLEVEL); sqlite3_bind_int(pStmt, 2, (iIndex+1)*FTS3_SEGDIR_MAXLEVEL-1); } }else{ /* "SELECT * FROM %_segdir WHERE level = ? ORDER BY ..." */ rc = fts3SqlStmt(p, SQL_SELECT_LEVEL, &pStmt, 0); if( rc==SQLITE_OK ){ sqlite3_bind_int(pStmt, 1, iLevel+iIndex*FTS3_SEGDIR_MAXLEVEL); } } *ppStmt = pStmt; return rc; } /* |
︙ | ︙ | |||
507 508 509 510 511 512 513 514 515 516 517 518 519 520 | *pRc = rc; if( p!=*pp ){ *pp = p; return 1; } return 0; } /* ** Tokenize the nul-terminated string zText and add all tokens to the ** pending-terms hash-table. The docid used is that currently stored in ** p->iPrevDocid, and the column is specified by argument iCol. ** ** If successful, SQLITE_OK is returned. Otherwise, an SQLite error code. | > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > | 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 | *pRc = rc; if( p!=*pp ){ *pp = p; return 1; } return 0; } /* ** Free a PendingList object allocated by fts3PendingListAppend(). */ static void fts3PendingListDelete(PendingList *pList){ sqlite3_free(pList); } /* ** Add an entry to one of the pending-terms hash tables. */ static int fts3PendingTermsAddOne( Fts3Table *p, int iCol, int iPos, Fts3Hash *pHash, /* Pending terms hash table to add entry to */ const char *zToken, int nToken ){ PendingList *pList; int rc = SQLITE_OK; pList = (PendingList *)fts3HashFind(pHash, zToken, nToken); if( pList ){ p->nPendingData -= (pList->nData + nToken + sizeof(Fts3HashElem)); } if( fts3PendingListAppend(&pList, p->iPrevDocid, iCol, iPos, &rc) ){ if( pList==fts3HashInsert(pHash, zToken, nToken, pList) ){ /* Malloc failed while inserting the new entry. This can only ** happen if there was no previous entry for this token. */ assert( 0==fts3HashFind(pHash, zToken, nToken) ); sqlite3_free(pList); rc = SQLITE_NOMEM; } } if( rc==SQLITE_OK ){ p->nPendingData += (pList->nData + nToken + sizeof(Fts3HashElem)); } return rc; } /* ** Tokenize the nul-terminated string zText and add all tokens to the ** pending-terms hash-table. The docid used is that currently stored in ** p->iPrevDocid, and the column is specified by argument iCol. ** ** If successful, SQLITE_OK is returned. Otherwise, an SQLite error code. |
︙ | ︙ | |||
556 557 558 559 560 561 562 | } pCsr->pTokenizer = pTokenizer; xNext = pModule->xNext; while( SQLITE_OK==rc && SQLITE_OK==(rc = xNext(pCsr, &zToken, &nToken, &iStart, &iEnd, &iPos)) ){ | < | | < < < | < < < < | | < | < > > | > > | > > | 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 | } pCsr->pTokenizer = pTokenizer; xNext = pModule->xNext; while( SQLITE_OK==rc && SQLITE_OK==(rc = xNext(pCsr, &zToken, &nToken, &iStart, &iEnd, &iPos)) ){ int i; if( iPos>=nWord ) nWord = iPos+1; /* Positions cannot be negative; we use -1 as a terminator internally. ** Tokens must have a non-zero length. */ if( iPos<0 || !zToken || nToken<=0 ){ rc = SQLITE_ERROR; break; } /* Add the term to the terms index */ rc = fts3PendingTermsAddOne( p, iCol, iPos, &p->aIndex[0].hPending, zToken, nToken ); /* Add the term to each of the prefix indexes that it is not too ** short for. */ for(i=1; rc==SQLITE_OK && i<p->nIndex; i++){ struct Fts3Index *pIndex = &p->aIndex[i]; if( nToken<pIndex->nPrefix ) continue; rc = fts3PendingTermsAddOne( p, iCol, iPos, &pIndex->hPending, zToken, pIndex->nPrefix ); } } pModule->xClose(pCsr); *pnWord = nWord; return (rc==SQLITE_DONE ? SQLITE_OK : rc); } |
︙ | ︙ | |||
613 614 615 616 617 618 619 | if( rc!=SQLITE_OK ) return rc; } p->iPrevDocid = iDocid; return SQLITE_OK; } /* | | > > | > | | > | | > | 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 | if( rc!=SQLITE_OK ) return rc; } p->iPrevDocid = iDocid; return SQLITE_OK; } /* ** Discard the contents of the pending-terms hash tables. */ void sqlite3Fts3PendingTermsClear(Fts3Table *p){ int i; for(i=0; i<p->nIndex; i++){ Fts3HashElem *pElem; Fts3Hash *pHash = &p->aIndex[i].hPending; for(pElem=fts3HashFirst(pHash); pElem; pElem=fts3HashNext(pElem)){ PendingList *pList = (PendingList *)fts3HashData(pElem); fts3PendingListDelete(pList); } fts3HashClear(pHash); } p->nPendingData = 0; } /* ** This function is called by the xUpdate() method as part of an INSERT ** operation. It adds entries for each term in the new record to the ** pendingTerms hash table. |
︙ | ︙ | |||
776 777 778 779 780 781 782 | *pRC = rc; } /* ** Forward declaration to account for the circular dependency between ** functions fts3SegmentMerge() and fts3AllocateSegdirIdx(). */ | | | > > > > > | | | 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 940 941 942 943 944 | *pRC = rc; } /* ** Forward declaration to account for the circular dependency between ** functions fts3SegmentMerge() and fts3AllocateSegdirIdx(). */ static int fts3SegmentMerge(Fts3Table *, int, int); /* ** This function allocates a new level iLevel index in the segdir table. ** Usually, indexes are allocated within a level sequentially starting ** with 0, so the allocated index is one greater than the value returned ** by: ** ** SELECT max(idx) FROM %_segdir WHERE level = :iLevel ** ** However, if there are already FTS3_MERGE_COUNT indexes at the requested ** level, they are merged into a single level (iLevel+1) segment and the ** allocated index is 0. ** ** If successful, *piIdx is set to the allocated index slot and SQLITE_OK ** returned. Otherwise, an SQLite error code is returned. */ static int fts3AllocateSegdirIdx( Fts3Table *p, int iIndex, /* Index for p->aIndex */ int iLevel, int *piIdx ){ int rc; /* Return Code */ sqlite3_stmt *pNextIdx; /* Query for next idx at level iLevel */ int iNext = 0; /* Result of query pNextIdx */ /* Set variable iNext to the next available segdir index at level iLevel. */ rc = fts3SqlStmt(p, SQL_NEXT_SEGMENT_INDEX, &pNextIdx, 0); if( rc==SQLITE_OK ){ sqlite3_bind_int(pNextIdx, 1, iIndex*FTS3_SEGDIR_MAXLEVEL + iLevel); if( SQLITE_ROW==sqlite3_step(pNextIdx) ){ iNext = sqlite3_column_int(pNextIdx, 0); } rc = sqlite3_reset(pNextIdx); } if( rc==SQLITE_OK ){ /* If iNext is FTS3_MERGE_COUNT, indicating that level iLevel is already ** full, merge all segments in level iLevel into a single iLevel+1 ** segment and allocate (newly freed) index 0 at level iLevel. Otherwise, ** if iNext is less than FTS3_MERGE_COUNT, allocate index iNext. */ if( iNext>=FTS3_MERGE_COUNT ){ rc = fts3SegmentMerge(p, iIndex, iLevel); *piIdx = 0; }else{ *piIdx = iNext; } } return rc; |
︙ | ︙ | |||
856 857 858 859 860 861 862 | ** method (xFilter etc.) that may directly or indirectly call this function ** must call sqlite3Fts3SegmentsClose() before returning. */ int sqlite3Fts3ReadBlock( Fts3Table *p, /* FTS3 table handle */ sqlite3_int64 iBlockid, /* Access the row with blockid=$iBlockid */ char **paBlob, /* OUT: Blob data in malloc'd buffer */ | | > > > > > > < > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > | > > > > > < > > | > > > > > > | > > > > > | < > > > > | | | > > > > > > > > > > > | | > > > | > | | | > < < < < < | > > > | | < | | | | | | | > > | | < < | > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > | | > > > | | | | > | > > | | < < < < < < < | | | | > > | < | < < < < | > > > > | | | | < | < < < < < < < < < < < < < < < < < | | < < < < < < < < < < < < < < < < < < < < < < < < < < | | | < | | | | | > | 971 972 973 974 975 976 977 978 979 980 981 982 983 984 985 986 987 988 989 990 991 992 993 994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 1008 1009 1010 1011 1012 1013 1014 1015 1016 1017 1018 1019 1020 1021 1022 1023 1024 1025 1026 1027 1028 1029 1030 1031 1032 1033 1034 1035 1036 1037 1038 1039 1040 1041 1042 1043 1044 1045 1046 1047 1048 1049 1050 1051 1052 1053 1054 1055 1056 1057 1058 1059 1060 1061 1062 1063 1064 1065 1066 1067 1068 1069 1070 1071 1072 1073 1074 1075 1076 1077 1078 1079 1080 1081 1082 1083 1084 1085 1086 1087 1088 1089 1090 1091 1092 1093 1094 1095 1096 1097 1098 1099 1100 1101 1102 1103 1104 1105 1106 1107 1108 1109 1110 1111 1112 1113 1114 1115 1116 1117 1118 1119 1120 1121 1122 1123 1124 1125 1126 1127 1128 1129 1130 1131 1132 1133 1134 1135 1136 1137 1138 1139 1140 1141 1142 1143 1144 1145 1146 1147 1148 1149 1150 1151 1152 1153 1154 1155 1156 1157 1158 1159 1160 1161 1162 1163 1164 1165 1166 1167 1168 1169 1170 1171 1172 1173 1174 1175 1176 1177 1178 1179 1180 1181 1182 1183 1184 1185 1186 1187 1188 1189 1190 1191 1192 1193 1194 1195 1196 1197 1198 1199 1200 1201 1202 1203 1204 1205 1206 1207 1208 1209 1210 1211 1212 1213 1214 1215 1216 1217 1218 1219 1220 1221 1222 1223 1224 1225 1226 1227 1228 1229 1230 1231 1232 1233 1234 1235 1236 1237 1238 1239 1240 1241 1242 1243 1244 1245 1246 1247 1248 1249 1250 1251 1252 1253 1254 1255 1256 1257 1258 1259 1260 1261 1262 1263 1264 1265 1266 1267 1268 1269 1270 1271 1272 1273 1274 1275 1276 1277 1278 1279 1280 1281 1282 1283 1284 1285 1286 1287 1288 1289 1290 1291 1292 1293 1294 1295 1296 1297 1298 1299 1300 1301 1302 1303 1304 1305 1306 1307 1308 1309 1310 1311 1312 1313 1314 1315 1316 1317 1318 1319 1320 1321 1322 1323 1324 1325 1326 1327 1328 1329 1330 1331 1332 1333 1334 1335 1336 1337 1338 1339 1340 1341 1342 1343 1344 1345 1346 1347 1348 1349 1350 1351 1352 1353 1354 1355 1356 1357 | ** method (xFilter etc.) that may directly or indirectly call this function ** must call sqlite3Fts3SegmentsClose() before returning. */ int sqlite3Fts3ReadBlock( Fts3Table *p, /* FTS3 table handle */ sqlite3_int64 iBlockid, /* Access the row with blockid=$iBlockid */ char **paBlob, /* OUT: Blob data in malloc'd buffer */ int *pnBlob, /* OUT: Size of blob data */ int *pnLoad /* OUT: Bytes actually loaded */ ){ int rc; /* Return code */ /* pnBlob must be non-NULL. paBlob may be NULL or non-NULL. */ assert( pnBlob); if( p->pSegments ){ rc = sqlite3_blob_reopen(p->pSegments, iBlockid); }else{ if( 0==p->zSegmentsTbl ){ p->zSegmentsTbl = sqlite3_mprintf("%s_segments", p->zName); if( 0==p->zSegmentsTbl ) return SQLITE_NOMEM; } rc = sqlite3_blob_open( p->db, p->zDb, p->zSegmentsTbl, "block", iBlockid, 0, &p->pSegments ); } if( rc==SQLITE_OK ){ int nByte = sqlite3_blob_bytes(p->pSegments); *pnBlob = nByte; if( paBlob ){ char *aByte = sqlite3_malloc(nByte + FTS3_NODE_PADDING); if( !aByte ){ rc = SQLITE_NOMEM; }else{ if( pnLoad && nByte>(FTS3_NODE_CHUNK_THRESHOLD) ){ nByte = FTS3_NODE_CHUNKSIZE; *pnLoad = nByte; } rc = sqlite3_blob_read(p->pSegments, aByte, nByte, 0); memset(&aByte[nByte], 0, FTS3_NODE_PADDING); if( rc!=SQLITE_OK ){ sqlite3_free(aByte); aByte = 0; } } *paBlob = aByte; } } return rc; } /* ** Close the blob handle at p->pSegments, if it is open. See comments above ** the sqlite3Fts3ReadBlock() function for details. */ void sqlite3Fts3SegmentsClose(Fts3Table *p){ sqlite3_blob_close(p->pSegments); p->pSegments = 0; } static int fts3SegReaderIncrRead(Fts3SegReader *pReader){ int nRead; /* Number of bytes to read */ int rc; /* Return code */ nRead = MIN(pReader->nNode - pReader->nPopulate, FTS3_NODE_CHUNKSIZE); rc = sqlite3_blob_read( pReader->pBlob, &pReader->aNode[pReader->nPopulate], nRead, pReader->nPopulate ); if( rc==SQLITE_OK ){ pReader->nPopulate += nRead; memset(&pReader->aNode[pReader->nPopulate], 0, FTS3_NODE_PADDING); if( pReader->nPopulate==pReader->nNode ){ sqlite3_blob_close(pReader->pBlob); pReader->pBlob = 0; pReader->nPopulate = 0; } } return rc; } static int fts3SegReaderRequire(Fts3SegReader *pReader, char *pFrom, int nByte){ int rc = SQLITE_OK; assert( !pReader->pBlob || (pFrom>=pReader->aNode && pFrom<&pReader->aNode[pReader->nNode]) ); while( pReader->pBlob && rc==SQLITE_OK && (pFrom - pReader->aNode + nByte)>pReader->nPopulate ){ rc = fts3SegReaderIncrRead(pReader); } return rc; } /* ** Move the iterator passed as the first argument to the next term in the ** segment. If successful, SQLITE_OK is returned. If there is no next term, ** SQLITE_DONE. Otherwise, an SQLite error code. */ static int fts3SegReaderNext( Fts3Table *p, Fts3SegReader *pReader, int bIncr ){ int rc; /* Return code of various sub-routines */ char *pNext; /* Cursor variable */ int nPrefix; /* Number of bytes in term prefix */ int nSuffix; /* Number of bytes in term suffix */ if( !pReader->aDoclist ){ pNext = pReader->aNode; }else{ pNext = &pReader->aDoclist[pReader->nDoclist]; } if( !pNext || pNext>=&pReader->aNode[pReader->nNode] ){ if( fts3SegReaderIsPending(pReader) ){ Fts3HashElem *pElem = *(pReader->ppNextElem); if( pElem==0 ){ pReader->aNode = 0; }else{ PendingList *pList = (PendingList *)fts3HashData(pElem); pReader->zTerm = (char *)fts3HashKey(pElem); pReader->nTerm = fts3HashKeysize(pElem); pReader->nNode = pReader->nDoclist = pList->nData + 1; pReader->aNode = pReader->aDoclist = pList->aData; pReader->ppNextElem++; assert( pReader->aNode ); } return SQLITE_OK; } if( !fts3SegReaderIsRootOnly(pReader) ){ sqlite3_free(pReader->aNode); sqlite3_blob_close(pReader->pBlob); pReader->pBlob = 0; } pReader->aNode = 0; /* If iCurrentBlock>=iLeafEndBlock, this is an EOF condition. All leaf ** blocks have already been traversed. */ assert( pReader->iCurrentBlock<=pReader->iLeafEndBlock ); if( pReader->iCurrentBlock>=pReader->iLeafEndBlock ){ return SQLITE_OK; } rc = sqlite3Fts3ReadBlock( p, ++pReader->iCurrentBlock, &pReader->aNode, &pReader->nNode, (bIncr ? &pReader->nPopulate : 0) ); if( rc!=SQLITE_OK ) return rc; assert( pReader->pBlob==0 ); if( bIncr && pReader->nPopulate<pReader->nNode ){ pReader->pBlob = p->pSegments; p->pSegments = 0; } pNext = pReader->aNode; } assert( !fts3SegReaderIsPending(pReader) ); rc = fts3SegReaderRequire(pReader, pNext, FTS3_VARINT_MAX*2); if( rc!=SQLITE_OK ) return rc; /* Because of the FTS3_NODE_PADDING bytes of padding, the following is ** safe (no risk of overread) even if the node data is corrupted. */ pNext += sqlite3Fts3GetVarint32(pNext, &nPrefix); pNext += sqlite3Fts3GetVarint32(pNext, &nSuffix); if( nPrefix<0 || nSuffix<=0 || &pNext[nSuffix]>&pReader->aNode[pReader->nNode] ){ return SQLITE_CORRUPT_VTAB; } if( nPrefix+nSuffix>pReader->nTermAlloc ){ int nNew = (nPrefix+nSuffix)*2; char *zNew = sqlite3_realloc(pReader->zTerm, nNew); if( !zNew ){ return SQLITE_NOMEM; } pReader->zTerm = zNew; pReader->nTermAlloc = nNew; } rc = fts3SegReaderRequire(pReader, pNext, nSuffix+FTS3_VARINT_MAX); if( rc!=SQLITE_OK ) return rc; memcpy(&pReader->zTerm[nPrefix], pNext, nSuffix); pReader->nTerm = nPrefix+nSuffix; pNext += nSuffix; pNext += sqlite3Fts3GetVarint32(pNext, &pReader->nDoclist); pReader->aDoclist = pNext; pReader->pOffsetList = 0; /* Check that the doclist does not appear to extend past the end of the ** b-tree node. And that the final byte of the doclist is 0x00. If either ** of these statements is untrue, then the data structure is corrupt. */ if( &pReader->aDoclist[pReader->nDoclist]>&pReader->aNode[pReader->nNode] || (pReader->nPopulate==0 && pReader->aDoclist[pReader->nDoclist-1]) ){ return SQLITE_CORRUPT_VTAB; } return SQLITE_OK; } /* ** Set the SegReader to point to the first docid in the doclist associated ** with the current term. */ static int fts3SegReaderFirstDocid(Fts3Table *pTab, Fts3SegReader *pReader){ int rc = SQLITE_OK; assert( pReader->aDoclist ); assert( !pReader->pOffsetList ); if( pTab->bDescIdx && fts3SegReaderIsPending(pReader) ){ u8 bEof = 0; pReader->iDocid = 0; pReader->nOffsetList = 0; sqlite3Fts3DoclistPrev(0, pReader->aDoclist, pReader->nDoclist, &pReader->pOffsetList, &pReader->iDocid, &pReader->nOffsetList, &bEof ); }else{ rc = fts3SegReaderRequire(pReader, pReader->aDoclist, FTS3_VARINT_MAX); if( rc==SQLITE_OK ){ int n = sqlite3Fts3GetVarint(pReader->aDoclist, &pReader->iDocid); pReader->pOffsetList = &pReader->aDoclist[n]; } } return rc; } /* ** Advance the SegReader to point to the next docid in the doclist ** associated with the current term. ** ** If arguments ppOffsetList and pnOffsetList are not NULL, then ** *ppOffsetList is set to point to the first column-offset list ** in the doclist entry (i.e. immediately past the docid varint). ** *pnOffsetList is set to the length of the set of column-offset ** lists, not including the nul-terminator byte. For example: */ static int fts3SegReaderNextDocid( Fts3Table *pTab, Fts3SegReader *pReader, /* Reader to advance to next docid */ char **ppOffsetList, /* OUT: Pointer to current position-list */ int *pnOffsetList /* OUT: Length of *ppOffsetList in bytes */ ){ int rc = SQLITE_OK; char *p = pReader->pOffsetList; char c = 0; assert( p ); if( pTab->bDescIdx && fts3SegReaderIsPending(pReader) ){ /* A pending-terms seg-reader for an FTS4 table that uses order=desc. ** Pending-terms doclists are always built up in ascending order, so ** we have to iterate through them backwards here. */ u8 bEof = 0; if( ppOffsetList ){ *ppOffsetList = pReader->pOffsetList; *pnOffsetList = pReader->nOffsetList - 1; } sqlite3Fts3DoclistPrev(0, pReader->aDoclist, pReader->nDoclist, &p, &pReader->iDocid, &pReader->nOffsetList, &bEof ); if( bEof ){ pReader->pOffsetList = 0; }else{ pReader->pOffsetList = p; } }else{ char *pEnd = &pReader->aDoclist[pReader->nDoclist]; /* Pointer p currently points at the first byte of an offset list. The ** following block advances it to point one byte past the end of ** the same offset list. */ while( 1 ){ /* The following line of code (and the "p++" below the while() loop) is ** normally all that is required to move pointer p to the desired ** position. The exception is if this node is being loaded from disk ** incrementally and pointer "p" now points to the first byte passed ** the populated part of pReader->aNode[]. */ while( *p | c ) c = *p++ & 0x80; assert( *p==0 ); if( pReader->pBlob==0 || p<&pReader->aNode[pReader->nPopulate] ) break; rc = fts3SegReaderIncrRead(pReader); if( rc!=SQLITE_OK ) return rc; } p++; /* If required, populate the output variables with a pointer to and the ** size of the previous offset-list. */ if( ppOffsetList ){ *ppOffsetList = pReader->pOffsetList; *pnOffsetList = (int)(p - pReader->pOffsetList - 1); } while( p<pEnd && *p==0 ) p++; /* If there are no more entries in the doclist, set pOffsetList to ** NULL. Otherwise, set Fts3SegReader.iDocid to the next docid and ** Fts3SegReader.pOffsetList to point to the next offset list before ** returning. */ if( p>=pEnd ){ pReader->pOffsetList = 0; }else{ rc = fts3SegReaderRequire(pReader, p, FTS3_VARINT_MAX); if( rc==SQLITE_OK ){ sqlite3_int64 iDelta; pReader->pOffsetList = p + sqlite3Fts3GetVarint(p, &iDelta); if( pTab->bDescIdx ){ pReader->iDocid -= iDelta; }else{ pReader->iDocid += iDelta; } } } } return SQLITE_OK; } int sqlite3Fts3MsrOvfl( Fts3Cursor *pCsr, Fts3MultiSegReader *pMsr, int *pnOvfl ){ Fts3Table *p = (Fts3Table*)pCsr->base.pVtab; int nOvfl = 0; int ii; int rc = SQLITE_OK; int pgsz = p->nPgsz; assert( p->bHasStat ); assert( pgsz>0 ); for(ii=0; rc==SQLITE_OK && ii<pMsr->nSegment; ii++){ Fts3SegReader *pReader = pMsr->apSegment[ii]; if( !fts3SegReaderIsPending(pReader) && !fts3SegReaderIsRootOnly(pReader) ){ int jj; for(jj=pReader->iStartBlock; jj<=pReader->iLeafEndBlock; jj++){ int nBlob; rc = sqlite3Fts3ReadBlock(p, jj, 0, &nBlob, 0); if( rc!=SQLITE_OK ) break; if( (nBlob+35)>pgsz ){ nOvfl += (nBlob + 34)/pgsz; } } } } *pnOvfl = nOvfl; return rc; } /* ** Free all allocations associated with the iterator passed as the ** second argument. */ void sqlite3Fts3SegReaderFree(Fts3SegReader *pReader){ if( pReader && !fts3SegReaderIsPending(pReader) ){ sqlite3_free(pReader->zTerm); if( !fts3SegReaderIsRootOnly(pReader) ){ sqlite3_free(pReader->aNode); sqlite3_blob_close(pReader->pBlob); } } sqlite3_free(pReader); } /* ** Allocate a new SegReader object. |
︙ | ︙ | |||
1231 1232 1233 1234 1235 1236 1237 1238 1239 1240 1241 1242 | } return c; } /* ** This function is used to allocate an Fts3SegReader that iterates through ** a subset of the terms stored in the Fts3Table.pendingTerms array. */ int sqlite3Fts3SegReaderPending( Fts3Table *p, /* Virtual table handle */ const char *zTerm, /* Term to search for */ int nTerm, /* Size of buffer zTerm */ | > > > > > > > > > > > > > > > > | > > | | > > > | | | 1420 1421 1422 1423 1424 1425 1426 1427 1428 1429 1430 1431 1432 1433 1434 1435 1436 1437 1438 1439 1440 1441 1442 1443 1444 1445 1446 1447 1448 1449 1450 1451 1452 1453 1454 1455 1456 1457 1458 1459 1460 1461 1462 1463 1464 1465 1466 1467 1468 1469 1470 1471 1472 1473 1474 1475 1476 1477 1478 1479 1480 1481 1482 1483 1484 1485 1486 1487 1488 1489 1490 1491 1492 1493 1494 1495 1496 1497 1498 1499 1500 1501 1502 1503 1504 1505 1506 1507 1508 1509 1510 1511 1512 1513 1514 1515 1516 1517 1518 1519 1520 1521 1522 | } return c; } /* ** This function is used to allocate an Fts3SegReader that iterates through ** a subset of the terms stored in the Fts3Table.pendingTerms array. ** ** If the isPrefixIter parameter is zero, then the returned SegReader iterates ** through each term in the pending-terms table. Or, if isPrefixIter is ** non-zero, it iterates through each term and its prefixes. For example, if ** the pending terms hash table contains the terms "sqlite", "mysql" and ** "firebird", then the iterator visits the following 'terms' (in the order ** shown): ** ** f fi fir fire fireb firebi firebir firebird ** m my mys mysq mysql ** s sq sql sqli sqlit sqlite ** ** Whereas if isPrefixIter is zero, the terms visited are: ** ** firebird mysql sqlite */ int sqlite3Fts3SegReaderPending( Fts3Table *p, /* Virtual table handle */ int iIndex, /* Index for p->aIndex */ const char *zTerm, /* Term to search for */ int nTerm, /* Size of buffer zTerm */ int bPrefix, /* True for a prefix iterator */ Fts3SegReader **ppReader /* OUT: SegReader for pending-terms */ ){ Fts3SegReader *pReader = 0; /* Fts3SegReader object to return */ Fts3HashElem **aElem = 0; /* Array of term hash entries to scan */ int nElem = 0; /* Size of array at aElem */ int rc = SQLITE_OK; /* Return Code */ Fts3Hash *pHash; pHash = &p->aIndex[iIndex].hPending; if( bPrefix ){ int nAlloc = 0; /* Size of allocated array at aElem */ Fts3HashElem *pE = 0; /* Iterator variable */ for(pE=fts3HashFirst(pHash); pE; pE=fts3HashNext(pE)){ char *zKey = (char *)fts3HashKey(pE); int nKey = fts3HashKeysize(pE); if( nTerm==0 || (nKey>=nTerm && 0==memcmp(zKey, zTerm, nTerm)) ){ if( nElem==nAlloc ){ Fts3HashElem **aElem2; nAlloc += 16; aElem2 = (Fts3HashElem **)sqlite3_realloc( aElem, nAlloc*sizeof(Fts3HashElem *) ); if( !aElem2 ){ rc = SQLITE_NOMEM; nElem = 0; break; } aElem = aElem2; } aElem[nElem++] = pE; } } /* If more than one term matches the prefix, sort the Fts3HashElem ** objects in term order using qsort(). This uses the same comparison ** callback as is used when flushing terms to disk. */ if( nElem>1 ){ qsort(aElem, nElem, sizeof(Fts3HashElem *), fts3CompareElemByTerm); } }else{ /* The query is a simple term lookup that matches at most one term in ** the index. All that is required is a straight hash-lookup. */ Fts3HashElem *pE = fts3HashFindElem(pHash, zTerm, nTerm); if( pE ){ aElem = &pE; nElem = 1; } } if( nElem>0 ){ int nByte = sizeof(Fts3SegReader) + (nElem+1)*sizeof(Fts3HashElem *); pReader = (Fts3SegReader *)sqlite3_malloc(nByte); if( !pReader ){ rc = SQLITE_NOMEM; }else{ memset(pReader, 0, nByte); pReader->iIdx = 0x7FFFFFFF; pReader->ppNextElem = (Fts3HashElem **)&pReader[1]; memcpy(pReader->ppNextElem, aElem, nElem*sizeof(Fts3HashElem *)); } } if( bPrefix ){ sqlite3_free(aElem); } *ppReader = pReader; return rc; } /* |
︙ | ︙ | |||
1358 1359 1360 1361 1362 1363 1364 1365 1366 1367 1368 1369 1370 1371 | int rc = (pLhs->pOffsetList==0)-(pRhs->pOffsetList==0); if( rc==0 ){ if( pLhs->iDocid==pRhs->iDocid ){ rc = pRhs->iIdx - pLhs->iIdx; }else{ rc = (pLhs->iDocid > pRhs->iDocid) ? 1 : -1; } } assert( pLhs->aNode && pRhs->aNode ); return rc; } /* ** Compare the term that the Fts3SegReader object passed as the first argument | > > > > > > > > > > > > | 1568 1569 1570 1571 1572 1573 1574 1575 1576 1577 1578 1579 1580 1581 1582 1583 1584 1585 1586 1587 1588 1589 1590 1591 1592 1593 | int rc = (pLhs->pOffsetList==0)-(pRhs->pOffsetList==0); if( rc==0 ){ if( pLhs->iDocid==pRhs->iDocid ){ rc = pRhs->iIdx - pLhs->iIdx; }else{ rc = (pLhs->iDocid > pRhs->iDocid) ? 1 : -1; } } assert( pLhs->aNode && pRhs->aNode ); return rc; } static int fts3SegReaderDoclistCmpRev(Fts3SegReader *pLhs, Fts3SegReader *pRhs){ int rc = (pLhs->pOffsetList==0)-(pRhs->pOffsetList==0); if( rc==0 ){ if( pLhs->iDocid==pRhs->iDocid ){ rc = pRhs->iIdx - pLhs->iIdx; }else{ rc = (pLhs->iDocid < pRhs->iDocid) ? 1 : -1; } } assert( pLhs->aNode && pRhs->aNode ); return rc; } /* ** Compare the term that the Fts3SegReader object passed as the first argument |
︙ | ︙ | |||
1910 1911 1912 1913 1914 1915 1916 | } rc = sqlite3_reset(pStmt); } return rc; } /* | < | > > | | > > > > > > > | > > < | > | 2132 2133 2134 2135 2136 2137 2138 2139 2140 2141 2142 2143 2144 2145 2146 2147 2148 2149 2150 2151 2152 2153 2154 2155 2156 2157 2158 2159 2160 2161 2162 2163 2164 2165 2166 2167 2168 2169 2170 2171 2172 2173 2174 2175 2176 2177 2178 2179 2180 2181 2182 2183 2184 2185 2186 2187 2188 2189 2190 | } rc = sqlite3_reset(pStmt); } return rc; } /* ** Set *pnMax to the largest segment level in the database for the index ** iIndex. ** ** Segment levels are stored in the 'level' column of the %_segdir table. ** ** Return SQLITE_OK if successful, or an SQLite error code if not. */ static int fts3SegmentMaxLevel(Fts3Table *p, int iIndex, int *pnMax){ sqlite3_stmt *pStmt; int rc; assert( iIndex>=0 && iIndex<p->nIndex ); /* Set pStmt to the compiled version of: ** ** SELECT max(level) FROM %Q.'%q_segdir' WHERE level BETWEEN ? AND ? ** ** (1024 is actually the value of macro FTS3_SEGDIR_PREFIXLEVEL_STR). */ rc = fts3SqlStmt(p, SQL_SELECT_SEGDIR_MAX_LEVEL, &pStmt, 0); if( rc!=SQLITE_OK ) return rc; sqlite3_bind_int(pStmt, 1, iIndex*FTS3_SEGDIR_MAXLEVEL); sqlite3_bind_int(pStmt, 2, (iIndex+1)*FTS3_SEGDIR_MAXLEVEL - 1); if( SQLITE_ROW==sqlite3_step(pStmt) ){ *pnMax = sqlite3_column_int(pStmt, 0); } return sqlite3_reset(pStmt); } /* ** This function is used after merging multiple segments into a single large ** segment to delete the old, now redundant, segment b-trees. Specifically, ** it: ** ** 1) Deletes all %_segments entries for the segments associated with ** each of the SegReader objects in the array passed as the third ** argument, and ** ** 2) deletes all %_segdir entries with level iLevel, or all %_segdir ** entries regardless of level if (iLevel<0). ** ** SQLITE_OK is returned if successful, otherwise an SQLite error code. */ static int fts3DeleteSegdir( Fts3Table *p, /* Virtual table handle */ int iIndex, /* Index for p->aIndex */ int iLevel, /* Level of %_segdir entries to delete */ Fts3SegReader **apSegment, /* Array of SegReader objects */ int nReader /* Size of array apSegment */ ){ int rc; /* Return Code */ int i; /* Iterator variable */ sqlite3_stmt *pDelete; /* SQL statement to delete rows */ |
︙ | ︙ | |||
1967 1968 1969 1970 1971 1972 1973 1974 | rc = sqlite3_reset(pDelete); } } if( rc!=SQLITE_OK ){ return rc; } if( iLevel==FTS3_SEGCURSOR_ALL ){ | > | > | | > < | | > > > > | | | < | 2199 2200 2201 2202 2203 2204 2205 2206 2207 2208 2209 2210 2211 2212 2213 2214 2215 2216 2217 2218 2219 2220 2221 2222 2223 2224 2225 2226 2227 2228 2229 2230 | rc = sqlite3_reset(pDelete); } } if( rc!=SQLITE_OK ){ return rc; } assert( iLevel>=0 || iLevel==FTS3_SEGCURSOR_ALL ); if( iLevel==FTS3_SEGCURSOR_ALL ){ rc = fts3SqlStmt(p, SQL_DELETE_SEGDIR_RANGE, &pDelete, 0); if( rc==SQLITE_OK ){ sqlite3_bind_int(pDelete, 1, iIndex*FTS3_SEGDIR_MAXLEVEL); sqlite3_bind_int(pDelete, 2, (iIndex+1) * FTS3_SEGDIR_MAXLEVEL - 1); } }else{ rc = fts3SqlStmt(p, SQL_DELETE_SEGDIR_LEVEL, &pDelete, 0); if( rc==SQLITE_OK ){ sqlite3_bind_int(pDelete, 1, iIndex*FTS3_SEGDIR_MAXLEVEL + iLevel); } } if( rc==SQLITE_OK ){ sqlite3_step(pDelete); rc = sqlite3_reset(pDelete); } return rc; } /* ** When this function is called, buffer *ppList (size *pnList bytes) contains ** a position list that may (or may not) feature multiple columns. This |
︙ | ︙ | |||
2027 2028 2029 2030 2031 2032 2033 | p += sqlite3Fts3GetVarint32(p, &iCurrent); } *ppList = pList; *pnList = nList; } | > > > > > > > > > > > > > > > > > > > > > > > > | > > > > > > > > > > > > > > > > > | > | > > > > > > > > > > > > > > > > | > > > > > | > > | | > > > > > > > > > > > > > > > | > > > > > > > > > > > | < < | > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > | | > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > | > > > | | 2264 2265 2266 2267 2268 2269 2270 2271 2272 2273 2274 2275 2276 2277 2278 2279 2280 2281 2282 2283 2284 2285 2286 2287 2288 2289 2290 2291 2292 2293 2294 2295 2296 2297 2298 2299 2300 2301 2302 2303 2304 2305 2306 2307 2308 2309 2310 2311 2312 2313 2314 2315 2316 2317 2318 2319 2320 2321 2322 2323 2324 2325 2326 2327 2328 2329 2330 2331 2332 2333 2334 2335 2336 2337 2338 2339 2340 2341 2342 2343 2344 2345 2346 2347 2348 2349 2350 2351 2352 2353 2354 2355 2356 2357 2358 2359 2360 2361 2362 2363 2364 2365 2366 2367 2368 2369 2370 2371 2372 2373 2374 2375 2376 2377 2378 2379 2380 2381 2382 2383 2384 2385 2386 2387 2388 2389 2390 2391 2392 2393 2394 2395 2396 2397 2398 2399 2400 2401 2402 2403 2404 2405 2406 2407 2408 2409 2410 2411 2412 2413 2414 2415 2416 2417 2418 2419 2420 2421 2422 2423 2424 2425 2426 2427 2428 2429 2430 2431 2432 2433 2434 2435 2436 2437 2438 2439 2440 2441 2442 2443 2444 2445 2446 2447 2448 2449 2450 2451 2452 2453 2454 2455 2456 2457 2458 2459 2460 2461 2462 2463 2464 2465 2466 2467 2468 2469 2470 2471 2472 2473 2474 2475 2476 2477 2478 2479 2480 2481 2482 2483 2484 2485 2486 2487 2488 2489 2490 2491 2492 2493 2494 2495 2496 2497 2498 2499 2500 2501 2502 2503 2504 2505 2506 2507 2508 2509 2510 | p += sqlite3Fts3GetVarint32(p, &iCurrent); } *ppList = pList; *pnList = nList; } /* ** Cache data in the Fts3MultiSegReader.aBuffer[] buffer (overwriting any ** existing data). Grow the buffer if required. ** ** If successful, return SQLITE_OK. Otherwise, if an OOM error is encountered ** trying to resize the buffer, return SQLITE_NOMEM. */ static int fts3MsrBufferData( Fts3MultiSegReader *pMsr, /* Multi-segment-reader handle */ char *pList, int nList ){ if( nList>pMsr->nBuffer ){ char *pNew; pMsr->nBuffer = nList*2; pNew = (char *)sqlite3_realloc(pMsr->aBuffer, pMsr->nBuffer); if( !pNew ) return SQLITE_NOMEM; pMsr->aBuffer = pNew; } memcpy(pMsr->aBuffer, pList, nList); return SQLITE_OK; } int sqlite3Fts3MsrIncrNext( Fts3Table *p, /* Virtual table handle */ Fts3MultiSegReader *pMsr, /* Multi-segment-reader handle */ sqlite3_int64 *piDocid, /* OUT: Docid value */ char **paPoslist, /* OUT: Pointer to position list */ int *pnPoslist /* OUT: Size of position list in bytes */ ){ int nMerge = pMsr->nAdvance; Fts3SegReader **apSegment = pMsr->apSegment; int (*xCmp)(Fts3SegReader *, Fts3SegReader *) = ( p->bDescIdx ? fts3SegReaderDoclistCmpRev : fts3SegReaderDoclistCmp ); if( nMerge==0 ){ *paPoslist = 0; return SQLITE_OK; } while( 1 ){ Fts3SegReader *pSeg; pSeg = pMsr->apSegment[0]; if( pSeg->pOffsetList==0 ){ *paPoslist = 0; break; }else{ int rc; char *pList; int nList; int j; sqlite3_int64 iDocid = apSegment[0]->iDocid; rc = fts3SegReaderNextDocid(p, apSegment[0], &pList, &nList); j = 1; while( rc==SQLITE_OK && j<nMerge && apSegment[j]->pOffsetList && apSegment[j]->iDocid==iDocid ){ rc = fts3SegReaderNextDocid(p, apSegment[j], 0, 0); j++; } if( rc!=SQLITE_OK ) return rc; fts3SegReaderSort(pMsr->apSegment, nMerge, j, xCmp); if( pMsr->iColFilter>=0 ){ fts3ColumnFilter(pMsr->iColFilter, &pList, &nList); } if( nList>0 ){ if( fts3SegReaderIsPending(apSegment[0]) ){ rc = fts3MsrBufferData(pMsr, pList, nList+1); if( rc!=SQLITE_OK ) return rc; *paPoslist = pMsr->aBuffer; assert( (pMsr->aBuffer[nList] & 0xFE)==0x00 ); }else{ *paPoslist = pList; } *piDocid = iDocid; *pnPoslist = nList; break; } } } return SQLITE_OK; } static int fts3SegReaderStart( Fts3Table *p, /* Virtual table handle */ Fts3MultiSegReader *pCsr, /* Cursor object */ const char *zTerm, /* Term searched for (or NULL) */ int nTerm /* Length of zTerm in bytes */ ){ int i; int nSeg = pCsr->nSegment; /* If the Fts3SegFilter defines a specific term (or term prefix) to search ** for, then advance each segment iterator until it points to a term of ** equal or greater value than the specified term. This prevents many ** unnecessary merge/sort operations for the case where single segment ** b-tree leaf nodes contain more than one term. */ for(i=0; pCsr->bRestart==0 && i<pCsr->nSegment; i++){ Fts3SegReader *pSeg = pCsr->apSegment[i]; do { int rc = fts3SegReaderNext(p, pSeg, 0); if( rc!=SQLITE_OK ) return rc; }while( zTerm && fts3SegReaderTermCmp(pSeg, zTerm, nTerm)<0 ); } fts3SegReaderSort(pCsr->apSegment, nSeg, nSeg, fts3SegReaderCmp); return SQLITE_OK; } int sqlite3Fts3SegReaderStart( Fts3Table *p, /* Virtual table handle */ Fts3MultiSegReader *pCsr, /* Cursor object */ Fts3SegFilter *pFilter /* Restrictions on range of iteration */ ){ pCsr->pFilter = pFilter; return fts3SegReaderStart(p, pCsr, pFilter->zTerm, pFilter->nTerm); } int sqlite3Fts3MsrIncrStart( Fts3Table *p, /* Virtual table handle */ Fts3MultiSegReader *pCsr, /* Cursor object */ int iCol, /* Column to match on. */ const char *zTerm, /* Term to iterate through a doclist for */ int nTerm /* Number of bytes in zTerm */ ){ int i; int rc; int nSegment = pCsr->nSegment; int (*xCmp)(Fts3SegReader *, Fts3SegReader *) = ( p->bDescIdx ? fts3SegReaderDoclistCmpRev : fts3SegReaderDoclistCmp ); assert( pCsr->pFilter==0 ); assert( zTerm && nTerm>0 ); /* Advance each segment iterator until it points to the term zTerm/nTerm. */ rc = fts3SegReaderStart(p, pCsr, zTerm, nTerm); if( rc!=SQLITE_OK ) return rc; /* Determine how many of the segments actually point to zTerm/nTerm. */ for(i=0; i<nSegment; i++){ Fts3SegReader *pSeg = pCsr->apSegment[i]; if( !pSeg->aNode || fts3SegReaderTermCmp(pSeg, zTerm, nTerm) ){ break; } } pCsr->nAdvance = i; /* Advance each of the segments to point to the first docid. */ for(i=0; i<pCsr->nAdvance; i++){ rc = fts3SegReaderFirstDocid(p, pCsr->apSegment[i]); if( rc!=SQLITE_OK ) return rc; } fts3SegReaderSort(pCsr->apSegment, i, i, xCmp); assert( iCol<0 || iCol<p->nColumn ); pCsr->iColFilter = iCol; return SQLITE_OK; } /* ** This function is called on a MultiSegReader that has been started using ** sqlite3Fts3MsrIncrStart(). One or more calls to MsrIncrNext() may also ** have been made. Calling this function puts the MultiSegReader in such ** a state that if the next two calls are: ** ** sqlite3Fts3SegReaderStart() ** sqlite3Fts3SegReaderStep() ** ** then the entire doclist for the term is available in ** MultiSegReader.aDoclist/nDoclist. */ int sqlite3Fts3MsrIncrRestart(Fts3MultiSegReader *pCsr){ int i; /* Used to iterate through segment-readers */ assert( pCsr->zTerm==0 ); assert( pCsr->nTerm==0 ); assert( pCsr->aDoclist==0 ); assert( pCsr->nDoclist==0 ); pCsr->nAdvance = 0; pCsr->bRestart = 1; for(i=0; i<pCsr->nSegment; i++){ pCsr->apSegment[i]->pOffsetList = 0; pCsr->apSegment[i]->nOffsetList = 0; pCsr->apSegment[i]->iDocid = 0; } return SQLITE_OK; } int sqlite3Fts3SegReaderStep( Fts3Table *p, /* Virtual table handle */ Fts3MultiSegReader *pCsr /* Cursor object */ ){ int rc = SQLITE_OK; int isIgnoreEmpty = (pCsr->pFilter->flags & FTS3_SEGMENT_IGNORE_EMPTY); int isRequirePos = (pCsr->pFilter->flags & FTS3_SEGMENT_REQUIRE_POS); int isColFilter = (pCsr->pFilter->flags & FTS3_SEGMENT_COLUMN_FILTER); int isPrefix = (pCsr->pFilter->flags & FTS3_SEGMENT_PREFIX); int isScan = (pCsr->pFilter->flags & FTS3_SEGMENT_SCAN); Fts3SegReader **apSegment = pCsr->apSegment; int nSegment = pCsr->nSegment; Fts3SegFilter *pFilter = pCsr->pFilter; int (*xCmp)(Fts3SegReader *, Fts3SegReader *) = ( p->bDescIdx ? fts3SegReaderDoclistCmpRev : fts3SegReaderDoclistCmp ); if( pCsr->nSegment==0 ) return SQLITE_OK; do { int nMerge; int i; /* Advance the first pCsr->nAdvance entries in the apSegment[] array ** forward. Then sort the list in order of current term again. */ for(i=0; i<pCsr->nAdvance; i++){ rc = fts3SegReaderNext(p, apSegment[i], 0); if( rc!=SQLITE_OK ) return rc; } fts3SegReaderSort(apSegment, nSegment, pCsr->nAdvance, fts3SegReaderCmp); pCsr->nAdvance = 0; /* If all the seg-readers are at EOF, we're finished. return SQLITE_OK. */ assert( rc==SQLITE_OK ); |
︙ | ︙ | |||
2123 2124 2125 2126 2127 2128 2129 | && apSegment[nMerge]->nTerm==pCsr->nTerm && 0==memcmp(pCsr->zTerm, apSegment[nMerge]->zTerm, pCsr->nTerm) ){ nMerge++; } assert( isIgnoreEmpty || (isRequirePos && !isColFilter) ); | | > > > > > > | > | > | | | | | > > > > > > > > > > > > | | < < | > | | 2535 2536 2537 2538 2539 2540 2541 2542 2543 2544 2545 2546 2547 2548 2549 2550 2551 2552 2553 2554 2555 2556 2557 2558 2559 2560 2561 2562 2563 2564 2565 2566 2567 2568 2569 2570 2571 2572 2573 2574 2575 2576 2577 2578 2579 2580 2581 2582 2583 2584 2585 2586 2587 2588 2589 2590 2591 2592 2593 2594 2595 2596 2597 2598 2599 2600 2601 2602 2603 2604 2605 2606 2607 2608 2609 2610 2611 2612 2613 2614 2615 2616 2617 2618 2619 2620 2621 2622 2623 2624 2625 2626 2627 2628 2629 2630 2631 2632 2633 2634 2635 2636 2637 2638 2639 2640 2641 | && apSegment[nMerge]->nTerm==pCsr->nTerm && 0==memcmp(pCsr->zTerm, apSegment[nMerge]->zTerm, pCsr->nTerm) ){ nMerge++; } assert( isIgnoreEmpty || (isRequirePos && !isColFilter) ); if( nMerge==1 && !isIgnoreEmpty && (p->bDescIdx==0 || fts3SegReaderIsPending(apSegment[0])==0) ){ pCsr->nDoclist = apSegment[0]->nDoclist; if( fts3SegReaderIsPending(apSegment[0]) ){ rc = fts3MsrBufferData(pCsr, apSegment[0]->aDoclist, pCsr->nDoclist); pCsr->aDoclist = pCsr->aBuffer; }else{ pCsr->aDoclist = apSegment[0]->aDoclist; } if( rc==SQLITE_OK ) rc = SQLITE_ROW; }else{ int nDoclist = 0; /* Size of doclist */ sqlite3_int64 iPrev = 0; /* Previous docid stored in doclist */ /* The current term of the first nMerge entries in the array ** of Fts3SegReader objects is the same. The doclists must be merged ** and a single term returned with the merged doclist. */ for(i=0; i<nMerge; i++){ fts3SegReaderFirstDocid(p, apSegment[i]); } fts3SegReaderSort(apSegment, nMerge, nMerge, xCmp); while( apSegment[0]->pOffsetList ){ int j; /* Number of segments that share a docid */ char *pList; int nList; int nByte; sqlite3_int64 iDocid = apSegment[0]->iDocid; fts3SegReaderNextDocid(p, apSegment[0], &pList, &nList); j = 1; while( j<nMerge && apSegment[j]->pOffsetList && apSegment[j]->iDocid==iDocid ){ fts3SegReaderNextDocid(p, apSegment[j], 0, 0); j++; } if( isColFilter ){ fts3ColumnFilter(pFilter->iCol, &pList, &nList); } if( !isIgnoreEmpty || nList>0 ){ /* Calculate the 'docid' delta value to write into the merged ** doclist. */ sqlite3_int64 iDelta; if( p->bDescIdx && nDoclist>0 ){ iDelta = iPrev - iDocid; }else{ iDelta = iDocid - iPrev; } assert( iDelta>0 || (nDoclist==0 && iDelta==iDocid) ); assert( nDoclist>0 || iDelta==iDocid ); nByte = sqlite3Fts3VarintLen(iDelta) + (isRequirePos?nList+1:0); if( nDoclist+nByte>pCsr->nBuffer ){ char *aNew; pCsr->nBuffer = (nDoclist+nByte)*2; aNew = sqlite3_realloc(pCsr->aBuffer, pCsr->nBuffer); if( !aNew ){ return SQLITE_NOMEM; } pCsr->aBuffer = aNew; } nDoclist += sqlite3Fts3PutVarint(&pCsr->aBuffer[nDoclist], iDelta); iPrev = iDocid; if( isRequirePos ){ memcpy(&pCsr->aBuffer[nDoclist], pList, nList); nDoclist += nList; pCsr->aBuffer[nDoclist++] = '\0'; } } fts3SegReaderSort(apSegment, nMerge, j, xCmp); } if( nDoclist>0 ){ pCsr->aDoclist = pCsr->aBuffer; pCsr->nDoclist = nDoclist; rc = SQLITE_ROW; } } pCsr->nAdvance = nMerge; }while( rc==SQLITE_OK ); return rc; } void sqlite3Fts3SegReaderFinish( Fts3MultiSegReader *pCsr /* Cursor object */ ){ if( pCsr ){ int i; for(i=0; i<pCsr->nSegment; i++){ sqlite3Fts3SegReaderFree(pCsr->apSegment[i]); } sqlite3_free(pCsr->apSegment); |
︙ | ︙ | |||
2223 2224 2225 2226 2227 2228 2229 | ** currently present in the database. ** ** If this function is called with iLevel<0, but there is only one ** segment in the database, SQLITE_DONE is returned immediately. ** Otherwise, if successful, SQLITE_OK is returned. If an error occurs, ** an SQLite error code is returned. */ | | | | > > > > > > > > | | | < | > > > > > | < | > | > | > | | > | > > > | > > > > | 2654 2655 2656 2657 2658 2659 2660 2661 2662 2663 2664 2665 2666 2667 2668 2669 2670 2671 2672 2673 2674 2675 2676 2677 2678 2679 2680 2681 2682 2683 2684 2685 2686 2687 2688 2689 2690 2691 2692 2693 2694 2695 2696 2697 2698 2699 2700 2701 2702 2703 2704 2705 2706 2707 2708 2709 2710 2711 2712 2713 2714 2715 2716 2717 2718 2719 2720 2721 2722 2723 2724 2725 2726 2727 2728 2729 2730 2731 2732 2733 2734 2735 2736 2737 2738 2739 2740 2741 2742 2743 2744 2745 2746 2747 2748 2749 2750 2751 2752 2753 | ** currently present in the database. ** ** If this function is called with iLevel<0, but there is only one ** segment in the database, SQLITE_DONE is returned immediately. ** Otherwise, if successful, SQLITE_OK is returned. If an error occurs, ** an SQLite error code is returned. */ static int fts3SegmentMerge(Fts3Table *p, int iIndex, int iLevel){ int rc; /* Return code */ int iIdx = 0; /* Index of new segment */ int iNewLevel = 0; /* Level/index to create new segment at */ SegmentWriter *pWriter = 0; /* Used to write the new, merged, segment */ Fts3SegFilter filter; /* Segment term filter condition */ Fts3MultiSegReader csr; /* Cursor to iterate through level(s) */ int bIgnoreEmpty = 0; /* True to ignore empty segments */ assert( iLevel==FTS3_SEGCURSOR_ALL || iLevel==FTS3_SEGCURSOR_PENDING || iLevel>=0 ); assert( iLevel<FTS3_SEGDIR_MAXLEVEL ); assert( iIndex>=0 && iIndex<p->nIndex ); rc = sqlite3Fts3SegReaderCursor(p, iIndex, iLevel, 0, 0, 1, 0, &csr); if( rc!=SQLITE_OK || csr.nSegment==0 ) goto finished; if( iLevel==FTS3_SEGCURSOR_ALL ){ /* This call is to merge all segments in the database to a single ** segment. The level of the new segment is equal to the the numerically ** greatest segment level currently present in the database for this ** index. The idx of the new segment is always 0. */ if( csr.nSegment==1 ){ rc = SQLITE_DONE; goto finished; } rc = fts3SegmentMaxLevel(p, iIndex, &iNewLevel); bIgnoreEmpty = 1; }else if( iLevel==FTS3_SEGCURSOR_PENDING ){ iNewLevel = iIndex * FTS3_SEGDIR_MAXLEVEL; rc = fts3AllocateSegdirIdx(p, iIndex, 0, &iIdx); }else{ /* This call is to merge all segments at level iLevel. find the next ** available segment index at level iLevel+1. The call to ** fts3AllocateSegdirIdx() will merge the segments at level iLevel+1 to ** a single iLevel+2 segment if necessary. */ rc = fts3AllocateSegdirIdx(p, iIndex, iLevel+1, &iIdx); iNewLevel = iIndex * FTS3_SEGDIR_MAXLEVEL + iLevel+1; } if( rc!=SQLITE_OK ) goto finished; assert( csr.nSegment>0 ); assert( iNewLevel>=(iIndex*FTS3_SEGDIR_MAXLEVEL) ); assert( iNewLevel<((iIndex+1)*FTS3_SEGDIR_MAXLEVEL) ); memset(&filter, 0, sizeof(Fts3SegFilter)); filter.flags = FTS3_SEGMENT_REQUIRE_POS; filter.flags |= (bIgnoreEmpty ? FTS3_SEGMENT_IGNORE_EMPTY : 0); rc = sqlite3Fts3SegReaderStart(p, &csr, &filter); while( SQLITE_OK==rc ){ rc = sqlite3Fts3SegReaderStep(p, &csr); if( rc!=SQLITE_ROW ) break; rc = fts3SegWriterAdd(p, &pWriter, 1, csr.zTerm, csr.nTerm, csr.aDoclist, csr.nDoclist); } if( rc!=SQLITE_OK ) goto finished; assert( pWriter ); if( iLevel!=FTS3_SEGCURSOR_PENDING ){ rc = fts3DeleteSegdir(p, iIndex, iLevel, csr.apSegment, csr.nSegment); if( rc!=SQLITE_OK ) goto finished; } rc = fts3SegWriterFlush(p, pWriter, iNewLevel, iIdx); finished: fts3SegWriterFree(pWriter); sqlite3Fts3SegReaderFinish(&csr); return rc; } /* ** Flush the contents of pendingTerms to level 0 segments. */ int sqlite3Fts3PendingTermsFlush(Fts3Table *p){ int rc = SQLITE_OK; int i; for(i=0; rc==SQLITE_OK && i<p->nIndex; i++){ rc = fts3SegmentMerge(p, i, FTS3_SEGCURSOR_PENDING); if( rc==SQLITE_DONE ) rc = SQLITE_OK; } sqlite3Fts3PendingTermsClear(p); return rc; } /* ** Encode N integers as varints into a blob. */ static void fts3EncodeIntArray( int N, /* The number of integers to encode */ |
︙ | ︙ | |||
2436 2437 2438 2439 2440 2441 2442 2443 2444 2445 2446 2447 2448 2449 2450 2451 2452 2453 2454 2455 2456 2457 2458 2459 | return; } sqlite3_bind_blob(pStmt, 1, pBlob, nBlob, SQLITE_STATIC); sqlite3_step(pStmt); *pRC = sqlite3_reset(pStmt); sqlite3_free(a); } /* ** Handle a 'special' INSERT of the form: ** ** "INSERT INTO tbl(tbl) VALUES(<expr>)" ** ** Argument pVal contains the result of <expr>. Currently the only ** meaningful value to insert is the text 'optimize'. */ static int fts3SpecialInsert(Fts3Table *p, sqlite3_value *pVal){ int rc; /* Return Code */ const char *zVal = (const char *)sqlite3_value_text(pVal); int nVal = sqlite3_value_bytes(pVal); if( !zVal ){ return SQLITE_NOMEM; }else if( nVal==8 && 0==sqlite3_strnicmp(zVal, "optimize", 8) ){ | > > > > > > > > > > > > > > > > > | < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < | < < < | | 2889 2890 2891 2892 2893 2894 2895 2896 2897 2898 2899 2900 2901 2902 2903 2904 2905 2906 2907 2908 2909 2910 2911 2912 2913 2914 2915 2916 2917 2918 2919 2920 2921 2922 2923 2924 2925 2926 2927 2928 2929 2930 2931 2932 2933 2934 2935 2936 2937 2938 2939 2940 2941 2942 2943 2944 2945 2946 2947 2948 2949 2950 2951 2952 2953 2954 2955 2956 2957 2958 2959 2960 2961 2962 2963 2964 2965 2966 2967 2968 2969 2970 2971 2972 2973 2974 | return; } sqlite3_bind_blob(pStmt, 1, pBlob, nBlob, SQLITE_STATIC); sqlite3_step(pStmt); *pRC = sqlite3_reset(pStmt); sqlite3_free(a); } static int fts3DoOptimize(Fts3Table *p, int bReturnDone){ int i; int bSeenDone = 0; int rc = SQLITE_OK; for(i=0; rc==SQLITE_OK && i<p->nIndex; i++){ rc = fts3SegmentMerge(p, i, FTS3_SEGCURSOR_ALL); if( rc==SQLITE_DONE ){ bSeenDone = 1; rc = SQLITE_OK; } } sqlite3Fts3SegmentsClose(p); sqlite3Fts3PendingTermsClear(p); return (rc==SQLITE_OK && bReturnDone && bSeenDone) ? SQLITE_DONE : rc; } /* ** Handle a 'special' INSERT of the form: ** ** "INSERT INTO tbl(tbl) VALUES(<expr>)" ** ** Argument pVal contains the result of <expr>. Currently the only ** meaningful value to insert is the text 'optimize'. */ static int fts3SpecialInsert(Fts3Table *p, sqlite3_value *pVal){ int rc; /* Return Code */ const char *zVal = (const char *)sqlite3_value_text(pVal); int nVal = sqlite3_value_bytes(pVal); if( !zVal ){ return SQLITE_NOMEM; }else if( nVal==8 && 0==sqlite3_strnicmp(zVal, "optimize", 8) ){ rc = fts3DoOptimize(p, 0); #ifdef SQLITE_TEST }else if( nVal>9 && 0==sqlite3_strnicmp(zVal, "nodesize=", 9) ){ p->nNodeSize = atoi(&zVal[9]); rc = SQLITE_OK; }else if( nVal>11 && 0==sqlite3_strnicmp(zVal, "maxpending=", 9) ){ p->nMaxPendingData = atoi(&zVal[11]); rc = SQLITE_OK; #endif }else{ rc = SQLITE_ERROR; } return rc; } /* ** Delete all cached deferred doclists. Deferred doclists are cached ** (allocated) by the sqlite3Fts3CacheDeferredDoclists() function. */ void sqlite3Fts3FreeDeferredDoclists(Fts3Cursor *pCsr){ Fts3DeferredToken *pDef; for(pDef=pCsr->pDeferred; pDef; pDef=pDef->pNext){ fts3PendingListDelete(pDef->pList); pDef->pList = 0; } } /* ** Free all entries in the pCsr->pDeffered list. Entries are added to ** this list using sqlite3Fts3DeferToken(). */ void sqlite3Fts3FreeDeferredTokens(Fts3Cursor *pCsr){ Fts3DeferredToken *pDef; Fts3DeferredToken *pNext; for(pDef=pCsr->pDeferred; pDef; pDef=pNext){ pNext = pDef->pNext; fts3PendingListDelete(pDef->pList); sqlite3_free(pDef); } pCsr->pDeferred = 0; } /* ** Generate deferred-doclists for all tokens in the pCsr->pDeferred list |
︙ | ︙ | |||
2597 2598 2599 2600 2601 2602 2603 2604 2605 2606 2607 2608 2609 2610 | rc = fts3PendingListAppendVarint(&pDef->pList, 0); } } } return rc; } /* ** Add an entry for token pToken to the pCsr->pDeferred list. */ int sqlite3Fts3DeferToken( Fts3Cursor *pCsr, /* Fts3 table cursor */ Fts3PhraseToken *pToken, /* Token to defer */ | > > > > > > > > > > > > > > > > > > > > > > > > > > > | 3024 3025 3026 3027 3028 3029 3030 3031 3032 3033 3034 3035 3036 3037 3038 3039 3040 3041 3042 3043 3044 3045 3046 3047 3048 3049 3050 3051 3052 3053 3054 3055 3056 3057 3058 3059 3060 3061 3062 3063 3064 | rc = fts3PendingListAppendVarint(&pDef->pList, 0); } } } return rc; } int sqlite3Fts3DeferredTokenList( Fts3DeferredToken *p, char **ppData, int *pnData ){ char *pRet; int nSkip; sqlite3_int64 dummy; *ppData = 0; *pnData = 0; if( p->pList==0 ){ return SQLITE_OK; } pRet = (char *)sqlite3_malloc(p->pList->nData); if( !pRet ) return SQLITE_NOMEM; nSkip = sqlite3Fts3GetVarint(p->pList->aData, &dummy); *pnData = p->pList->nData - nSkip; *ppData = pRet; memcpy(pRet, &p->pList->aData[nSkip], *pnData); return SQLITE_OK; } /* ** Add an entry for token pToken to the pCsr->pDeferred list. */ int sqlite3Fts3DeferToken( Fts3Cursor *pCsr, /* Fts3 table cursor */ Fts3PhraseToken *pToken, /* Token to defer */ |
︙ | ︙ | |||
2672 2673 2674 2675 2676 2677 2678 | sqlite3_value **apVal, /* Array of arguments */ sqlite_int64 *pRowid /* OUT: The affected (or effected) rowid */ ){ Fts3Table *p = (Fts3Table *)pVtab; int rc = SQLITE_OK; /* Return Code */ int isRemove = 0; /* True for an UPDATE or DELETE */ sqlite3_int64 iRemove = 0; /* Rowid removed by UPDATE or DELETE */ | | | > | > > > | 3126 3127 3128 3129 3130 3131 3132 3133 3134 3135 3136 3137 3138 3139 3140 3141 3142 3143 3144 3145 3146 3147 3148 3149 3150 3151 3152 3153 3154 3155 3156 3157 3158 3159 3160 3161 3162 3163 3164 | sqlite3_value **apVal, /* Array of arguments */ sqlite_int64 *pRowid /* OUT: The affected (or effected) rowid */ ){ Fts3Table *p = (Fts3Table *)pVtab; int rc = SQLITE_OK; /* Return Code */ int isRemove = 0; /* True for an UPDATE or DELETE */ sqlite3_int64 iRemove = 0; /* Rowid removed by UPDATE or DELETE */ u32 *aSzIns = 0; /* Sizes of inserted documents */ u32 *aSzDel; /* Sizes of deleted documents */ int nChng = 0; /* Net change in number of documents */ int bInsertDone = 0; assert( p->pSegments==0 ); /* Check for a "special" INSERT operation. One of the form: ** ** INSERT INTO xyz(xyz) VALUES('command'); */ if( nArg>1 && sqlite3_value_type(apVal[0])==SQLITE_NULL && sqlite3_value_type(apVal[p->nColumn+2])!=SQLITE_NULL ){ rc = fts3SpecialInsert(p, apVal[p->nColumn+2]); goto update_out; } /* Allocate space to hold the change in document sizes */ aSzIns = sqlite3_malloc( sizeof(aSzIns[0])*(p->nColumn+1)*2 ); if( aSzIns==0 ){ rc = SQLITE_NOMEM; goto update_out; } aSzDel = &aSzIns[p->nColumn+1]; memset(aSzIns, 0, sizeof(aSzIns[0])*(p->nColumn+1)*2); /* If this is an INSERT operation, or an UPDATE that modifies the rowid ** value, then this operation requires constraint handling. ** ** If the on-conflict mode is REPLACE, this means that the existing row |
︙ | ︙ | |||
2742 2743 2744 2745 2746 2747 2748 | }else{ rc = fts3InsertData(p, apVal, pRowid); bInsertDone = 1; } } } if( rc!=SQLITE_OK ){ | | < | 3200 3201 3202 3203 3204 3205 3206 3207 3208 3209 3210 3211 3212 3213 3214 | }else{ rc = fts3InsertData(p, apVal, pRowid); bInsertDone = 1; } } } if( rc!=SQLITE_OK ){ goto update_out; } /* If this is a DELETE or UPDATE operation, remove the old record. */ if( sqlite3_value_type(apVal[0])!=SQLITE_NULL ){ assert( sqlite3_value_type(apVal[0])==SQLITE_INTEGER ); rc = fts3DeleteByRowid(p, apVal[0], &nChng, aSzDel); isRemove = 1; |
︙ | ︙ | |||
2776 2777 2778 2779 2780 2781 2782 2783 2784 2785 2786 2787 2788 2789 2790 2791 2792 2793 2794 2795 2796 | nChng++; } if( p->bHasStat ){ fts3UpdateDocTotals(&rc, p, aSzIns, aSzDel, nChng); } sqlite3_free(aSzIns); sqlite3Fts3SegmentsClose(p); return rc; } /* ** Flush any data in the pending-terms hash table to disk. If successful, ** merge all segments in the database (including the new segment, if ** there was any data to flush) into a single segment. */ int sqlite3Fts3Optimize(Fts3Table *p){ int rc; rc = sqlite3_exec(p->db, "SAVEPOINT fts3", 0, 0, 0); if( rc==SQLITE_OK ){ | > | | | | < < | 3233 3234 3235 3236 3237 3238 3239 3240 3241 3242 3243 3244 3245 3246 3247 3248 3249 3250 3251 3252 3253 3254 3255 3256 3257 3258 3259 3260 3261 3262 3263 3264 3265 3266 3267 3268 | nChng++; } if( p->bHasStat ){ fts3UpdateDocTotals(&rc, p, aSzIns, aSzDel, nChng); } update_out: sqlite3_free(aSzIns); sqlite3Fts3SegmentsClose(p); return rc; } /* ** Flush any data in the pending-terms hash table to disk. If successful, ** merge all segments in the database (including the new segment, if ** there was any data to flush) into a single segment. */ int sqlite3Fts3Optimize(Fts3Table *p){ int rc; rc = sqlite3_exec(p->db, "SAVEPOINT fts3", 0, 0, 0); if( rc==SQLITE_OK ){ rc = fts3DoOptimize(p, 1); if( rc==SQLITE_OK || rc==SQLITE_DONE ){ int rc2 = sqlite3_exec(p->db, "RELEASE fts3", 0, 0, 0); if( rc2!=SQLITE_OK ) rc = rc2; }else{ sqlite3_exec(p->db, "ROLLBACK TO fts3", 0, 0, 0); sqlite3_exec(p->db, "RELEASE fts3", 0, 0, 0); } } sqlite3Fts3SegmentsClose(p); return rc; } #endif |
Changes to main.mk.
︙ | ︙ | |||
217 218 219 220 221 222 223 224 225 226 227 228 229 230 | parse.h \ sqlite3.h # Source code to the test files. # TESTSRC = \ $(TOP)/src/test1.c \ $(TOP)/src/test2.c \ $(TOP)/src/test3.c \ $(TOP)/src/test4.c \ $(TOP)/src/test5.c \ $(TOP)/src/test6.c \ $(TOP)/src/test7.c \ | > | 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 | parse.h \ sqlite3.h # Source code to the test files. # TESTSRC = \ $(TOP)/ext/fts3/fts3_test.c \ $(TOP)/src/test1.c \ $(TOP)/src/test2.c \ $(TOP)/src/test3.c \ $(TOP)/src/test4.c \ $(TOP)/src/test5.c \ $(TOP)/src/test6.c \ $(TOP)/src/test7.c \ |
︙ | ︙ |
Changes to src/alter.c.
︙ | ︙ | |||
354 355 356 357 358 359 360 | /* Drop the table and index from the internal schema. */ sqlite3VdbeAddOp4(v, OP_DropTable, iDb, 0, 0, pTab->zName, 0); /* Reload the table, index and permanent trigger schemas. */ zWhere = sqlite3MPrintf(pParse->db, "tbl_name=%Q", zName); if( !zWhere ) return; | | | | 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 | /* Drop the table and index from the internal schema. */ sqlite3VdbeAddOp4(v, OP_DropTable, iDb, 0, 0, pTab->zName, 0); /* Reload the table, index and permanent trigger schemas. */ zWhere = sqlite3MPrintf(pParse->db, "tbl_name=%Q", zName); if( !zWhere ) return; sqlite3VdbeAddParseSchemaOp(v, iDb, zWhere); #ifndef SQLITE_OMIT_TRIGGER /* Now, if the table is not stored in the temp database, reload any temp ** triggers. Don't use IN(...) in case SQLITE_OMIT_SUBQUERY is defined. */ if( (zWhere=whereTempTriggers(pParse, pTab))!=0 ){ sqlite3VdbeAddParseSchemaOp(v, 1, zWhere); } #endif } /* ** Parameter zName is the name of a table that is about to be altered ** (either with ALTER TABLE ... RENAME TO or ALTER TABLE ... ADD COLUMN). |
︙ | ︙ |
Changes to src/btree.c.
︙ | ︙ | |||
853 854 855 856 857 858 859 860 861 862 863 864 865 866 | ** the page, 1 means the second cell, and so forth) return a pointer ** to the cell content. ** ** This routine works only for pages that do not contain overflow cells. */ #define findCell(P,I) \ ((P)->aData + ((P)->maskPage & get2byte(&(P)->aData[(P)->cellOffset+2*(I)]))) /* ** This a more complex version of findCell() that works for ** pages that do contain overflow cells. */ static u8 *findOverflowCell(MemPage *pPage, int iCell){ int i; | > > | 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 | ** the page, 1 means the second cell, and so forth) return a pointer ** to the cell content. ** ** This routine works only for pages that do not contain overflow cells. */ #define findCell(P,I) \ ((P)->aData + ((P)->maskPage & get2byte(&(P)->aData[(P)->cellOffset+2*(I)]))) #define findCellv2(D,M,O,I) (D+(M&get2byte(D+(O+2*(I))))) /* ** This a more complex version of findCell() that works for ** pages that do contain overflow cells. */ static u8 *findOverflowCell(MemPage *pPage, int iCell){ int i; |
︙ | ︙ | |||
4447 4448 4449 4450 4451 4452 4453 | if( pCur->eState==CURSOR_INVALID ){ *pRes = -1; assert( pCur->apPage[pCur->iPage]->nCell==0 ); return SQLITE_OK; } assert( pCur->apPage[0]->intKey || pIdxKey ); for(;;){ | | | | < > | 4449 4450 4451 4452 4453 4454 4455 4456 4457 4458 4459 4460 4461 4462 4463 4464 4465 4466 4467 4468 4469 4470 4471 4472 4473 4474 4475 4476 4477 4478 4479 4480 4481 4482 4483 4484 4485 4486 | if( pCur->eState==CURSOR_INVALID ){ *pRes = -1; assert( pCur->apPage[pCur->iPage]->nCell==0 ); return SQLITE_OK; } assert( pCur->apPage[0]->intKey || pIdxKey ); for(;;){ int lwr, upr, idx; Pgno chldPg; MemPage *pPage = pCur->apPage[pCur->iPage]; int c; /* pPage->nCell must be greater than zero. If this is the root-page ** the cursor would have been INVALID above and this for(;;) loop ** not run. If this is not the root-page, then the moveToChild() routine ** would have already detected db corruption. Similarly, pPage must ** be the right kind (index or table) of b-tree page. Otherwise ** a moveToChild() or moveToRoot() call would have detected corruption. */ assert( pPage->nCell>0 ); assert( pPage->intKey==(pIdxKey==0) ); lwr = 0; upr = pPage->nCell-1; if( biasRight ){ pCur->aiIdx[pCur->iPage] = (u16)(idx = upr); }else{ pCur->aiIdx[pCur->iPage] = (u16)(idx = (upr+lwr)/2); } for(;;){ u8 *pCell; /* Pointer to current cell in pPage */ assert( idx==pCur->aiIdx[pCur->iPage] ); pCur->info.nSize = 0; pCell = findCell(pPage, idx) + pPage->childPtrSize; if( pPage->intKey ){ i64 nCellKey; if( pPage->hasData ){ u32 dummy; pCell += getVarint32(pCell, dummy); |
︙ | ︙ | |||
4553 4554 4555 4556 4557 4558 4559 | lwr = idx+1; }else{ upr = idx-1; } if( lwr>upr ){ break; } | | | 4555 4556 4557 4558 4559 4560 4561 4562 4563 4564 4565 4566 4567 4568 4569 | lwr = idx+1; }else{ upr = idx-1; } if( lwr>upr ){ break; } pCur->aiIdx[pCur->iPage] = (u16)(idx = (lwr+upr)/2); } assert( lwr==upr+1 ); assert( pPage->isInit ); if( pPage->leaf ){ chldPg = 0; }else if( lwr>=pPage->nCell ){ chldPg = get4byte(&pPage->aData[pPage->hdrOffset+8]); |
︙ | ︙ | |||
5386 5387 5388 5389 5390 5391 5392 | ** The cell content is not freed or deallocated. It is assumed that ** the cell content has been copied someplace else. This routine just ** removes the reference to the cell from pPage. ** ** "sz" must be the number of bytes in the cell. */ static void dropCell(MemPage *pPage, int idx, int sz, int *pRC){ | < > | 5388 5389 5390 5391 5392 5393 5394 5395 5396 5397 5398 5399 5400 5401 5402 5403 5404 5405 | ** The cell content is not freed or deallocated. It is assumed that ** the cell content has been copied someplace else. This routine just ** removes the reference to the cell from pPage. ** ** "sz" must be the number of bytes in the cell. */ static void dropCell(MemPage *pPage, int idx, int sz, int *pRC){ u32 pc; /* Offset to cell content of cell being deleted */ u8 *data; /* pPage->aData */ u8 *ptr; /* Used to move bytes around within data[] */ u8 *endPtr; /* End of loop */ int rc; /* The return code */ int hdr; /* Beginning of the header. 0 most pages. 100 page 1 */ if( *pRC ) return; assert( idx>=0 && idx<pPage->nCell ); assert( sz==cellSize(pPage, idx) ); |
︙ | ︙ | |||
5414 5415 5416 5417 5418 5419 5420 | return; } rc = freeSpace(pPage, pc, sz); if( rc ){ *pRC = rc; return; } | | > | > | | 5416 5417 5418 5419 5420 5421 5422 5423 5424 5425 5426 5427 5428 5429 5430 5431 5432 5433 5434 | return; } rc = freeSpace(pPage, pc, sz); if( rc ){ *pRC = rc; return; } endPtr = &data[pPage->cellOffset + 2*pPage->nCell - 2]; assert( (SQLITE_PTR_TO_INT(ptr)&1)==0 ); /* ptr is always 2-byte aligned */ while( ptr<endPtr ){ *(u16*)ptr = *(u16*)&ptr[2]; ptr += 2; } pPage->nCell--; put2byte(&data[hdr+3], pPage->nCell); pPage->nFree += 2; } /* |
︙ | ︙ | |||
5456 5457 5458 5459 5460 5461 5462 5463 5464 5465 5466 5467 5468 5469 | int idx = 0; /* Where to write new cell content in data[] */ int j; /* Loop counter */ int end; /* First byte past the last cell pointer in data[] */ int ins; /* Index in data[] where new cell pointer is inserted */ int cellOffset; /* Address of first cell pointer in data[] */ u8 *data; /* The content of the whole page */ u8 *ptr; /* Used for moving information around in data[] */ int nSkip = (iChild ? 4 : 0); if( *pRC ) return; assert( i>=0 && i<=pPage->nCell+pPage->nOverflow ); assert( pPage->nCell<=MX_CELL(pPage->pBt) && MX_CELL(pPage->pBt)<=10921 ); | > | 5460 5461 5462 5463 5464 5465 5466 5467 5468 5469 5470 5471 5472 5473 5474 | int idx = 0; /* Where to write new cell content in data[] */ int j; /* Loop counter */ int end; /* First byte past the last cell pointer in data[] */ int ins; /* Index in data[] where new cell pointer is inserted */ int cellOffset; /* Address of first cell pointer in data[] */ u8 *data; /* The content of the whole page */ u8 *ptr; /* Used for moving information around in data[] */ u8 *endPtr; /* End of the loop */ int nSkip = (iChild ? 4 : 0); if( *pRC ) return; assert( i>=0 && i<=pPage->nCell+pPage->nOverflow ); assert( pPage->nCell<=MX_CELL(pPage->pBt) && MX_CELL(pPage->pBt)<=10921 ); |
︙ | ︙ | |||
5506 5507 5508 5509 5510 5511 5512 | assert( idx+sz <= (int)pPage->pBt->usableSize ); pPage->nCell++; pPage->nFree -= (u16)(2 + sz); memcpy(&data[idx+nSkip], pCell+nSkip, sz-nSkip); if( iChild ){ put4byte(&data[idx], iChild); } | | > > | > | | 5511 5512 5513 5514 5515 5516 5517 5518 5519 5520 5521 5522 5523 5524 5525 5526 5527 5528 5529 5530 | assert( idx+sz <= (int)pPage->pBt->usableSize ); pPage->nCell++; pPage->nFree -= (u16)(2 + sz); memcpy(&data[idx+nSkip], pCell+nSkip, sz-nSkip); if( iChild ){ put4byte(&data[idx], iChild); } ptr = &data[end]; endPtr = &data[ins]; assert( (SQLITE_PTR_TO_INT(ptr)&1)==0 ); /* ptr is always 2-byte aligned */ while( ptr>endPtr ){ *(u16*)ptr = *(u16*)&ptr[-2]; ptr -= 2; } put2byte(&data[ins], idx); put2byte(&data[pPage->hdrOffset+3], pPage->nCell); #ifndef SQLITE_OMIT_AUTOVACUUM if( pPage->pBt->autoVacuum ){ /* The cell may contain a pointer to an overflow page. If so, write ** the entry for the overflow page into the pointer map. |
︙ | ︙ | |||
5553 5554 5555 5556 5557 5558 5559 5560 | /* Check that the page has just been zeroed by zeroPage() */ assert( pPage->nCell==0 ); assert( get2byteNotZero(&data[hdr+5])==nUsable ); pCellptr = &data[pPage->cellOffset + nCell*2]; cellbody = nUsable; for(i=nCell-1; i>=0; i--){ pCellptr -= 2; | > | | | 5561 5562 5563 5564 5565 5566 5567 5568 5569 5570 5571 5572 5573 5574 5575 5576 5577 5578 5579 | /* Check that the page has just been zeroed by zeroPage() */ assert( pPage->nCell==0 ); assert( get2byteNotZero(&data[hdr+5])==nUsable ); pCellptr = &data[pPage->cellOffset + nCell*2]; cellbody = nUsable; for(i=nCell-1; i>=0; i--){ u16 sz = aSize[i]; pCellptr -= 2; cellbody -= sz; put2byte(pCellptr, cellbody); memcpy(&data[cellbody], apCell[i], sz); } put2byte(&data[hdr+3], nCell); put2byte(&data[hdr+5], cellbody); pPage->nFree -= (nCell*2 + nUsable - cellbody); pPage->nCell = (u16)nCell; } |
︙ | ︙ | |||
6010 6011 6012 6013 6014 6015 6016 | ** process of being overwritten. */ MemPage *pOld = apCopy[i] = (MemPage*)&aSpace1[pBt->pageSize + k*i]; memcpy(pOld, apOld[i], sizeof(MemPage)); pOld->aData = (void*)&pOld[1]; memcpy(pOld->aData, apOld[i]->aData, pBt->pageSize); limit = pOld->nCell+pOld->nOverflow; | > | | | | | | > > > > > > > > > > > | 6019 6020 6021 6022 6023 6024 6025 6026 6027 6028 6029 6030 6031 6032 6033 6034 6035 6036 6037 6038 6039 6040 6041 6042 6043 6044 6045 6046 6047 6048 6049 6050 | ** process of being overwritten. */ MemPage *pOld = apCopy[i] = (MemPage*)&aSpace1[pBt->pageSize + k*i]; memcpy(pOld, apOld[i], sizeof(MemPage)); pOld->aData = (void*)&pOld[1]; memcpy(pOld->aData, apOld[i]->aData, pBt->pageSize); limit = pOld->nCell+pOld->nOverflow; if( pOld->nOverflow>0 ){ for(j=0; j<limit; j++){ assert( nCell<nMaxCells ); apCell[nCell] = findOverflowCell(pOld, j); szCell[nCell] = cellSizePtr(pOld, apCell[nCell]); nCell++; } }else{ u8 *aData = pOld->aData; u16 maskPage = pOld->maskPage; u16 cellOffset = pOld->cellOffset; for(j=0; j<limit; j++){ assert( nCell<nMaxCells ); apCell[nCell] = findCellv2(aData, maskPage, cellOffset, j); szCell[nCell] = cellSizePtr(pOld, apCell[nCell]); nCell++; } } if( i<nOld-1 && !leafData){ u16 sz = (u16)szNew[i]; u8 *pTemp; assert( nCell<nMaxCells ); szCell[nCell] = sz; pTemp = &aSpace1[iSpace1]; iSpace1 += sz; |
︙ | ︙ |
Changes to src/build.c.
︙ | ︙ | |||
196 197 198 199 200 201 202 | FILE *trace = (db->flags & SQLITE_VdbeTrace)!=0 ? stdout : 0; sqlite3VdbeTrace(v, trace); #endif assert( pParse->iCacheLevel==0 ); /* Disables and re-enables match */ /* A minimum of one cursor is required if autoincrement is used * See ticket [a696379c1f08866] */ if( pParse->pAinc!=0 && pParse->nTab==0 ) pParse->nTab = 1; | | < < | 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 | FILE *trace = (db->flags & SQLITE_VdbeTrace)!=0 ? stdout : 0; sqlite3VdbeTrace(v, trace); #endif assert( pParse->iCacheLevel==0 ); /* Disables and re-enables match */ /* A minimum of one cursor is required if autoincrement is used * See ticket [a696379c1f08866] */ if( pParse->pAinc!=0 && pParse->nTab==0 ) pParse->nTab = 1; sqlite3VdbeMakeReady(v, pParse); pParse->rc = SQLITE_DONE; pParse->colNamesSet = 0; }else{ pParse->rc = SQLITE_ERROR; } pParse->nTab = 0; pParse->nMem = 0; |
︙ | ︙ | |||
1617 1618 1619 1620 1621 1622 1623 | pDb->zName ); } } #endif /* Reparse everything to update our internal data structures */ | | | | 1615 1616 1617 1618 1619 1620 1621 1622 1623 1624 1625 1626 1627 1628 1629 1630 | pDb->zName ); } } #endif /* Reparse everything to update our internal data structures */ sqlite3VdbeAddParseSchemaOp(v, iDb, sqlite3MPrintf(db, "tbl_name='%q'", p->zName)); } /* Add the table to the in-memory representation of the database. */ if( db->init.busy ){ Table *pOld; |
︙ | ︙ | |||
2815 2816 2817 2818 2819 2820 2821 | /* Fill the index with data and reparse the schema. Code an OP_Expire ** to invalidate all pre-compiled statements. */ if( pTblName ){ sqlite3RefillIndex(pParse, pIndex, iMem); sqlite3ChangeCookie(pParse, iDb); | | | < | 2813 2814 2815 2816 2817 2818 2819 2820 2821 2822 2823 2824 2825 2826 2827 2828 | /* Fill the index with data and reparse the schema. Code an OP_Expire ** to invalidate all pre-compiled statements. */ if( pTblName ){ sqlite3RefillIndex(pParse, pIndex, iMem); sqlite3ChangeCookie(pParse, iDb); sqlite3VdbeAddParseSchemaOp(v, iDb, sqlite3MPrintf(db, "name='%q' AND type='index'", pIndex->zName)); sqlite3VdbeAddOp1(v, OP_Expire, 0); } } /* When adding an index to the list of indices for a table, make ** sure all indices labeled OE_Replace come after all those labeled ** OE_Ignore. This is necessary for the correct constraint check |
︙ | ︙ |
Changes to src/delete.c.
︙ | ︙ | |||
404 405 406 407 408 409 410 411 412 413 414 415 416 417 | /* Delete the row */ #ifndef SQLITE_OMIT_VIRTUALTABLE if( IsVirtual(pTab) ){ const char *pVTab = (const char *)sqlite3GetVTable(db, pTab); sqlite3VtabMakeWritable(pParse, pTab); sqlite3VdbeAddOp4(v, OP_VUpdate, 0, 1, iRowid, (char*)pVTab, P4_VTAB); sqlite3MayAbort(pParse); }else #endif { int count = (pParse->nested==0); /* True to count changes */ sqlite3GenerateRowDelete(pParse, pTab, iCur, iRowid, count, pTrigger, OE_Default); } | > | 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 | /* Delete the row */ #ifndef SQLITE_OMIT_VIRTUALTABLE if( IsVirtual(pTab) ){ const char *pVTab = (const char *)sqlite3GetVTable(db, pTab); sqlite3VtabMakeWritable(pParse, pTab); sqlite3VdbeAddOp4(v, OP_VUpdate, 0, 1, iRowid, (char*)pVTab, P4_VTAB); sqlite3VdbeChangeP5(v, OE_Abort); sqlite3MayAbort(pParse); }else #endif { int count = (pParse->nested==0); /* True to count changes */ sqlite3GenerateRowDelete(pParse, pTab, iCur, iRowid, count, pTrigger, OE_Default); } |
︙ | ︙ |
Changes to src/expr.c.
︙ | ︙ | |||
551 552 553 554 555 556 557 | z = pExpr->u.zToken; assert( z!=0 ); assert( z[0]!=0 ); if( z[1]==0 ){ /* Wildcard of the form "?". Assign the next variable number */ assert( z[0]=='?' ); pExpr->iColumn = (ynVar)(++pParse->nVar); | < < < < < < < < < < < < < < < < < > > > > > > > > > > > > > > > > > > > > > | | | | | < < | | < < | | | | < | > | < | > | < > | > | < | | | | 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 | z = pExpr->u.zToken; assert( z!=0 ); assert( z[0]!=0 ); if( z[1]==0 ){ /* Wildcard of the form "?". Assign the next variable number */ assert( z[0]=='?' ); pExpr->iColumn = (ynVar)(++pParse->nVar); }else{ ynVar x = 0; u32 n = sqlite3Strlen30(z); if( z[0]=='?' ){ /* Wildcard of the form "?nnn". Convert "nnn" to an integer and ** use it as the variable number */ i64 i; int bOk = 0==sqlite3Atoi64(&z[1], &i, n-1, SQLITE_UTF8); pExpr->iColumn = x = (ynVar)i; testcase( i==0 ); testcase( i==1 ); testcase( i==db->aLimit[SQLITE_LIMIT_VARIABLE_NUMBER]-1 ); testcase( i==db->aLimit[SQLITE_LIMIT_VARIABLE_NUMBER] ); if( bOk==0 || i<1 || i>db->aLimit[SQLITE_LIMIT_VARIABLE_NUMBER] ){ sqlite3ErrorMsg(pParse, "variable number must be between ?1 and ?%d", db->aLimit[SQLITE_LIMIT_VARIABLE_NUMBER]); x = 0; } if( i>pParse->nVar ){ pParse->nVar = (int)i; } }else{ /* Wildcards like ":aaa", "$aaa" or "@aaa". Reuse the same variable ** number as the prior appearance of the same name, or if the name ** has never appeared before, reuse the same variable number */ ynVar i; for(i=0; i<pParse->nzVar; i++){ if( pParse->azVar[i] && memcmp(pParse->azVar[i],z,n+1)==0 ){ pExpr->iColumn = x = (ynVar)i+1; break; } } if( x==0 ) x = pExpr->iColumn = (ynVar)(++pParse->nVar); } if( x>0 ){ if( x>pParse->nzVar ){ char **a; a = sqlite3DbRealloc(db, pParse->azVar, x*sizeof(a[0])); if( a==0 ) return; /* Error reported through db->mallocFailed */ pParse->azVar = a; memset(&a[pParse->nzVar], 0, (x-pParse->nzVar)*sizeof(a[0])); pParse->nzVar = x; } if( z[0]!='?' || pParse->azVar[x-1]==0 ){ sqlite3DbFree(db, pParse->azVar[x-1]); pParse->azVar[x-1] = sqlite3DbStrNDup(db, z, n); } } } if( !pParse->nErr && pParse->nVar>db->aLimit[SQLITE_LIMIT_VARIABLE_NUMBER] ){ sqlite3ErrorMsg(pParse, "too many SQL variables"); } } |
︙ | ︙ | |||
2341 2342 2343 2344 2345 2346 2347 | #endif case TK_VARIABLE: { assert( !ExprHasProperty(pExpr, EP_IntValue) ); assert( pExpr->u.zToken!=0 ); assert( pExpr->u.zToken[0]!=0 ); sqlite3VdbeAddOp2(v, OP_Variable, pExpr->iColumn, target); if( pExpr->u.zToken[1]!=0 ){ | > > | | 2341 2342 2343 2344 2345 2346 2347 2348 2349 2350 2351 2352 2353 2354 2355 2356 2357 | #endif case TK_VARIABLE: { assert( !ExprHasProperty(pExpr, EP_IntValue) ); assert( pExpr->u.zToken!=0 ); assert( pExpr->u.zToken[0]!=0 ); sqlite3VdbeAddOp2(v, OP_Variable, pExpr->iColumn, target); if( pExpr->u.zToken[1]!=0 ){ assert( pExpr->u.zToken[0]=='?' || strcmp(pExpr->u.zToken, pParse->azVar[pExpr->iColumn-1])==0 ); sqlite3VdbeChangeP4(v, -1, pParse->azVar[pExpr->iColumn-1], P4_STATIC); } break; } case TK_REGISTER: { inReg = pExpr->iTable; break; } |
︙ | ︙ |
Changes to src/fkey.c.
︙ | ︙ | |||
382 383 384 385 386 387 388 | for(i=0; i<nCol; i++){ sqlite3VdbeAddOp2(v, OP_Copy, aiCol[i]+1+regData, regTemp+i); } /* If the parent table is the same as the child table, and we are about ** to increment the constraint-counter (i.e. this is an INSERT operation), ** then check if the row being inserted matches itself. If so, do not | | > > > > > > > > > > > > | 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 | for(i=0; i<nCol; i++){ sqlite3VdbeAddOp2(v, OP_Copy, aiCol[i]+1+regData, regTemp+i); } /* If the parent table is the same as the child table, and we are about ** to increment the constraint-counter (i.e. this is an INSERT operation), ** then check if the row being inserted matches itself. If so, do not ** increment the constraint-counter. ** ** If any of the parent-key values are NULL, then the row cannot match ** itself. So set JUMPIFNULL to make sure we do the OP_Found if any ** of the parent-key values are NULL (at this point it is known that ** none of the child key values are). */ if( pTab==pFKey->pFrom && nIncr==1 ){ int iJump = sqlite3VdbeCurrentAddr(v) + nCol + 1; for(i=0; i<nCol; i++){ int iChild = aiCol[i]+1+regData; int iParent = pIdx->aiColumn[i]+1+regData; assert( aiCol[i]!=pTab->iPKey ); if( pIdx->aiColumn[i]==pTab->iPKey ){ /* The parent key is a composite key that includes the IPK column */ iParent = regData; } sqlite3VdbeAddOp3(v, OP_Ne, iChild, iJump, iParent); sqlite3VdbeChangeP5(v, SQLITE_JUMPIFNULL); } sqlite3VdbeAddOp2(v, OP_Goto, 0, iOk); } sqlite3VdbeAddOp3(v, OP_MakeRecord, regTemp, nCol, regRec); sqlite3VdbeChangeP4(v, -1, sqlite3IndexAffinityStr(v,pIdx), P4_TRANSIENT); sqlite3VdbeAddOp4Int(v, OP_Found, iCur, iOk, regRec, 0); |
︙ | ︙ |
Changes to src/func.c.
︙ | ︙ | |||
502 503 504 505 506 507 508 | /* ** For LIKE and GLOB matching on EBCDIC machines, assume that every ** character is exactly one byte in size. Also, all characters are ** able to participate in upper-case-to-lower-case mappings in EBCDIC ** whereas only characters less than 0x80 do in ASCII. */ #if defined(SQLITE_EBCDIC) | | | | | 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 | /* ** For LIKE and GLOB matching on EBCDIC machines, assume that every ** character is exactly one byte in size. Also, all characters are ** able to participate in upper-case-to-lower-case mappings in EBCDIC ** whereas only characters less than 0x80 do in ASCII. */ #if defined(SQLITE_EBCDIC) # define sqlite3Utf8Read(A,C) (*(A++)) # define GlogUpperToLower(A) A = sqlite3UpperToLower[A] #else # define GlogUpperToLower(A) if( !((A)&~0x7f) ){ A = sqlite3UpperToLower[A]; } #endif static const struct compareInfo globInfo = { '*', '?', '[', 0 }; /* The correct SQL-92 behavior is for the LIKE operator to ignore ** case. Thus 'a' LIKE 'A' would be true. */ static const struct compareInfo likeInfoNorm = { '%', '_', 0, 1 }; /* If SQLITE_CASE_SENSITIVE_LIKE is defined, then the LIKE operator |
︙ | ︙ | |||
548 549 550 551 552 553 554 | ** ** abc[*]xyz Matches "abc*xyz" only */ static int patternCompare( const u8 *zPattern, /* The glob pattern */ const u8 *zString, /* The string to compare against the glob */ const struct compareInfo *pInfo, /* Information about how to do the compare */ | | | | 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 | ** ** abc[*]xyz Matches "abc*xyz" only */ static int patternCompare( const u8 *zPattern, /* The glob pattern */ const u8 *zString, /* The string to compare against the glob */ const struct compareInfo *pInfo, /* Information about how to do the compare */ u32 esc /* The escape character */ ){ u32 c, c2; int invert; int seen; u8 matchOne = pInfo->matchOne; u8 matchAll = pInfo->matchAll; u8 matchSet = pInfo->matchSet; u8 noCase = pInfo->noCase; int prevEscape = 0; /* True if the previous character was 'escape' */ |
︙ | ︙ | |||
604 605 606 607 608 609 610 | } return 0; }else if( !prevEscape && c==matchOne ){ if( sqlite3Utf8Read(zString, &zString)==0 ){ return 0; } }else if( c==matchSet ){ | | | 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 | } return 0; }else if( !prevEscape && c==matchOne ){ if( sqlite3Utf8Read(zString, &zString)==0 ){ return 0; } }else if( c==matchSet ){ u32 prior_c = 0; assert( esc==0 ); /* This only occurs for GLOB, not LIKE */ seen = 0; invert = 0; c = sqlite3Utf8Read(zString, &zString); if( c==0 ) return 0; c2 = sqlite3Utf8Read(zPattern, &zPattern); if( c2=='^' ){ |
︙ | ︙ | |||
680 681 682 683 684 685 686 | */ static void likeFunc( sqlite3_context *context, int argc, sqlite3_value **argv ){ const unsigned char *zA, *zB; | | | 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 | */ static void likeFunc( sqlite3_context *context, int argc, sqlite3_value **argv ){ const unsigned char *zA, *zB; u32 escape = 0; int nPat; sqlite3 *db = sqlite3_context_db_handle(context); zB = sqlite3_value_text(argv[0]); zA = sqlite3_value_text(argv[1]); /* Limit the length of the LIKE or GLOB pattern to avoid problems |
︙ | ︙ |
Changes to src/main.c.
︙ | ︙ | |||
1955 1956 1957 1958 1959 1960 1961 | if( nOpt==3 && memcmp("vfs", zOpt, 3)==0 ){ zVfs = zVal; }else{ struct OpenMode { const char *z; int mode; } *aMode = 0; | | | | | 1955 1956 1957 1958 1959 1960 1961 1962 1963 1964 1965 1966 1967 1968 1969 1970 1971 | if( nOpt==3 && memcmp("vfs", zOpt, 3)==0 ){ zVfs = zVal; }else{ struct OpenMode { const char *z; int mode; } *aMode = 0; char *zModeType = 0; int mask = 0; int limit = 0; if( nOpt==5 && memcmp("cache", zOpt, 5)==0 ){ static struct OpenMode aCacheMode[] = { { "shared", SQLITE_OPEN_SHAREDCACHE }, { "private", SQLITE_OPEN_PRIVATECACHE }, { 0, 0 } }; |
︙ | ︙ |
Changes to src/os_unix.c.
︙ | ︙ | |||
3533 3534 3535 3536 3537 3538 3539 | */ struct unixShmNode { unixInodeInfo *pInode; /* unixInodeInfo that owns this SHM node */ sqlite3_mutex *mutex; /* Mutex to access this object */ char *zFilename; /* Name of the mmapped file */ int h; /* Open file descriptor */ int szRegion; /* Size of shared-memory regions */ | | > | 3533 3534 3535 3536 3537 3538 3539 3540 3541 3542 3543 3544 3545 3546 3547 3548 | */ struct unixShmNode { unixInodeInfo *pInode; /* unixInodeInfo that owns this SHM node */ sqlite3_mutex *mutex; /* Mutex to access this object */ char *zFilename; /* Name of the mmapped file */ int h; /* Open file descriptor */ int szRegion; /* Size of shared-memory regions */ u16 nRegion; /* Size of array apRegion */ u8 isReadonly; /* True if read-only */ char **apRegion; /* Array of mapped shared-memory regions */ int nRef; /* Number of unixShm objects pointing to this */ unixShm *pFirst; /* All unixShm objects pointing to this */ #ifdef SQLITE_DEBUG u8 exclMask; /* Mask of exclusive locks held */ u8 sharedMask; /* Mask of shared locks held */ u8 nextShmId; /* Next available unixShm.id value */ |
︙ | ︙ | |||
3780 3781 3782 3783 3784 3785 3786 | goto shm_open_err; } if( pInode->bProcessLock==0 ){ pShmNode->h = robust_open(zShmFilename, O_RDWR|O_CREAT, (sStat.st_mode & 0777)); if( pShmNode->h<0 ){ | > > > > > > > > | | > | 3781 3782 3783 3784 3785 3786 3787 3788 3789 3790 3791 3792 3793 3794 3795 3796 3797 3798 3799 3800 3801 3802 3803 3804 3805 | goto shm_open_err; } if( pInode->bProcessLock==0 ){ pShmNode->h = robust_open(zShmFilename, O_RDWR|O_CREAT, (sStat.st_mode & 0777)); if( pShmNode->h<0 ){ const char *zRO; zRO = sqlite3_uri_parameter(pDbFd->zPath, "readonly_shm"); if( zRO && sqlite3GetBoolean(zRO) ){ pShmNode->h = robust_open(zShmFilename, O_RDONLY, (sStat.st_mode & 0777)); pShmNode->isReadonly = 1; } if( pShmNode->h<0 ){ rc = unixLogError(SQLITE_CANTOPEN_BKPT, "open", zShmFilename); goto shm_open_err; } } /* Check to see if another process is holding the dead-man switch. ** If not, truncate the file to zero length. */ rc = SQLITE_OK; if( unixShmSystemLock(pShmNode, F_WRLCK, UNIX_SHM_DMS, 1)==SQLITE_OK ){ |
︙ | ︙ | |||
3920 3921 3922 3923 3924 3925 3926 | rc = SQLITE_IOERR_NOMEM; goto shmpage_out; } pShmNode->apRegion = apNew; while(pShmNode->nRegion<=iRegion){ void *pMem; if( pShmNode->h>=0 ){ | | > | 3930 3931 3932 3933 3934 3935 3936 3937 3938 3939 3940 3941 3942 3943 3944 3945 | rc = SQLITE_IOERR_NOMEM; goto shmpage_out; } pShmNode->apRegion = apNew; while(pShmNode->nRegion<=iRegion){ void *pMem; if( pShmNode->h>=0 ){ pMem = mmap(0, szRegion, pShmNode->isReadonly ? PROT_READ : PROT_READ|PROT_WRITE, MAP_SHARED, pShmNode->h, pShmNode->nRegion*szRegion ); if( pMem==MAP_FAILED ){ rc = unixLogError(SQLITE_IOERR_SHMMAP, "mmap", pShmNode->zFilename); goto shmpage_out; } }else{ |
︙ | ︙ | |||
3946 3947 3948 3949 3950 3951 3952 3953 3954 3955 3956 3957 3958 3959 | shmpage_out: if( pShmNode->nRegion>iRegion ){ *pp = pShmNode->apRegion[iRegion]; }else{ *pp = 0; } sqlite3_mutex_leave(pShmNode->mutex); return rc; } /* ** Change the lock state for a shared-memory segment. ** | > | 3957 3958 3959 3960 3961 3962 3963 3964 3965 3966 3967 3968 3969 3970 3971 | shmpage_out: if( pShmNode->nRegion>iRegion ){ *pp = pShmNode->apRegion[iRegion]; }else{ *pp = 0; } if( pShmNode->isReadonly && rc==SQLITE_OK ) rc = SQLITE_READONLY; sqlite3_mutex_leave(pShmNode->mutex); return rc; } /* ** Change the lock state for a shared-memory segment. ** |
︙ | ︙ |
Changes to src/pcache1.c.
︙ | ︙ | |||
570 571 572 573 574 575 576 | pCache = (PCache1 *)sqlite3_malloc(sz); if( pCache ){ memset(pCache, 0, sz); if( separateCache ){ pGroup = (PGroup*)&pCache[1]; pGroup->mxPinned = 10; }else{ | | | 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 | pCache = (PCache1 *)sqlite3_malloc(sz); if( pCache ){ memset(pCache, 0, sz); if( separateCache ){ pGroup = (PGroup*)&pCache[1]; pGroup->mxPinned = 10; }else{ pGroup = &pcache1.grp; } pCache->pGroup = pGroup; pCache->szPage = szPage; pCache->bPurgeable = (bPurgeable ? 1 : 0); if( bPurgeable ){ pCache->nMin = 10; pcache1EnterMutex(pGroup); |
︙ | ︙ |
Changes to src/pragma.c.
︙ | ︙ | |||
9 10 11 12 13 14 15 | ** May you share freely, never taking more than you give. ** ************************************************************************* ** This file contains code used to implement the PRAGMA command. */ #include "sqliteInt.h" | < < < < | 9 10 11 12 13 14 15 16 17 18 19 20 21 22 | ** May you share freely, never taking more than you give. ** ************************************************************************* ** This file contains code used to implement the PRAGMA command. */ #include "sqliteInt.h" /* ** Interpret the given string as a safety level. Return 0 for OFF, ** 1 for ON or NORMAL and 2 for FULL. Return 1 for an empty or ** unrecognized string argument. ** ** Note that the values returned are one less that the values that ** should be passed into sqlite3BtreeSetSafetyLevel(). The is done |
︙ | ︙ | |||
48 49 50 51 52 53 54 55 56 57 58 59 60 61 | /* ** Interpret the given string as a boolean value. */ u8 sqlite3GetBoolean(const char *z){ return getSafetyLevel(z)&1; } /* ** Interpret the given string as a locking mode value. */ static int getLockingMode(const char *z){ if( z ){ if( 0==sqlite3StrICmp(z, "exclusive") ) return PAGER_LOCKINGMODE_EXCLUSIVE; | > > > > > > | 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 | /* ** Interpret the given string as a boolean value. */ u8 sqlite3GetBoolean(const char *z){ return getSafetyLevel(z)&1; } /* The sqlite3GetBoolean() function is used by other modules but the ** remainder of this file is specific to PRAGMA processing. So omit ** the rest of the file if PRAGMAs are omitted from the build. */ #if !defined(SQLITE_OMIT_PRAGMA) /* ** Interpret the given string as a locking mode value. */ static int getLockingMode(const char *z){ if( z ){ if( 0==sqlite3StrICmp(z, "exclusive") ) return PAGER_LOCKINGMODE_EXCLUSIVE; |
︙ | ︙ |
Changes to src/shell.c.
︙ | ︙ | |||
2298 2299 2300 2301 2302 2303 2304 2305 2306 2307 2308 2309 2310 2311 | if( HAS_TIMER && c=='t' && n>=5 && strncmp(azArg[0], "timer", n)==0 && nArg==2 ){ enableTimer = booleanValue(azArg[1]); }else if( c=='w' && strncmp(azArg[0], "width", n)==0 && nArg>1 ){ int j; assert( nArg<=ArraySize(azArg) ); for(j=1; j<nArg && j<ArraySize(p->colWidth); j++){ p->colWidth[j-1] = atoi(azArg[j]); } }else | > > > > > | 2298 2299 2300 2301 2302 2303 2304 2305 2306 2307 2308 2309 2310 2311 2312 2313 2314 2315 2316 | if( HAS_TIMER && c=='t' && n>=5 && strncmp(azArg[0], "timer", n)==0 && nArg==2 ){ enableTimer = booleanValue(azArg[1]); }else if( c=='v' && strncmp(azArg[0], "version", n)==0 ){ printf("SQLite %s %s\n", sqlite3_libversion(), sqlite3_sourceid()); }else if( c=='w' && strncmp(azArg[0], "width", n)==0 && nArg>1 ){ int j; assert( nArg<=ArraySize(azArg) ); for(j=1; j<nArg && j<ArraySize(p->colWidth); j++){ p->colWidth[j-1] = atoi(azArg[j]); } }else |
︙ | ︙ | |||
2832 2833 2834 2835 2836 2837 2838 | }else if( strcmp(z,"-echo")==0 ){ data.echoOn = 1; }else if( strcmp(z,"-stats")==0 ){ data.statsOn = 1; }else if( strcmp(z,"-bail")==0 ){ bail_on_error = 1; }else if( strcmp(z,"-version")==0 ){ | | | 2837 2838 2839 2840 2841 2842 2843 2844 2845 2846 2847 2848 2849 2850 2851 | }else if( strcmp(z,"-echo")==0 ){ data.echoOn = 1; }else if( strcmp(z,"-stats")==0 ){ data.statsOn = 1; }else if( strcmp(z,"-bail")==0 ){ bail_on_error = 1; }else if( strcmp(z,"-version")==0 ){ printf("%s %s\n", sqlite3_libversion(), sqlite3_sourceid()); return 0; }else if( strcmp(z,"-interactive")==0 ){ stdin_is_interactive = 1; }else if( strcmp(z,"-batch")==0 ){ stdin_is_interactive = 0; }else if( strcmp(z,"-heap")==0 ){ i++; |
︙ | ︙ | |||
2877 2878 2879 2880 2881 2882 2883 | /* Run commands received from standard input */ if( stdin_is_interactive ){ char *zHome; char *zHistory = 0; int nHistory; printf( | | | | 2882 2883 2884 2885 2886 2887 2888 2889 2890 2891 2892 2893 2894 2895 2896 2897 2898 2899 | /* Run commands received from standard input */ if( stdin_is_interactive ){ char *zHome; char *zHistory = 0; int nHistory; printf( "SQLite version %s %.19s\n" "Enter \".help\" for instructions\n" "Enter SQL statements terminated with a \";\"\n", sqlite3_libversion(), sqlite3_sourceid() ); zHome = find_home_dir(); if( zHome ){ nHistory = strlen30(zHome) + 20; if( (zHistory = malloc(nHistory))!=0 ){ sqlite3_snprintf(nHistory, zHistory,"%s/.sqlite_history", zHome); } |
︙ | ︙ |
Changes to src/sqlite.h.in.
︙ | ︙ | |||
450 451 452 453 454 455 456 457 458 459 460 461 462 463 | #define SQLITE_IOERR_SHMLOCK (SQLITE_IOERR | (20<<8)) #define SQLITE_IOERR_SHMMAP (SQLITE_IOERR | (21<<8)) #define SQLITE_IOERR_SEEK (SQLITE_IOERR | (22<<8)) #define SQLITE_LOCKED_SHAREDCACHE (SQLITE_LOCKED | (1<<8)) #define SQLITE_BUSY_RECOVERY (SQLITE_BUSY | (1<<8)) #define SQLITE_CANTOPEN_NOTEMPDIR (SQLITE_CANTOPEN | (1<<8)) #define SQLITE_CORRUPT_VTAB (SQLITE_CORRUPT | (1<<8)) /* ** CAPI3REF: Flags For File Open Operations ** ** These bit values are intended for use in the ** 3rd parameter to the [sqlite3_open_v2()] interface and ** in the 4th parameter to the [sqlite3_vfs.xOpen] method. | > > | 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 | #define SQLITE_IOERR_SHMLOCK (SQLITE_IOERR | (20<<8)) #define SQLITE_IOERR_SHMMAP (SQLITE_IOERR | (21<<8)) #define SQLITE_IOERR_SEEK (SQLITE_IOERR | (22<<8)) #define SQLITE_LOCKED_SHAREDCACHE (SQLITE_LOCKED | (1<<8)) #define SQLITE_BUSY_RECOVERY (SQLITE_BUSY | (1<<8)) #define SQLITE_CANTOPEN_NOTEMPDIR (SQLITE_CANTOPEN | (1<<8)) #define SQLITE_CORRUPT_VTAB (SQLITE_CORRUPT | (1<<8)) #define SQLITE_READONLY_RECOVERY (SQLITE_READONLY | (1<<8)) #define SQLITE_READONLY_CANTLOCK (SQLITE_READONLY | (2<<8)) /* ** CAPI3REF: Flags For File Open Operations ** ** These bit values are intended for use in the ** 3rd parameter to the [sqlite3_open_v2()] interface and ** in the 4th parameter to the [sqlite3_vfs.xOpen] method. |
︙ | ︙ |
Changes to src/sqliteInt.h.
︙ | ︙ | |||
679 680 681 682 683 684 685 | ** Schema objects are automatically deallocated when the last Btree that ** references them is destroyed. The TEMP Schema is manually freed by ** sqlite3_close(). * ** A thread must be holding a mutex on the corresponding Btree in order ** to access Schema content. This implies that the thread must also be ** holding a mutex on the sqlite3 connection pointer that owns the Btree. | | | 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 | ** Schema objects are automatically deallocated when the last Btree that ** references them is destroyed. The TEMP Schema is manually freed by ** sqlite3_close(). * ** A thread must be holding a mutex on the corresponding Btree in order ** to access Schema content. This implies that the thread must also be ** holding a mutex on the sqlite3 connection pointer that owns the Btree. ** For a TEMP Schema, only the connection mutex is required. */ struct Schema { int schema_cookie; /* Database schema version number for this file */ int iGeneration; /* Generation counter. Incremented with each change */ Hash tblHash; /* All tables indexed by name */ Hash idxHash; /* All (named) indices indexed by name */ Hash trigHash; /* All triggers indexed by name */ |
︙ | ︙ | |||
2234 2235 2236 2237 2238 2239 2240 | u8 disableTriggers; /* True to disable triggers */ double nQueryLoop; /* Estimated number of iterations of a query */ /* Above is constant between recursions. Below is reset before and after ** each recursion */ int nVar; /* Number of '?' variables seen in the SQL so far */ | < | | | 2234 2235 2236 2237 2238 2239 2240 2241 2242 2243 2244 2245 2246 2247 2248 2249 | u8 disableTriggers; /* True to disable triggers */ double nQueryLoop; /* Estimated number of iterations of a query */ /* Above is constant between recursions. Below is reset before and after ** each recursion */ int nVar; /* Number of '?' variables seen in the SQL so far */ int nzVar; /* Number of available slots in azVar[] */ char **azVar; /* Pointers to names of parameters */ Vdbe *pReprepare; /* VM being reprepared (sqlite3Reprepare()) */ int nAlias; /* Number of aliased result set columns */ int nAliasAlloc; /* Number of allocated slots for aAlias[] */ int *aAlias; /* Register used to hold aliased result */ u8 explain; /* True if the EXPLAIN flag is found on the query */ Token sNameToken; /* Token with unqualified schema object name */ Token sLastToken; /* The last token parsed */ |
︙ | ︙ | |||
2885 2886 2887 2888 2889 2890 2891 | int sqlite3FixExprList(DbFixer*, ExprList*); int sqlite3FixTriggerStep(DbFixer*, TriggerStep*); int sqlite3AtoF(const char *z, double*, int, u8); int sqlite3GetInt32(const char *, int*); int sqlite3Atoi(const char*); int sqlite3Utf16ByteLen(const void *pData, int nChar); int sqlite3Utf8CharLen(const char *pData, int nByte); | | | 2884 2885 2886 2887 2888 2889 2890 2891 2892 2893 2894 2895 2896 2897 2898 | int sqlite3FixExprList(DbFixer*, ExprList*); int sqlite3FixTriggerStep(DbFixer*, TriggerStep*); int sqlite3AtoF(const char *z, double*, int, u8); int sqlite3GetInt32(const char *, int*); int sqlite3Atoi(const char*); int sqlite3Utf16ByteLen(const void *pData, int nChar); int sqlite3Utf8CharLen(const char *pData, int nByte); u32 sqlite3Utf8Read(const u8*, const u8**); /* ** Routines to read and write variable-length integers. These used to ** be defined locally, but now we use the varint routines in the util.c ** file. Code should use the MACRO forms below, as the Varint32 versions ** are coded to assume the single byte case is already handled (which ** the MACRO form does). |
︙ | ︙ |
Changes to src/tclsqlite.c.
︙ | ︙ | |||
3729 3730 3731 3732 3733 3734 3735 3736 3737 3738 3739 3740 3741 3742 | extern int SqliteSuperlock_Init(Tcl_Interp*); extern int SqlitetestSyscall_Init(Tcl_Interp*); extern int Sqlitetestfuzzer_Init(Tcl_Interp*); extern int Sqlitetestwholenumber_Init(Tcl_Interp*); #if defined(SQLITE_ENABLE_SESSION) && defined(SQLITE_ENABLE_PREUPDATE_HOOK) extern int TestSession_Init(Tcl_Interp*); #endif #ifdef SQLITE_ENABLE_ZIPVFS extern int Zipvfs_Init(Tcl_Interp*); Zipvfs_Init(interp); #endif Sqliteconfig_Init(interp); Sqlitetest1_Init(interp); | > > > > | 3729 3730 3731 3732 3733 3734 3735 3736 3737 3738 3739 3740 3741 3742 3743 3744 3745 3746 | extern int SqliteSuperlock_Init(Tcl_Interp*); extern int SqlitetestSyscall_Init(Tcl_Interp*); extern int Sqlitetestfuzzer_Init(Tcl_Interp*); extern int Sqlitetestwholenumber_Init(Tcl_Interp*); #if defined(SQLITE_ENABLE_SESSION) && defined(SQLITE_ENABLE_PREUPDATE_HOOK) extern int TestSession_Init(Tcl_Interp*); #endif #ifdef SQLITE_ENABLE_FTS3 extern int Sqlitetestfts3_Init(Tcl_Interp *interp); #endif #ifdef SQLITE_ENABLE_ZIPVFS extern int Zipvfs_Init(Tcl_Interp*); Zipvfs_Init(interp); #endif Sqliteconfig_Init(interp); Sqlitetest1_Init(interp); |
︙ | ︙ | |||
3771 3772 3773 3774 3775 3776 3777 3778 3779 3780 3781 3782 3783 3784 | SqliteSuperlock_Init(interp); SqlitetestSyscall_Init(interp); Sqlitetestfuzzer_Init(interp); Sqlitetestwholenumber_Init(interp); #if defined(SQLITE_ENABLE_SESSION) && defined(SQLITE_ENABLE_PREUPDATE_HOOK) TestSession_Init(interp); #endif Tcl_CreateObjCommand(interp,"load_testfixture_extensions",init_all_cmd,0,0); #ifdef SQLITE_SSE Sqlitetestsse_Init(interp); #endif } | > > > | 3775 3776 3777 3778 3779 3780 3781 3782 3783 3784 3785 3786 3787 3788 3789 3790 3791 | SqliteSuperlock_Init(interp); SqlitetestSyscall_Init(interp); Sqlitetestfuzzer_Init(interp); Sqlitetestwholenumber_Init(interp); #if defined(SQLITE_ENABLE_SESSION) && defined(SQLITE_ENABLE_PREUPDATE_HOOK) TestSession_Init(interp); #endif #ifdef SQLITE_ENABLE_FTS3 Sqlitetestfts3_Init(interp); #endif Tcl_CreateObjCommand(interp,"load_testfixture_extensions",init_all_cmd,0,0); #ifdef SQLITE_SSE Sqlitetestsse_Init(interp); #endif } |
︙ | ︙ |
Changes to src/test1.c.
︙ | ︙ | |||
160 161 162 163 164 165 166 | case SQLITE_IOERR_BLOCKED: zName = "SQLITE_IOERR_BLOCKED"; break; case SQLITE_IOERR_NOMEM: zName = "SQLITE_IOERR_NOMEM"; break; case SQLITE_IOERR_ACCESS: zName = "SQLITE_IOERR_ACCESS"; break; case SQLITE_IOERR_CHECKRESERVEDLOCK: zName = "SQLITE_IOERR_CHECKRESERVEDLOCK"; break; case SQLITE_IOERR_LOCK: zName = "SQLITE_IOERR_LOCK"; break; case SQLITE_CORRUPT_VTAB: zName = "SQLITE_CORRUPT_VTAB"; break; | | > | 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 | case SQLITE_IOERR_BLOCKED: zName = "SQLITE_IOERR_BLOCKED"; break; case SQLITE_IOERR_NOMEM: zName = "SQLITE_IOERR_NOMEM"; break; case SQLITE_IOERR_ACCESS: zName = "SQLITE_IOERR_ACCESS"; break; case SQLITE_IOERR_CHECKRESERVEDLOCK: zName = "SQLITE_IOERR_CHECKRESERVEDLOCK"; break; case SQLITE_IOERR_LOCK: zName = "SQLITE_IOERR_LOCK"; break; case SQLITE_CORRUPT_VTAB: zName = "SQLITE_CORRUPT_VTAB"; break; case SQLITE_READONLY_RECOVERY: zName = "SQLITE_READONLY_RECOVERY"; break; case SQLITE_READONLY_CANTLOCK: zName = "SQLITE_READONLY_CANTLOCK"; break; default: zName = "SQLITE_Unknown"; break; } return zName; } #define t1ErrorName sqlite3TestErrorName /* |
︙ | ︙ |
Changes to src/test_quota.c.
︙ | ︙ | |||
319 320 321 322 323 324 325 | pSubOpen = quotaSubOpen(pConn); rc = pOrigVfs->xOpen(pOrigVfs, zName, pSubOpen, flags, pOutFlags); if( rc==SQLITE_OK ){ for(pFile=pGroup->pFiles; pFile && strcmp(pFile->zFilename, zName); pFile=pFile->pNext){} if( pFile==0 ){ int nName = strlen(zName); | | | 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 | pSubOpen = quotaSubOpen(pConn); rc = pOrigVfs->xOpen(pOrigVfs, zName, pSubOpen, flags, pOutFlags); if( rc==SQLITE_OK ){ for(pFile=pGroup->pFiles; pFile && strcmp(pFile->zFilename, zName); pFile=pFile->pNext){} if( pFile==0 ){ int nName = strlen(zName); pFile = (quotaFile *)sqlite3_malloc( sizeof(*pFile) + nName + 1 ); if( pFile==0 ){ quotaLeave(); pSubOpen->pMethods->xClose(pSubOpen); return SQLITE_NOMEM; } memset(pFile, 0, sizeof(*pFile)); pFile->zFilename = (char*)&pFile[1]; |
︙ | ︙ | |||
679 680 681 682 683 684 685 | } if( pGroup==0 ){ int nPattern = strlen(zPattern); if( iLimit<=0 ){ quotaLeave(); return SQLITE_OK; } | | | 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 | } if( pGroup==0 ){ int nPattern = strlen(zPattern); if( iLimit<=0 ){ quotaLeave(); return SQLITE_OK; } pGroup = (quotaGroup *)sqlite3_malloc( sizeof(*pGroup) + nPattern + 1 ); if( pGroup==0 ){ quotaLeave(); return SQLITE_NOMEM; } memset(pGroup, 0, sizeof(*pGroup)); pGroup->zPattern = (char*)&pGroup[1]; memcpy((char *)pGroup->zPattern, zPattern, nPattern+1); |
︙ | ︙ |
Changes to src/tokenize.c.
︙ | ︙ | |||
349 350 351 352 353 354 355 | return i; } #ifndef SQLITE_OMIT_BLOB_LITERAL case 'x': case 'X': { testcase( z[0]=='x' ); testcase( z[0]=='X' ); if( z[1]=='\'' ){ *tokenType = TK_BLOB; | < | > | > | < < | | 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 | return i; } #ifndef SQLITE_OMIT_BLOB_LITERAL case 'x': case 'X': { testcase( z[0]=='x' ); testcase( z[0]=='X' ); if( z[1]=='\'' ){ *tokenType = TK_BLOB; for(i=2; sqlite3Isxdigit(z[i]); i++){} if( z[i]!='\'' || i%2 ){ *tokenType = TK_ILLEGAL; while( z[i] && z[i]!='\'' ){ i++; } } if( z[i] ) i++; return i; } /* Otherwise fall through to the next case */ } #endif default: { if( !IdChar(*z) ){ |
︙ | ︙ | |||
408 409 410 411 412 413 414 | if( pEngine==0 ){ db->mallocFailed = 1; return SQLITE_NOMEM; } assert( pParse->pNewTable==0 ); assert( pParse->pNewTrigger==0 ); assert( pParse->nVar==0 ); | | < | | 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 | if( pEngine==0 ){ db->mallocFailed = 1; return SQLITE_NOMEM; } assert( pParse->pNewTable==0 ); assert( pParse->pNewTrigger==0 ); assert( pParse->nVar==0 ); assert( pParse->nzVar==0 ); assert( pParse->azVar==0 ); enableLookaside = db->lookaside.bEnabled; if( db->lookaside.pStart ) db->lookaside.bEnabled = 1; while( !db->mallocFailed && zSql[i]!=0 ){ assert( i>=0 ); pParse->sLastToken.z = &zSql[i]; pParse->sLastToken.n = sqlite3GetToken((unsigned char*)&zSql[i],&tokenType); i += pParse->sLastToken.n; |
︙ | ︙ | |||
504 505 506 507 508 509 510 | ** structure built up in pParse->pNewTable. The calling code (see vtab.c) ** will take responsibility for freeing the Table structure. */ sqlite3DeleteTable(db, pParse->pNewTable); } sqlite3DeleteTrigger(db, pParse->pNewTrigger); | > | | 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 | ** structure built up in pParse->pNewTable. The calling code (see vtab.c) ** will take responsibility for freeing the Table structure. */ sqlite3DeleteTable(db, pParse->pNewTable); } sqlite3DeleteTrigger(db, pParse->pNewTrigger); for(i=pParse->nzVar-1; i>=0; i--) sqlite3DbFree(db, pParse->azVar[i]); sqlite3DbFree(db, pParse->azVar); sqlite3DbFree(db, pParse->aAlias); while( pParse->pAinc ){ AutoincInfo *p = pParse->pAinc; pParse->pAinc = p->pNext; sqlite3DbFree(db, p); } while( pParse->pZombieTab ){ |
︙ | ︙ |
Changes to src/trigger.c.
︙ | ︙ | |||
297 298 299 300 301 302 303 | z = sqlite3DbStrNDup(db, (char*)pAll->z, pAll->n); sqlite3NestedParse(pParse, "INSERT INTO %Q.%s VALUES('trigger',%Q,%Q,0,'CREATE TRIGGER %q')", db->aDb[iDb].zName, SCHEMA_TABLE(iDb), zName, pTrig->table, z); sqlite3DbFree(db, z); sqlite3ChangeCookie(pParse, iDb); | | | < | 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 | z = sqlite3DbStrNDup(db, (char*)pAll->z, pAll->n); sqlite3NestedParse(pParse, "INSERT INTO %Q.%s VALUES('trigger',%Q,%Q,0,'CREATE TRIGGER %q')", db->aDb[iDb].zName, SCHEMA_TABLE(iDb), zName, pTrig->table, z); sqlite3DbFree(db, z); sqlite3ChangeCookie(pParse, iDb); sqlite3VdbeAddParseSchemaOp(v, iDb, sqlite3MPrintf(db, "type='trigger' AND name='%q'", zName)); } if( db->init.busy ){ Trigger *pLink = pTrig; Hash *pHash = &db->aDb[iDb].pSchema->trigHash; assert( sqlite3SchemaMutexHeld(db, iDb, 0) ); pTrig = sqlite3HashInsert(pHash, zName, sqlite3Strlen30(zName), pTrig); |
︙ | ︙ |
Changes to src/update.c.
︙ | ︙ | |||
240 241 242 243 244 245 246 | for(nIdx=0, pIdx=pTab->pIndex; pIdx; pIdx=pIdx->pNext, nIdx++){} if( nIdx>0 ){ aRegIdx = sqlite3DbMallocRaw(db, sizeof(Index*) * nIdx ); if( aRegIdx==0 ) goto update_cleanup; } for(j=0, pIdx=pTab->pIndex; pIdx; pIdx=pIdx->pNext, j++){ int reg; | | | 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 | for(nIdx=0, pIdx=pTab->pIndex; pIdx; pIdx=pIdx->pNext, nIdx++){} if( nIdx>0 ){ aRegIdx = sqlite3DbMallocRaw(db, sizeof(Index*) * nIdx ); if( aRegIdx==0 ) goto update_cleanup; } for(j=0, pIdx=pTab->pIndex; pIdx; pIdx=pIdx->pNext, j++){ int reg; if( hasFK || chngRowid ){ reg = ++pParse->nMem; }else{ reg = 0; for(i=0; i<pIdx->nColumn; i++){ if( aXRef[pIdx->aiColumn[i]]>=0 ){ reg = ++pParse->nMem; break; |
︙ | ︙ |
Changes to src/utf.c.
︙ | ︙ | |||
159 160 161 162 163 164 165 | while( zIn!=zTerm && (*zIn & 0xc0)==0x80 ){ \ c = (c<<6) + (0x3f & *(zIn++)); \ } \ if( c<0x80 \ || (c&0xFFFFF800)==0xD800 \ || (c&0xFFFFFFFE)==0xFFFE ){ c = 0xFFFD; } \ } | | | 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 | while( zIn!=zTerm && (*zIn & 0xc0)==0x80 ){ \ c = (c<<6) + (0x3f & *(zIn++)); \ } \ if( c<0x80 \ || (c&0xFFFFF800)==0xD800 \ || (c&0xFFFFFFFE)==0xFFFE ){ c = 0xFFFD; } \ } u32 sqlite3Utf8Read( const unsigned char *zIn, /* First byte of UTF-8 character */ const unsigned char **pzNext /* Write first byte past UTF-8 char here */ ){ unsigned int c; /* Same as READ_UTF8() above but without the zTerm parameter. ** For this routine, we assume the UTF8 string is always zero-terminated. |
︙ | ︙ |
Changes to src/vdbe.c.
︙ | ︙ | |||
994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 | ** If the parameter is named, then its name appears in P4 and P3==1. ** The P4 value is used by sqlite3_bind_parameter_name(). */ case OP_Variable: { /* out2-prerelease */ Mem *pVar; /* Value being transferred */ assert( pOp->p1>0 && pOp->p1<=p->nVar ); pVar = &p->aVar[pOp->p1 - 1]; if( sqlite3VdbeMemTooBig(pVar) ){ goto too_big; } sqlite3VdbeMemShallowCopy(pOut, pVar, MEM_Static); UPDATE_MAX_BLOBSIZE(pOut); break; | > | 994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 1008 | ** If the parameter is named, then its name appears in P4 and P3==1. ** The P4 value is used by sqlite3_bind_parameter_name(). */ case OP_Variable: { /* out2-prerelease */ Mem *pVar; /* Value being transferred */ assert( pOp->p1>0 && pOp->p1<=p->nVar ); assert( pOp->p4.z==0 || pOp->p4.z==p->azVar[pOp->p1-1] ); pVar = &p->aVar[pOp->p1 - 1]; if( sqlite3VdbeMemTooBig(pVar) ){ goto too_big; } sqlite3VdbeMemShallowCopy(pOut, pVar, MEM_Static); UPDATE_MAX_BLOBSIZE(pOut); break; |
︙ | ︙ | |||
1404 1405 1406 1407 1408 1409 1410 1411 1412 1413 1414 1415 1416 1417 1418 1419 1420 | assert( pOp[-1].p4type==P4_COLLSEQ ); assert( pOp[-1].opcode==OP_CollSeq ); ctx.pColl = pOp[-1].p4.pColl; } db->lastRowid = lastRowid; (*ctx.pFunc->xFunc)(&ctx, n, apVal); /* IMP: R-24505-23230 */ lastRowid = db->lastRowid; if( db->mallocFailed ){ /* Even though a malloc() has failed, the implementation of the ** user function may have called an sqlite3_result_XXX() function ** to return a value. The following call releases any resources ** associated with such a value. */ sqlite3VdbeMemRelease(&ctx.s); goto no_mem; } | > > > > > > > > > > < < < < < < < < < | 1405 1406 1407 1408 1409 1410 1411 1412 1413 1414 1415 1416 1417 1418 1419 1420 1421 1422 1423 1424 1425 1426 1427 1428 1429 1430 1431 1432 1433 1434 1435 1436 1437 1438 | assert( pOp[-1].p4type==P4_COLLSEQ ); assert( pOp[-1].opcode==OP_CollSeq ); ctx.pColl = pOp[-1].p4.pColl; } db->lastRowid = lastRowid; (*ctx.pFunc->xFunc)(&ctx, n, apVal); /* IMP: R-24505-23230 */ lastRowid = db->lastRowid; /* If any auxiliary data functions have been called by this user function, ** immediately call the destructor for any non-static values. */ if( ctx.pVdbeFunc ){ sqlite3VdbeDeleteAuxData(ctx.pVdbeFunc, pOp->p1); pOp->p4.pVdbeFunc = ctx.pVdbeFunc; pOp->p4type = P4_VDBEFUNC; } if( db->mallocFailed ){ /* Even though a malloc() has failed, the implementation of the ** user function may have called an sqlite3_result_XXX() function ** to return a value. The following call releases any resources ** associated with such a value. */ sqlite3VdbeMemRelease(&ctx.s); goto no_mem; } /* If the function returned an error, throw an exception */ if( ctx.isError ){ sqlite3SetString(&p->zErrMsg, db, "%s", sqlite3_value_text(&ctx.s)); rc = ctx.isError; } /* Copy the result of the function into register P3 */ |
︙ | ︙ | |||
1772 1773 1774 1775 1776 1777 1778 | u16 flags1; /* Copy of initial value of pIn1->flags */ u16 flags3; /* Copy of initial value of pIn3->flags */ pIn1 = &aMem[pOp->p1]; pIn3 = &aMem[pOp->p3]; flags1 = pIn1->flags; flags3 = pIn3->flags; | | | | 1774 1775 1776 1777 1778 1779 1780 1781 1782 1783 1784 1785 1786 1787 1788 1789 1790 1791 1792 1793 1794 1795 1796 | u16 flags1; /* Copy of initial value of pIn1->flags */ u16 flags3; /* Copy of initial value of pIn3->flags */ pIn1 = &aMem[pOp->p1]; pIn3 = &aMem[pOp->p3]; flags1 = pIn1->flags; flags3 = pIn3->flags; if( (flags1 | flags3)&MEM_Null ){ /* One or both operands are NULL */ if( pOp->p5 & SQLITE_NULLEQ ){ /* If SQLITE_NULLEQ is set (which will only happen if the operator is ** OP_Eq or OP_Ne) then take the jump or not depending on whether ** or not both operands are null. */ assert( pOp->opcode==OP_Eq || pOp->opcode==OP_Ne ); res = (flags1 & flags3 & MEM_Null)==0; }else{ /* SQLITE_NULLEQ is clear and at least one operand is NULL, ** then the result is always NULL. ** The jump is taken if the SQLITE_JUMPIFNULL bit is set. */ if( pOp->p5 & SQLITE_STOREP2 ){ pOut = &aMem[pOp->p2]; |
︙ | ︙ | |||
2590 2591 2592 2593 2594 2595 2596 | */ sqlite3SetString(&p->zErrMsg, db, "cannot open savepoint - " "SQL statements in progress"); rc = SQLITE_BUSY; }else{ nName = sqlite3Strlen30(zName); | | | 2592 2593 2594 2595 2596 2597 2598 2599 2600 2601 2602 2603 2604 2605 2606 | */ sqlite3SetString(&p->zErrMsg, db, "cannot open savepoint - " "SQL statements in progress"); rc = SQLITE_BUSY; }else{ nName = sqlite3Strlen30(zName); #ifndef SQLITE_OMIT_VIRTUALTABLE /* This call is Ok even if this savepoint is actually a transaction ** savepoint (and therefore should not prompt xSavepoint()) callbacks. ** If this is a transaction savepoint being opened, it is guaranteed ** that the db->aVTrans[] array is empty. */ assert( db->autoCommit==0 || db->nVTrans==0 ); rc = sqlite3VtabSavepoint(db, SAVEPOINT_BEGIN, db->nStatement+db->nSavepoint); |
︙ | ︙ | |||
5930 5931 5932 5933 5934 5935 5936 5937 | /* Opcode: Trace * * * P4 * ** ** If tracing is enabled (by the sqlite3_trace()) interface, then ** the UTF-8 string contained in P4 is emitted on the trace callback. */ case OP_Trace: { char *zTrace; | > | < < | | | | | > > | | < | 5932 5933 5934 5935 5936 5937 5938 5939 5940 5941 5942 5943 5944 5945 5946 5947 5948 5949 5950 5951 5952 5953 5954 5955 5956 5957 5958 5959 | /* Opcode: Trace * * * P4 * ** ** If tracing is enabled (by the sqlite3_trace()) interface, then ** the UTF-8 string contained in P4 is emitted on the trace callback. */ case OP_Trace: { char *zTrace; char *z; if( db->xTrace && (zTrace = (pOp->p4.z ? pOp->p4.z : p->zSql))!=0 ){ z = sqlite3VdbeExpandSql(p, zTrace); db->xTrace(db->pTraceArg, z); sqlite3DbFree(db, z); } #ifdef SQLITE_DEBUG if( (db->flags & SQLITE_SqlTrace)!=0 && (zTrace = (pOp->p4.z ? pOp->p4.z : p->zSql))!=0 ){ sqlite3DebugPrintf("SQL-trace: %s\n", zTrace); } #endif /* SQLITE_DEBUG */ break; } #endif /* Opcode: Noop * * * * * ** |
︙ | ︙ |
Changes to src/vdbe.h.
︙ | ︙ | |||
170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 | int sqlite3VdbeAddOp0(Vdbe*,int); int sqlite3VdbeAddOp1(Vdbe*,int,int); int sqlite3VdbeAddOp2(Vdbe*,int,int,int); int sqlite3VdbeAddOp3(Vdbe*,int,int,int,int); int sqlite3VdbeAddOp4(Vdbe*,int,int,int,int,const char *zP4,int); int sqlite3VdbeAddOp4Int(Vdbe*,int,int,int,int,int); int sqlite3VdbeAddOpList(Vdbe*, int nOp, VdbeOpList const *aOp); void sqlite3VdbeChangeP1(Vdbe*, int addr, int P1); void sqlite3VdbeChangeP2(Vdbe*, int addr, int P2); void sqlite3VdbeChangeP3(Vdbe*, int addr, int P3); void sqlite3VdbeChangeP5(Vdbe*, u8 P5); void sqlite3VdbeJumpHere(Vdbe*, int addr); void sqlite3VdbeChangeToNoop(Vdbe*, int addr, int N); void sqlite3VdbeChangeP4(Vdbe*, int addr, const char *zP4, int N); void sqlite3VdbeUsesBtree(Vdbe*, int); VdbeOp *sqlite3VdbeGetOp(Vdbe*, int); int sqlite3VdbeMakeLabel(Vdbe*); void sqlite3VdbeRunOnlyOnce(Vdbe*); void sqlite3VdbeDelete(Vdbe*); void sqlite3VdbeDeleteObject(sqlite3*,Vdbe*); | > | > | 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 | int sqlite3VdbeAddOp0(Vdbe*,int); int sqlite3VdbeAddOp1(Vdbe*,int,int); int sqlite3VdbeAddOp2(Vdbe*,int,int,int); int sqlite3VdbeAddOp3(Vdbe*,int,int,int,int); int sqlite3VdbeAddOp4(Vdbe*,int,int,int,int,const char *zP4,int); int sqlite3VdbeAddOp4Int(Vdbe*,int,int,int,int,int); int sqlite3VdbeAddOpList(Vdbe*, int nOp, VdbeOpList const *aOp); void sqlite3VdbeAddParseSchemaOp(Vdbe*,int,char*); void sqlite3VdbeChangeP1(Vdbe*, int addr, int P1); void sqlite3VdbeChangeP2(Vdbe*, int addr, int P2); void sqlite3VdbeChangeP3(Vdbe*, int addr, int P3); void sqlite3VdbeChangeP5(Vdbe*, u8 P5); void sqlite3VdbeJumpHere(Vdbe*, int addr); void sqlite3VdbeChangeToNoop(Vdbe*, int addr, int N); void sqlite3VdbeChangeP4(Vdbe*, int addr, const char *zP4, int N); void sqlite3VdbeUsesBtree(Vdbe*, int); VdbeOp *sqlite3VdbeGetOp(Vdbe*, int); int sqlite3VdbeMakeLabel(Vdbe*); void sqlite3VdbeRunOnlyOnce(Vdbe*); void sqlite3VdbeDelete(Vdbe*); void sqlite3VdbeDeleteObject(sqlite3*,Vdbe*); void sqlite3VdbeMakeReady(Vdbe*,Parse*); int sqlite3VdbeFinalize(Vdbe*); void sqlite3VdbeResolveLabel(Vdbe*, int); int sqlite3VdbeCurrentAddr(Vdbe*); #ifdef SQLITE_DEBUG int sqlite3VdbeAssertMayAbort(Vdbe *, int); void sqlite3VdbeTrace(Vdbe*,FILE*); #endif void sqlite3VdbeResetStepResult(Vdbe*); void sqlite3VdbeRewind(Vdbe*); int sqlite3VdbeReset(Vdbe*); void sqlite3VdbeSetNumCols(Vdbe*,int); int sqlite3VdbeSetColName(Vdbe*, int, int, const char *, void(*)(void*)); void sqlite3VdbeCountChanges(Vdbe*); sqlite3 *sqlite3VdbeDb(Vdbe*); void sqlite3VdbeSetSql(Vdbe*, const char *z, int n, int); void sqlite3VdbeSwap(Vdbe*,Vdbe*); |
︙ | ︙ |
Changes to src/vdbeInt.h.
︙ | ︙ | |||
283 284 285 286 287 288 289 290 291 292 293 | u32 magic; /* Magic number for sanity checking */ char *zErrMsg; /* Error message written here */ Vdbe *pPrev,*pNext; /* Linked list of VDBEs with the same Vdbe.db */ VdbeCursor **apCsr; /* One element of this array for each open cursor */ Mem *aVar; /* Values for the OP_Variable opcode. */ char **azVar; /* Name of variables */ ynVar nVar; /* Number of entries in aVar[] */ u32 cacheCtr; /* VdbeCursor row cache generation counter */ int pc; /* The program counter */ int rc; /* Value to return */ u8 errorAction; /* Recovery action to do in case of an error */ | > < | 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 | u32 magic; /* Magic number for sanity checking */ char *zErrMsg; /* Error message written here */ Vdbe *pPrev,*pNext; /* Linked list of VDBEs with the same Vdbe.db */ VdbeCursor **apCsr; /* One element of this array for each open cursor */ Mem *aVar; /* Values for the OP_Variable opcode. */ char **azVar; /* Name of variables */ ynVar nVar; /* Number of entries in aVar[] */ ynVar nzVar; /* Number of entries in azVar[] */ u32 cacheCtr; /* VdbeCursor row cache generation counter */ int pc; /* The program counter */ int rc; /* Value to return */ u8 errorAction; /* Recovery action to do in case of an error */ u8 explain; /* True if EXPLAIN present on SQL command */ u8 changeCntOn; /* True to update the change-counter */ u8 expired; /* True if the VM needs to be recompiled */ u8 runOnlyOnce; /* Automatically expire on reset */ u8 minWriteFileFormat; /* Minimum file format for writable database files */ u8 inVtabMethod; /* See comments above */ u8 usesStmtJournal; /* True if uses a statement journal */ |
︙ | ︙ |
Changes to src/vdbeapi.c.
︙ | ︙ | |||
98 99 100 101 102 103 104 | int rc; if( pStmt==0 ){ rc = SQLITE_OK; }else{ Vdbe *v = (Vdbe*)pStmt; sqlite3_mutex_enter(v->db->mutex); rc = sqlite3VdbeReset(v); | | | 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 | int rc; if( pStmt==0 ){ rc = SQLITE_OK; }else{ Vdbe *v = (Vdbe*)pStmt; sqlite3_mutex_enter(v->db->mutex); rc = sqlite3VdbeReset(v); sqlite3VdbeRewind(v); assert( (rc & (v->db->errMask))==rc ); rc = sqlite3ApiExit(v->db, rc); sqlite3_mutex_leave(v->db->mutex); } return rc; } |
︙ | ︙ | |||
1183 1184 1185 1186 1187 1188 1189 | ** This routine is added to support DBD::SQLite. */ int sqlite3_bind_parameter_count(sqlite3_stmt *pStmt){ Vdbe *p = (Vdbe*)pStmt; return p ? p->nVar : 0; } | < < < < < < < < < < < < < < < < < < < < < < < < < < | < < | | 1183 1184 1185 1186 1187 1188 1189 1190 1191 1192 1193 1194 1195 1196 1197 1198 1199 1200 1201 1202 1203 1204 1205 1206 1207 1208 1209 1210 1211 1212 1213 1214 1215 1216 1217 1218 1219 1220 1221 1222 | ** This routine is added to support DBD::SQLite. */ int sqlite3_bind_parameter_count(sqlite3_stmt *pStmt){ Vdbe *p = (Vdbe*)pStmt; return p ? p->nVar : 0; } /* ** Return the name of a wildcard parameter. Return NULL if the index ** is out of range or if the wildcard is unnamed. ** ** The result is always UTF-8. */ const char *sqlite3_bind_parameter_name(sqlite3_stmt *pStmt, int i){ Vdbe *p = (Vdbe*)pStmt; if( p==0 || i<1 || i>p->nzVar ){ return 0; } return p->azVar[i-1]; } /* ** Given a wildcard parameter name, return the index of the variable ** with that name. If there is no variable with the given name, ** return 0. */ int sqlite3VdbeParameterIndex(Vdbe *p, const char *zName, int nName){ int i; if( p==0 ){ return 0; } if( zName ){ for(i=0; i<p->nzVar; i++){ const char *z = p->azVar[i]; if( z && memcmp(z,zName,nName)==0 && z[nName]==0 ){ return i+1; } } } return 0; |
︙ | ︙ |
Changes to src/vdbeaux.c.
︙ | ︙ | |||
153 154 155 156 157 158 159 | pOp->opcode = (u8)op; pOp->p5 = 0; pOp->p1 = p1; pOp->p2 = p2; pOp->p3 = p3; pOp->p4.p = 0; pOp->p4type = P4_NOTUSED; | < < < < < < < | 153 154 155 156 157 158 159 160 161 162 163 164 165 166 | pOp->opcode = (u8)op; pOp->p5 = 0; pOp->p1 = p1; pOp->p2 = p2; pOp->p3 = p3; pOp->p4.p = 0; pOp->p4type = P4_NOTUSED; #ifdef SQLITE_DEBUG pOp->zComment = 0; if( sqlite3VdbeAddopTrace ) sqlite3VdbePrintOp(0, i, &p->aOp[i]); #endif #ifdef VDBE_PROFILE pOp->cycles = 0; pOp->cnt = 0; |
︙ | ︙ | |||
197 198 199 200 201 202 203 204 205 206 207 208 209 210 | const char *zP4, /* The P4 operand */ int p4type /* P4 operand type */ ){ int addr = sqlite3VdbeAddOp3(p, op, p1, p2, p3); sqlite3VdbeChangeP4(p, addr, zP4, p4type); return addr; } /* ** Add an opcode that includes the p4 value as an integer. */ int sqlite3VdbeAddOp4Int( Vdbe *p, /* Add the opcode to this VM */ int op, /* The new opcode */ | > > > > > > > > > > > > > > | 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 | const char *zP4, /* The P4 operand */ int p4type /* P4 operand type */ ){ int addr = sqlite3VdbeAddOp3(p, op, p1, p2, p3); sqlite3VdbeChangeP4(p, addr, zP4, p4type); return addr; } /* ** Add an OP_ParseSchema opcode. This routine is broken out from ** sqlite3VdbeAddOp4() since it needs to also local all btrees. ** ** The zWhere string must have been obtained from sqlite3_malloc(). ** This routine will take ownership of the allocated memory. */ void sqlite3VdbeAddParseSchemaOp(Vdbe *p, int iDb, char *zWhere){ int j; int addr = sqlite3VdbeAddOp3(p, OP_ParseSchema, iDb, 0, 0); sqlite3VdbeChangeP4(p, addr, zWhere, P4_DYNAMIC); for(j=0; j<p->db->nDb; j++) sqlite3VdbeUsesBtree(p, j); } /* ** Add an opcode that includes the p4 value as an integer. */ int sqlite3VdbeAddOp4Int( Vdbe *p, /* Add the opcode to this VM */ int op, /* The new opcode */ |
︙ | ︙ | |||
1388 1389 1390 1391 1392 1393 1394 | }else{ *pnByte += nByte; } return pBuf; } /* | < < | < | < < < < < < < < < < | < < < < < < < < > | < | > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > | < < < | | < | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | > > > > > | | | | | | | | < < < < < < < < < < | < < < < < < < < | < < < < < < | 1395 1396 1397 1398 1399 1400 1401 1402 1403 1404 1405 1406 1407 1408 1409 1410 1411 1412 1413 1414 1415 1416 1417 1418 1419 1420 1421 1422 1423 1424 1425 1426 1427 1428 1429 1430 1431 1432 1433 1434 1435 1436 1437 1438 1439 1440 1441 1442 1443 1444 1445 1446 1447 1448 1449 1450 1451 1452 1453 1454 1455 1456 1457 1458 1459 1460 1461 1462 1463 1464 1465 1466 1467 1468 1469 1470 1471 1472 1473 1474 1475 1476 1477 1478 1479 1480 1481 1482 1483 1484 1485 1486 1487 1488 1489 1490 1491 1492 1493 1494 1495 1496 1497 1498 1499 1500 1501 1502 1503 1504 1505 1506 1507 1508 1509 1510 1511 1512 1513 1514 1515 1516 1517 1518 1519 1520 1521 1522 1523 1524 1525 1526 1527 1528 1529 1530 1531 1532 1533 1534 1535 1536 1537 1538 1539 1540 1541 1542 1543 1544 1545 1546 1547 1548 1549 1550 1551 1552 1553 1554 1555 1556 1557 1558 1559 1560 1561 1562 1563 1564 | }else{ *pnByte += nByte; } return pBuf; } /* ** Rewind the VDBE back to the beginning in preparation for ** running it. */ void sqlite3VdbeRewind(Vdbe *p){ #if defined(SQLITE_DEBUG) || defined(VDBE_PROFILE) int i; #endif assert( p!=0 ); assert( p->magic==VDBE_MAGIC_INIT ); /* There should be at least one opcode. */ assert( p->nOp>0 ); /* Set the magic to VDBE_MAGIC_RUN sooner rather than later. */ p->magic = VDBE_MAGIC_RUN; #ifdef SQLITE_DEBUG for(i=1; i<p->nMem; i++){ assert( p->aMem[i].db==p->db ); } #endif p->pc = -1; p->rc = SQLITE_OK; p->errorAction = OE_Abort; p->magic = VDBE_MAGIC_RUN; p->nChange = 0; p->cacheCtr = 1; p->minWriteFileFormat = 255; p->iStatement = 0; p->nFkConstraint = 0; #ifdef VDBE_PROFILE for(i=0; i<p->nOp; i++){ p->aOp[i].cnt = 0; p->aOp[i].cycles = 0; } #endif } /* ** Prepare a virtual machine for execution for the first time after ** creating the virtual machine. This involves things such ** as allocating stack space and initializing the program counter. ** After the VDBE has be prepped, it can be executed by one or more ** calls to sqlite3VdbeExec(). ** ** This function may be called exact once on a each virtual machine. ** After this routine is called the VM has been "packaged" and is ready ** to run. After this routine is called, futher calls to ** sqlite3VdbeAddOp() functions are prohibited. This routine disconnects ** the Vdbe from the Parse object that helped generate it so that the ** the Vdbe becomes an independent entity and the Parse object can be ** destroyed. ** ** Use the sqlite3VdbeRewind() procedure to restore a virtual machine back ** to its initial state after it has been run. */ void sqlite3VdbeMakeReady( Vdbe *p, /* The VDBE */ Parse *pParse /* Parsing context */ ){ sqlite3 *db; /* The database connection */ int nVar; /* Number of parameters */ int nMem; /* Number of VM memory registers */ int nCursor; /* Number of cursors required */ int nArg; /* Number of arguments in subprograms */ int n; /* Loop counter */ u8 *zCsr; /* Memory available for allocation */ u8 *zEnd; /* First byte past allocated memory */ int nByte; /* How much extra memory is needed */ assert( p!=0 ); assert( p->nOp>0 ); assert( pParse!=0 ); assert( p->magic==VDBE_MAGIC_INIT ); db = p->db; assert( db->mallocFailed==0 ); nVar = pParse->nVar; nMem = pParse->nMem; nCursor = pParse->nTab; nArg = pParse->nMaxArg; /* For each cursor required, also allocate a memory cell. Memory ** cells (nMem+1-nCursor)..nMem, inclusive, will never be used by ** the vdbe program. Instead they are used to allocate space for ** VdbeCursor/BtCursor structures. The blob of memory associated with ** cursor 0 is stored in memory cell nMem. Memory cell (nMem-1) ** stores the blob of memory associated with cursor 1, etc. ** ** See also: allocateCursor(). */ nMem += nCursor; /* Allocate space for memory registers, SQL variables, VDBE cursors and ** an array to marshal SQL function arguments in. */ zCsr = (u8*)&p->aOp[p->nOp]; /* Memory avaliable for allocation */ zEnd = (u8*)&p->aOp[p->nOpAlloc]; /* First byte past end of zCsr[] */ resolveP2Values(p, &nArg); p->usesStmtJournal = (u8)(pParse->isMultiWrite && pParse->mayAbort); if( pParse->explain && nMem<10 ){ nMem = 10; } memset(zCsr, 0, zEnd-zCsr); zCsr += (zCsr - (u8*)0)&7; assert( EIGHT_BYTE_ALIGNMENT(zCsr) ); /* Memory for registers, parameters, cursor, etc, is allocated in two ** passes. On the first pass, we try to reuse unused space at the ** end of the opcode array. If we are unable to satisfy all memory ** requirements by reusing the opcode array tail, then the second ** pass will fill in the rest using a fresh allocation. ** ** This two-pass approach that reuses as much memory as possible from ** the leftover space at the end of the opcode array can significantly ** reduce the amount of memory held by a prepared statement. */ do { nByte = 0; p->aMem = allocSpace(p->aMem, nMem*sizeof(Mem), &zCsr, zEnd, &nByte); p->aVar = allocSpace(p->aVar, nVar*sizeof(Mem), &zCsr, zEnd, &nByte); p->apArg = allocSpace(p->apArg, nArg*sizeof(Mem*), &zCsr, zEnd, &nByte); p->azVar = allocSpace(p->azVar, nVar*sizeof(char*), &zCsr, zEnd, &nByte); p->apCsr = allocSpace(p->apCsr, nCursor*sizeof(VdbeCursor*), &zCsr, zEnd, &nByte); if( nByte ){ p->pFree = sqlite3DbMallocZero(db, nByte); } zCsr = p->pFree; zEnd = &zCsr[nByte]; }while( nByte && !db->mallocFailed ); p->nCursor = (u16)nCursor; if( p->aVar ){ p->nVar = (ynVar)nVar; for(n=0; n<nVar; n++){ p->aVar[n].flags = MEM_Null; p->aVar[n].db = db; } } if( p->azVar ){ p->nzVar = pParse->nzVar; memcpy(p->azVar, pParse->azVar, p->nzVar*sizeof(p->azVar[0])); memset(pParse->azVar, 0, pParse->nzVar*sizeof(pParse->azVar[0])); } if( p->aMem ){ p->aMem--; /* aMem[] goes from 1..nMem */ p->nMem = nMem; /* not from 0..nMem-1 */ for(n=1; n<=nMem; n++){ p->aMem[n].flags = MEM_Null; p->aMem[n].db = db; } } p->explain = pParse->explain; sqlite3VdbeRewind(p); } /* ** Close a VDBE cursor and release all the resources that cursor ** happens to hold. */ void sqlite3VdbeFreeCursor(Vdbe *p, VdbeCursor *pCx){ |
︙ | ︙ | |||
2396 2397 2398 2399 2400 2401 2402 2403 2404 2405 2406 2407 2408 2409 2410 2411 2412 2413 2414 2415 2416 2417 | ** Free all memory associated with the Vdbe passed as the second argument. ** The difference between this function and sqlite3VdbeDelete() is that ** VdbeDelete() also unlinks the Vdbe from the list of VMs associated with ** the database connection. */ void sqlite3VdbeDeleteObject(sqlite3 *db, Vdbe *p){ SubProgram *pSub, *pNext; assert( p->db==0 || p->db==db ); releaseMemArray(p->aVar, p->nVar); releaseMemArray(p->aColName, p->nResColumn*COLNAME_N); for(pSub=p->pProgram; pSub; pSub=pNext){ pNext = pSub->pNext; vdbeFreeOpArray(db, pSub->aOp, pSub->nOp); sqlite3DbFree(db, pSub); } vdbeFreeOpArray(db, p->aOp, p->nOp); sqlite3DbFree(db, p->aLabel); sqlite3DbFree(db, p->aColName); sqlite3DbFree(db, p->zSql); sqlite3DbFree(db, p->pFree); sqlite3DbFree(db, p); } | > > | 2424 2425 2426 2427 2428 2429 2430 2431 2432 2433 2434 2435 2436 2437 2438 2439 2440 2441 2442 2443 2444 2445 2446 2447 | ** Free all memory associated with the Vdbe passed as the second argument. ** The difference between this function and sqlite3VdbeDelete() is that ** VdbeDelete() also unlinks the Vdbe from the list of VMs associated with ** the database connection. */ void sqlite3VdbeDeleteObject(sqlite3 *db, Vdbe *p){ SubProgram *pSub, *pNext; int i; assert( p->db==0 || p->db==db ); releaseMemArray(p->aVar, p->nVar); releaseMemArray(p->aColName, p->nResColumn*COLNAME_N); for(pSub=p->pProgram; pSub; pSub=pNext){ pNext = pSub->pNext; vdbeFreeOpArray(db, pSub->aOp, pSub->nOp); sqlite3DbFree(db, pSub); } for(i=p->nzVar-1; i>=0; i--) sqlite3DbFree(db, p->azVar[i]); vdbeFreeOpArray(db, p->aOp, p->nOp); sqlite3DbFree(db, p->aLabel); sqlite3DbFree(db, p->aColName); sqlite3DbFree(db, p->zSql); sqlite3DbFree(db, p->pFree); sqlite3DbFree(db, p); } |
︙ | ︙ | |||
2849 2850 2851 2852 2853 2854 2855 | u = 0; while( idx<szHdr && u<p->nField && d<=nKey ){ u32 serial_type; idx += getVarint32(&aKey[idx], serial_type); pMem->enc = pKeyInfo->enc; pMem->db = pKeyInfo->db; | | > > | 2879 2880 2881 2882 2883 2884 2885 2886 2887 2888 2889 2890 2891 2892 2893 2894 2895 2896 2897 2898 2899 2900 2901 2902 2903 2904 2905 2906 2907 2908 2909 2910 2911 2912 2913 2914 2915 2916 2917 2918 2919 2920 2921 2922 2923 | u = 0; while( idx<szHdr && u<p->nField && d<=nKey ){ u32 serial_type; idx += getVarint32(&aKey[idx], serial_type); pMem->enc = pKeyInfo->enc; pMem->db = pKeyInfo->db; /* pMem->flags = 0; // sqlite3VdbeSerialGet() will set this for us */ pMem->zMalloc = 0; pMem->z = 0; d += sqlite3VdbeSerialGet(&aKey[d], serial_type, pMem); pMem++; u++; } assert( u<=pKeyInfo->nField + 1 ); p->nField = u; return (void*)p; } /* ** This routine destroys a UnpackedRecord object. */ void sqlite3VdbeDeleteUnpackedRecord(UnpackedRecord *p){ #ifdef SQLITE_DEBUG int i; Mem *pMem; assert( p!=0 ); assert( p->flags & UNPACKED_NEED_DESTROY ); for(i=0, pMem=p->aMem; i<p->nField; i++, pMem++){ /* The unpacked record is always constructed by the ** sqlite3VdbeUnpackRecord() function above, which makes all ** strings and blobs static. And none of the elements are ** ever transformed, so there is never anything to delete. */ if( pMem->zMalloc ) sqlite3VdbeMemRelease(pMem); } #endif if( p->flags & UNPACKED_NEED_FREE ){ sqlite3DbFree(p->pKeyInfo->db, p); } } /* ** This function compares the two table rows or index records |
︙ | ︙ |
Changes to src/vdbeblob.c.
︙ | ︙ | |||
293 294 295 296 297 298 299 | ** always return an SQL NULL. This is useful because it means ** we can invoke OP_Column to fill in the vdbe cursors type ** and offset cache without causing any IO. */ sqlite3VdbeChangeP4(v, 3+flags, SQLITE_INT_TO_PTR(pTab->nCol+1),P4_INT32); sqlite3VdbeChangeP2(v, 7, pTab->nCol); if( !db->mallocFailed ){ | > > > | | 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 | ** always return an SQL NULL. This is useful because it means ** we can invoke OP_Column to fill in the vdbe cursors type ** and offset cache without causing any IO. */ sqlite3VdbeChangeP4(v, 3+flags, SQLITE_INT_TO_PTR(pTab->nCol+1),P4_INT32); sqlite3VdbeChangeP2(v, 7, pTab->nCol); if( !db->mallocFailed ){ pParse->nVar = 1; pParse->nMem = 1; pParse->nTab = 1; sqlite3VdbeMakeReady(v, pParse); } } pBlob->flags = flags; pBlob->iCol = iCol; pBlob->db = db; sqlite3BtreeLeaveAll(db); |
︙ | ︙ |
Changes to src/vtab.c.
︙ | ︙ | |||
379 380 381 382 383 384 385 | ); sqlite3DbFree(db, zStmt); v = sqlite3GetVdbe(pParse); sqlite3ChangeCookie(pParse, iDb); sqlite3VdbeAddOp2(v, OP_Expire, 0, 0); zWhere = sqlite3MPrintf(db, "name='%q' AND type='table'", pTab->zName); | | | 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 | ); sqlite3DbFree(db, zStmt); v = sqlite3GetVdbe(pParse); sqlite3ChangeCookie(pParse, iDb); sqlite3VdbeAddOp2(v, OP_Expire, 0, 0); zWhere = sqlite3MPrintf(db, "name='%q' AND type='table'", pTab->zName); sqlite3VdbeAddParseSchemaOp(v, iDb, zWhere); sqlite3VdbeAddOp4(v, OP_VCreate, iDb, 0, 0, pTab->zName, sqlite3Strlen30(pTab->zName) + 1); } /* If we are rereading the sqlite_master table create the in-memory ** record of the table. The xConnect() method is not called until ** the first time the virtual table is used in an SQL statement. This |
︙ | ︙ |
Changes to src/wal.c.
︙ | ︙ | |||
416 417 418 419 420 421 422 | int nWiData; /* Size of array apWiData */ volatile u32 **apWiData; /* Pointer to wal-index content in memory */ u32 szPage; /* Database page size */ i16 readLock; /* Which read lock is being held. -1 for none */ u8 exclusiveMode; /* Non-zero if connection is in exclusive mode */ u8 writeLock; /* True if in a write transaction */ u8 ckptLock; /* True if holding a checkpoint lock */ | | > > > > > > > | 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 | int nWiData; /* Size of array apWiData */ volatile u32 **apWiData; /* Pointer to wal-index content in memory */ u32 szPage; /* Database page size */ i16 readLock; /* Which read lock is being held. -1 for none */ u8 exclusiveMode; /* Non-zero if connection is in exclusive mode */ u8 writeLock; /* True if in a write transaction */ u8 ckptLock; /* True if holding a checkpoint lock */ u8 readOnly; /* WAL_RDWR, WAL_RDONLY, or WAL_SHM_RDONLY */ WalIndexHdr hdr; /* Wal-index header for current transaction */ const char *zWalName; /* Name of WAL file */ u32 nCkpt; /* Checkpoint sequence counter in the wal-header */ #ifdef SQLITE_DEBUG u8 lockError; /* True if a locking error has occurred */ #endif }; /* ** Candidate values for Wal.exclusiveMode. */ #define WAL_NORMAL_MODE 0 #define WAL_EXCLUSIVE_MODE 1 #define WAL_HEAPMEMORY_MODE 2 /* ** Possible values for WAL.readOnly */ #define WAL_RDWR 0 /* Normal read/write connection */ #define WAL_RDONLY 1 /* The WAL file is readonly */ #define WAL_SHM_RDONLY 2 /* The SHM file is readonly */ /* ** Each page of the wal-index mapping contains a hash-table made up of ** an array of HASHTABLE_NSLOT elements of the following type. */ typedef u16 ht_slot; /* |
︙ | ︙ | |||
525 526 527 528 529 530 531 532 533 534 535 536 537 538 | if( pWal->exclusiveMode==WAL_HEAPMEMORY_MODE ){ pWal->apWiData[iPage] = (u32 volatile *)sqlite3MallocZero(WALINDEX_PGSZ); if( !pWal->apWiData[iPage] ) rc = SQLITE_NOMEM; }else{ rc = sqlite3OsShmMap(pWal->pDbFd, iPage, WALINDEX_PGSZ, pWal->writeLock, (void volatile **)&pWal->apWiData[iPage] ); } } *ppPage = pWal->apWiData[iPage]; assert( iPage==0 || *ppPage || rc!=SQLITE_OK ); return rc; } | > > > > | 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 | if( pWal->exclusiveMode==WAL_HEAPMEMORY_MODE ){ pWal->apWiData[iPage] = (u32 volatile *)sqlite3MallocZero(WALINDEX_PGSZ); if( !pWal->apWiData[iPage] ) rc = SQLITE_NOMEM; }else{ rc = sqlite3OsShmMap(pWal->pDbFd, iPage, WALINDEX_PGSZ, pWal->writeLock, (void volatile **)&pWal->apWiData[iPage] ); if( rc==SQLITE_READONLY ){ pWal->readOnly |= WAL_SHM_RDONLY; rc = SQLITE_OK; } } } *ppPage = pWal->apWiData[iPage]; assert( iPage==0 || *ppPage || rc!=SQLITE_OK ); return rc; } |
︙ | ︙ | |||
1272 1273 1274 1275 1276 1277 1278 | pRet->zWalName = zWalName; pRet->exclusiveMode = (bNoShm ? WAL_HEAPMEMORY_MODE: WAL_NORMAL_MODE); /* Open file handle on the write-ahead log file. */ flags = (SQLITE_OPEN_READWRITE|SQLITE_OPEN_CREATE|SQLITE_OPEN_WAL); rc = sqlite3OsOpen(pVfs, zWalName, pRet->pWalFd, flags, &flags); if( rc==SQLITE_OK && flags&SQLITE_OPEN_READONLY ){ | | | 1283 1284 1285 1286 1287 1288 1289 1290 1291 1292 1293 1294 1295 1296 1297 | pRet->zWalName = zWalName; pRet->exclusiveMode = (bNoShm ? WAL_HEAPMEMORY_MODE: WAL_NORMAL_MODE); /* Open file handle on the write-ahead log file. */ flags = (SQLITE_OPEN_READWRITE|SQLITE_OPEN_CREATE|SQLITE_OPEN_WAL); rc = sqlite3OsOpen(pVfs, zWalName, pRet->pWalFd, flags, &flags); if( rc==SQLITE_OK && flags&SQLITE_OPEN_READONLY ){ pRet->readOnly = WAL_RDONLY; } if( rc!=SQLITE_OK ){ walIndexClose(pRet, 0); sqlite3OsClose(pRet->pWalFd); sqlite3_free(pRet); }else{ |
︙ | ︙ | |||
1913 1914 1915 1916 1917 1918 1919 | */ badHdr = (page0 ? walIndexTryHdr(pWal, pChanged) : 1); /* If the first attempt failed, it might have been due to a race ** with a writer. So get a WRITE lock and try again. */ assert( badHdr==0 || pWal->writeLock==0 ); | > > > > > > | | | | | | | | | | | | | | | > | 1924 1925 1926 1927 1928 1929 1930 1931 1932 1933 1934 1935 1936 1937 1938 1939 1940 1941 1942 1943 1944 1945 1946 1947 1948 1949 1950 1951 1952 1953 1954 1955 1956 1957 1958 1959 | */ badHdr = (page0 ? walIndexTryHdr(pWal, pChanged) : 1); /* If the first attempt failed, it might have been due to a race ** with a writer. So get a WRITE lock and try again. */ assert( badHdr==0 || pWal->writeLock==0 ); if( badHdr ){ if( pWal->readOnly & WAL_SHM_RDONLY ){ if( SQLITE_OK==(rc = walLockShared(pWal, WAL_WRITE_LOCK)) ){ walUnlockShared(pWal, WAL_WRITE_LOCK); rc = SQLITE_READONLY_RECOVERY; } }else if( SQLITE_OK==(rc = walLockExclusive(pWal, WAL_WRITE_LOCK, 1)) ){ pWal->writeLock = 1; if( SQLITE_OK==(rc = walIndexPage(pWal, 0, &page0)) ){ badHdr = walIndexTryHdr(pWal, pChanged); if( badHdr ){ /* If the wal-index header is still malformed even while holding ** a WRITE lock, it can only mean that the header is corrupted and ** needs to be reconstructed. So run recovery to do exactly that. */ rc = walIndexRecover(pWal); *pChanged = 1; } } pWal->writeLock = 0; walUnlockExclusive(pWal, WAL_WRITE_LOCK, 1); } } /* If the header is read successfully, check the version number to make ** sure the wal-index was not constructed with some future format that ** this version of SQLite cannot understand. */ if( badHdr==0 && pWal->hdr.iVersion!=WALINDEX_MAX_VERSION ){ |
︙ | ︙ | |||
2114 2115 2116 2117 2118 2119 2120 | assert( thisMark!=READMARK_NOT_USED ); mxReadMark = thisMark; mxI = i; } } /* There was once an "if" here. The extra "{" is to preserve indentation. */ { | > | > | | | 2132 2133 2134 2135 2136 2137 2138 2139 2140 2141 2142 2143 2144 2145 2146 2147 2148 2149 2150 2151 2152 2153 2154 2155 2156 2157 2158 2159 2160 2161 2162 2163 | assert( thisMark!=READMARK_NOT_USED ); mxReadMark = thisMark; mxI = i; } } /* There was once an "if" here. The extra "{" is to preserve indentation. */ { if( (pWal->readOnly & WAL_SHM_RDONLY)==0 && (mxReadMark<pWal->hdr.mxFrame || mxI==0) ){ for(i=1; i<WAL_NREADER; i++){ rc = walLockExclusive(pWal, WAL_READ_LOCK(i), 1); if( rc==SQLITE_OK ){ mxReadMark = pInfo->aReadMark[i] = pWal->hdr.mxFrame; mxI = i; walUnlockExclusive(pWal, WAL_READ_LOCK(i), 1); break; }else if( rc!=SQLITE_BUSY ){ return rc; } } } if( mxI==0 ){ assert( rc==SQLITE_BUSY || (pWal->readOnly & WAL_SHM_RDONLY)!=0 ); return rc==SQLITE_BUSY ? WAL_RETRY : SQLITE_READONLY_CANTLOCK; } rc = walLockShared(pWal, WAL_READ_LOCK(mxI)); if( rc ){ return rc==SQLITE_BUSY ? WAL_RETRY : rc; } /* Now that the read-lock has been obtained, check that neither the |
︙ | ︙ | |||
2771 2772 2773 2774 2775 2776 2777 2778 2779 2780 2781 2782 2783 2784 | int rc; /* Return code */ int isChanged = 0; /* True if a new wal-index header is loaded */ int eMode2 = eMode; /* Mode to pass to walCheckpoint() */ assert( pWal->ckptLock==0 ); assert( pWal->writeLock==0 ); WALTRACE(("WAL%p: checkpoint begins\n", pWal)); rc = walLockExclusive(pWal, WAL_CKPT_LOCK, 1); if( rc ){ /* Usually this is SQLITE_BUSY meaning that another thread or process ** is already running a checkpoint, or maybe a recovery. But it might ** also be SQLITE_IOERR. */ return rc; | > | 2791 2792 2793 2794 2795 2796 2797 2798 2799 2800 2801 2802 2803 2804 2805 | int rc; /* Return code */ int isChanged = 0; /* True if a new wal-index header is loaded */ int eMode2 = eMode; /* Mode to pass to walCheckpoint() */ assert( pWal->ckptLock==0 ); assert( pWal->writeLock==0 ); if( pWal->readOnly ) return SQLITE_READONLY; WALTRACE(("WAL%p: checkpoint begins\n", pWal)); rc = walLockExclusive(pWal, WAL_CKPT_LOCK, 1); if( rc ){ /* Usually this is SQLITE_BUSY meaning that another thread or process ** is already running a checkpoint, or maybe a recovery. But it might ** also be SQLITE_IOERR. */ return rc; |
︙ | ︙ |
Changes to test/e_uri.test.
︙ | ︙ | |||
44 45 46 47 48 49 50 | set e } # EVIDENCE-OF: R-35840-33204 If URI filename interpretation is enabled, # and the filename argument begins with "file:", then the filename is # interpreted as a URI. # | | | > | 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 | set e } # EVIDENCE-OF: R-35840-33204 If URI filename interpretation is enabled, # and the filename argument begins with "file:", then the filename is # interpreted as a URI. # # EVIDENCE-OF: R-32637-34037 URI filename interpretation is enabled if # the SQLITE_OPEN_URI flag is is set in the fourth argument to # sqlite3_open_v2(), or if it has been enabled globally using the # SQLITE_CONFIG_URI option with the sqlite3_config() method or by the # SQLITE_USE_URI compile-time option. # if {$tcl_platform(platform) == "unix"} { set flags [list SQLITE_OPEN_READWRITE SQLITE_OPEN_CREATE] # Tests with SQLITE_CONFIG_URI configured to false. URI intepretation is # only enabled if the SQLITE_OPEN_URI flag is specified. sqlite3_shutdown |
︙ | ︙ | |||
136 137 138 139 140 141 142 | set e [sqlite3_errmsg $DB] sqlite3_close $DB set e } $error } } | | | | | | > | | 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 | set e [sqlite3_errmsg $DB] sqlite3_close $DB set e } $error } } # EVIDENCE-OF: R-45981-25528 The fragment component of a URI, if # present, is ignored. # # It is difficult to test that something is ignore correctly. So these tests # just show that adding a fragment does not interfere with the pathname or # parameters passed through to the VFS xOpen() methods. # if {$tcl_platform(platform) == "unix"} { foreach {tn uri parse} " 1 {file:test.db#abc} {[pwd]/test.db {}} 2 {file:test.db?a=b#abc} {[pwd]/test.db {a b}} 3 {file:test.db?a=b#?c=d} {[pwd]/test.db {a b}} " { do_test 3.$tn { parse_uri $uri } $parse } } # EVIDENCE-OF: R-62557-09390 SQLite uses the path component of the URI # as the name of the disk file which contains the database. # # EVIDENCE-OF: R-28659-11035 If the path begins with a '/' character, # then it is interpreted as an absolute path. # # EVIDENCE-OF: R-46234-61323 If the path does not begin with a '/' # (meaning that the authority section is omitted from the URI) then the # path is interpreted as a relative path. # if {$tcl_platform(platform) == "unix"} { foreach {tn uri parse} " 1 {file:test.db} {[pwd]/test.db {}} 2 {file:/test.db} {/test.db {}} 3 {file:///test.db} {/test.db {}} 4 {file://localhost/test.db} {/test.db {}} |
︙ | ︙ |
Changes to test/fkey3.test.
︙ | ︙ | |||
16 17 18 19 20 21 22 23 24 25 26 27 28 29 | set testdir [file dirname $argv0] source $testdir/tester.tcl ifcapable {!foreignkey||!trigger} { finish_test return } # Create a table and some data to work with. # do_test fkey3-1.1 { execsql { PRAGMA foreign_keys=ON; CREATE TABLE t1(x INTEGER PRIMARY KEY); | > > | 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 | set testdir [file dirname $argv0] source $testdir/tester.tcl ifcapable {!foreignkey||!trigger} { finish_test return } set testprefix fkey3 # Create a table and some data to work with. # do_test fkey3-1.1 { execsql { PRAGMA foreign_keys=ON; CREATE TABLE t1(x INTEGER PRIMARY KEY); |
︙ | ︙ | |||
73 74 75 76 77 78 79 80 | INSERT INTO t2 VALUES(100); INSERT INTO t2 VALUES(101); SELECT 1, x FROM t1; SELECT 2, y FROM t2; } } {1 100 1 101 2 100 2 101} finish_test | > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > | 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 | INSERT INTO t2 VALUES(100); INSERT INTO t2 VALUES(101); SELECT 1, x FROM t1; SELECT 2, y FROM t2; } } {1 100 1 101 2 100 2 101} #------------------------------------------------------------------------- # The following tests - fkey-3.* - test some edge cases to do with # inserting rows into tables that have foreign keys where the parent # table is the same as the child table. Especially cases where the # new row being inserted matches itself. # do_execsql_test 3.1.1 { CREATE TABLE t3(a, b, c, d, UNIQUE(a, b), FOREIGN KEY(c, d) REFERENCES t3(a, b) ); INSERT INTO t3 VALUES(1, 2, 1, 2); } {} do_catchsql_test 3.1.2 { INSERT INTO t3 VALUES(NULL, 2, 5, 2); } {1 {foreign key constraint failed}} do_catchsql_test 3.1.3 { INSERT INTO t3 VALUES(NULL, 3, 5, 2); } {1 {foreign key constraint failed}} do_execsql_test 3.2.1 { CREATE TABLE t4(a UNIQUE, b REFERENCES t4(a)); } do_catchsql_test 3.2.2 { INSERT INTO t4 VALUES(NULL, 1); } {1 {foreign key constraint failed}} do_execsql_test 3.3.1 { CREATE TABLE t5(a INTEGER PRIMARY KEY, b REFERENCES t5(a)); INSERT INTO t5 VALUES(NULL, 1); } {} do_catchsql_test 3.3.2 { INSERT INTO t5 VALUES(NULL, 3); } {1 {foreign key constraint failed}} do_execsql_test 3.4.1 { CREATE TABLE t6(a INTEGER PRIMARY KEY, b, c, d, FOREIGN KEY(c, d) REFERENCES t6(a, b) ); CREATE UNIQUE INDEX t6i ON t6(b, a); } do_execsql_test 3.4.2 { INSERT INTO t6 VALUES(NULL, 'a', 1, 'a'); } {} do_execsql_test 3.4.3 { INSERT INTO t6 VALUES(2, 'a', 2, 'a'); } {} do_execsql_test 3.4.4 { INSERT INTO t6 VALUES(NULL, 'a', 1, 'a'); } {} do_execsql_test 3.4.5 { INSERT INTO t6 VALUES(5, 'a', 2, 'a'); } {} do_catchsql_test 3.4.6 { INSERT INTO t6 VALUES(NULL, 'a', 65, 'a'); } {1 {foreign key constraint failed}} do_execsql_test 3.4.7 { INSERT INTO t6 VALUES(100, 'one', 100, 'one'); DELETE FROM t6 WHERE a = 100; } do_execsql_test 3.4.8 { INSERT INTO t6 VALUES(100, 'one', 100, 'one'); UPDATE t6 SET c = 1, d = 'a' WHERE a = 100; DELETE FROM t6 WHERE a = 100; } do_execsql_test 3.5.1 { CREATE TABLE t7(a, b, c, d INTEGER PRIMARY KEY, FOREIGN KEY(c, d) REFERENCES t7(a, b) ); CREATE UNIQUE INDEX t7i ON t7(a, b); } do_execsql_test 3.5.2 { INSERT INTO t7 VALUES('x', 1, 'x', NULL) } {} do_execsql_test 3.5.3 { INSERT INTO t7 VALUES('x', 2, 'x', 2) } {} do_catchsql_test 3.5.4 { INSERT INTO t7 VALUES('x', 450, 'x', NULL); } {1 {foreign key constraint failed}} do_catchsql_test 3.5.5 { INSERT INTO t7 VALUES('x', 450, 'x', 451); } {1 {foreign key constraint failed}} do_execsql_test 3.6.1 { CREATE TABLE t8(a, b, c, d, e, FOREIGN KEY(c, d) REFERENCES t8(a, b)); CREATE UNIQUE INDEX t8i1 ON t8(a, b); CREATE UNIQUE INDEX t8i2 ON t8(c); INSERT INTO t8 VALUES(1, 1, 1, 1, 1); } do_catchsql_test 3.6.2 { UPDATE t8 SET d = 2; } {1 {foreign key constraint failed}} do_execsql_test 3.6.3 { UPDATE t8 SET d = 1; } do_execsql_test 3.6.4 { UPDATE t8 SET e = 2; } do_catchsql_test 3.6.5 { CREATE TABLE TestTable ( id INTEGER PRIMARY KEY, name text, source_id integer not null, parent_id integer, foreign key(source_id, parent_id) references TestTable(source_id, id) ); CREATE UNIQUE INDEX testindex on TestTable(source_id, id); PRAGMA foreign_keys=1; INSERT INTO TestTable VALUES (1, 'parent', 1, null); INSERT INTO TestTable VALUES (2, 'child', 1, 1); UPDATE TestTable SET parent_id=1000 where id=2; } {1 {foreign key constraint failed}} finish_test |
Added test/fts3auto.test.
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 | # 2011 June 10 # # May you do good and not evil. # May you find forgiveness for yourself and forgive others. # May you share freely, never taking more than you give. # #*********************************************************************** # set testdir [file dirname $argv0] source $testdir/tester.tcl # If this build does not include FTS3, skip the tests in this file. # ifcapable !fts3 { finish_test ; return } source $testdir/fts3_common.tcl source $testdir/malloc_common.tcl set testprefix fts3auto set sfep $sqlite_fts3_enable_parentheses set sqlite_fts3_enable_parentheses 1 #-------------------------------------------------------------------------- # Start of Tcl infrastructure used by tests. The entry points are: # # do_fts3query_test # fts3_make_deferrable # fts3_zero_long_segments # # # do_fts3query_test TESTNAME ?OPTIONS? TABLE MATCHEXPR # # This proc runs several test cases on FTS3/4 table $TABLE using match # expression $MATCHEXPR. All documents in $TABLE must be formatted so that # they can be "tokenized" using the Tcl list commands (llength, lindex etc.). # The name and column names used by $TABLE must not require any quoting or # escaping when used in SQL statements. # # $MATCHINFO may be any expression accepted by the FTS4 MATCH operator, # except that the "<column-name>:token" syntax is not supported. Tcl list # commands are used to tokenize the expression. Any parenthesis must appear # either as separate list elements, or as the first (for opening) or last # (for closing) character of a list element. i.e. the expression "(a OR b)c" # will not be parsed correctly, but "( a OR b) c" will. # # Available OPTIONS are: # # -deferred TOKENLIST # # If the "deferred" option is supplied, it is passed a list of tokens that # are deferred by FTS and result in the relevant matchinfo() stats being an # approximation. # set sqlite_fts3_enable_parentheses 1 proc do_fts3query_test {tn args} { set nArg [llength $args] if {$nArg < 2 || ($nArg % 2)} { set cmd do_fts3query_test error "wrong # args: should be \"$cmd ?-deferred LIST? TABLE MATCHEXPR\"" } set tbl [lindex $args [expr $nArg-2]] set match [lindex $args [expr $nArg-1]] set deferred [list] foreach {k v} [lrange $args 0 [expr $nArg-3]] { switch -- $k { -deferred { set deferred $v } default { error "bad option \"$k\": must be -deferred" } } } get_near_results $tbl $match $deferred aMatchinfo set matchinfo_asc [list] foreach docid [lsort -integer -incr [array names aMatchinfo]] { lappend matchinfo_asc $docid $aMatchinfo($docid) } set matchinfo_desc [list] foreach docid [lsort -integer -decr [array names aMatchinfo]] { lappend matchinfo_desc $docid $aMatchinfo($docid) } set title "(\"$match\" -> [llength [array names aMatchinfo]] rows)" do_execsql_test $tn$title.1 " SELECT docid FROM $tbl WHERE $tbl MATCH '$match' ORDER BY docid ASC " [lsort -integer -incr [array names aMatchinfo]] do_execsql_test $tn$title.2 " SELECT docid FROM $tbl WHERE $tbl MATCH '$match' ORDER BY docid DESC " [lsort -integer -decr [array names aMatchinfo]] do_execsql_test $tn$title.3 " SELECT docid, mit(matchinfo($tbl, 'x')) FROM $tbl WHERE $tbl MATCH '$match' ORDER BY docid DESC " $matchinfo_desc do_execsql_test $tn$title.4 " SELECT docid, mit(matchinfo($tbl, 'x')) FROM $tbl WHERE $tbl MATCH '$match' ORDER BY docid ASC " $matchinfo_asc } # fts3_make_deferrable TABLE TOKEN # proc fts3_make_deferrable {tbl token} { set stmt [sqlite3_prepare db "SELECT * FROM $tbl" -1 dummy] set name [sqlite3_column_name $stmt 0] sqlite3_finalize $stmt set nRow [db one "SELECT count(*) FROM $tbl"] set pgsz [db one "PRAGMA page_size"] execsql BEGIN for {set i 0} {$i < ($nRow * $pgsz * 1.2)/100} {incr i} { set doc [string repeat "$token " 100] execsql "INSERT INTO $tbl ($name) VALUES(\$doc)" } execsql "INSERT INTO $tbl ($name) VALUES('aaaaaaa ${token}aaaaa')" execsql COMMIT return [expr $nRow*$pgsz] } # fts3_zero_long_segments TABLE ?LIMIT? # proc fts3_zero_long_segments {tbl limit} { execsql " UPDATE ${tbl}_segments SET block = zeroblob(length(block)) WHERE length(block)>$limit " return [db changes] } proc mit {blob} { set scan(littleEndian) i* set scan(bigEndian) I* binary scan $blob $scan($::tcl_platform(byteOrder)) r return $r } db func mit mit proc fix_near_expr {expr} { set out [list] lappend out [lindex $expr 0] foreach {a b} [lrange $expr 1 end] { if {[string match -nocase near $a]} { set a 10 } if {[string match -nocase near/* $a]} { set a [string range $a 5 end] } lappend out $a lappend out $b } return $out } proc get_single_near_results {tbl expr deferred arrayvar nullvar} { upvar $arrayvar aMatchinfo upvar $nullvar nullentry catch {array unset aMatchinfo} set expr [fix_near_expr $expr] # Calculate the expected results using [fts3_near_match]. The following # loop populates the "hits" and "counts" arrays as follows: # # 1. For each document in the table that matches the NEAR expression, # hits($docid) is set to 1. The set of docids that match the expression # can therefore be found using [array names hits]. # # 2. For each column of each document in the table, counts($docid,$iCol) # is set to the -phrasecountvar output. # set res [list] catch { array unset hits } db eval "SELECT docid, * FROM $tbl" d { set iCol 0 foreach col [lrange $d(*) 1 end] { set docid $d(docid) set hit [fts3_near_match $d($col) $expr -p counts($docid,$iCol)] if {$hit} { set hits($docid) 1 } incr iCol } } set nPhrase [expr ([llength $expr]+1)/2] set nCol $iCol # This block populates the nHit and nDoc arrays. For each phrase/column # in the query/table, array elements are set as follows: # # nHit($iPhrase,$iCol) - Total number of hits for phrase $iPhrase in # column $iCol. # # nDoc($iPhrase,$iCol) - Number of documents with at least one hit for # phrase $iPhrase in column $iCol. # for {set iPhrase 0} {$iPhrase < $nPhrase} {incr iPhrase} { for {set iCol 0} {$iCol < $nCol} {incr iCol} { set nHit($iPhrase,$iCol) 0 set nDoc($iPhrase,$iCol) 0 } } foreach key [array names counts] { set iCol [lindex [split $key ,] 1] set iPhrase 0 foreach c $counts($key) { if {$c>0} { incr nDoc($iPhrase,$iCol) 1 } incr nHit($iPhrase,$iCol) $c incr iPhrase } } if {[llength $deferred] && [llength $expr]==1} { set phrase [lindex $expr 0] set rewritten [list] set partial 0 foreach tok $phrase { if {[lsearch $deferred $tok]>=0} { lappend rewritten * } else { lappend rewritten $tok set partial 1 } } if {$partial==0} { set tblsize [db one "SELECT count(*) FROM $tbl"] for {set iCol 0} {$iCol < $nCol} {incr iCol} { set nHit(0,$iCol) $tblsize set nDoc(0,$iCol) $tblsize } } elseif {$rewritten != $phrase} { while {[lindex $rewritten end] == "*"} { set rewritten [lrange $rewritten 0 end-1] } while {[lindex $rewritten 0] == "*"} { set rewritten [lrange $rewritten 1 end] } get_single_near_results $tbl [list $rewritten] {} aRewrite nullentry foreach docid [array names hits] { set aMatchinfo($docid) $aRewrite($docid) } return } } # Set up the aMatchinfo array. For each document, set aMatchinfo($docid) to # contain the output of matchinfo('x') for the document. # foreach docid [array names hits] { set mi [list] for {set iPhrase 0} {$iPhrase<$nPhrase} {incr iPhrase} { for {set iCol 0} {$iCol<$nCol} {incr iCol} { lappend mi [lindex $counts($docid,$iCol) $iPhrase] lappend mi $nHit($iPhrase,$iCol) lappend mi $nDoc($iPhrase,$iCol) } } set aMatchinfo($docid) $mi } # Set up the nullentry output. # set nullentry [list] for {set iPhrase 0} {$iPhrase<$nPhrase} {incr iPhrase} { for {set iCol 0} {$iCol<$nCol} {incr iCol} { lappend nullentry 0 $nHit($iPhrase,$iCol) $nDoc($iPhrase,$iCol) } } } proc matching_brackets {expr} { if {[string range $expr 0 0]!="(" || [string range $expr end end] !=")"} { return 0 } set iBracket 1 set nExpr [string length $expr] for {set i 1} {$iBracket && $i < $nExpr} {incr i} { set c [string range $expr $i $i] if {$c == "("} {incr iBracket} if {$c == ")"} {incr iBracket -1} } return [expr ($iBracket==0 && $i==$nExpr)] } proc get_near_results {tbl expr deferred arrayvar {nullvar ""}} { upvar $arrayvar aMatchinfo if {$nullvar != ""} { upvar $nullvar nullentry } set expr [string trim $expr] while { [matching_brackets $expr] } { set expr [string trim [string range $expr 1 end-1]] } set prec(NOT) 1 set prec(AND) 2 set prec(OR) 3 set currentprec 0 set iBracket 0 set expr_length [llength $expr] for {set i 0} {$i < $expr_length} {incr i} { set op [lindex $expr $i] if {$iBracket==0 && [info exists prec($op)] && $prec($op)>=$currentprec } { set opidx $i set currentprec $prec($op) } else { for {set j 0} {$j < [string length $op]} {incr j} { set c [string range $op $j $j] if {$c == "("} { incr iBracket +1 } if {$c == ")"} { incr iBracket -1 } } } } if {$iBracket!=0} { error "mismatched brackets in: $expr" } if {[info exists opidx]==0} { get_single_near_results $tbl $expr $deferred aMatchinfo nullentry } else { set eLeft [lrange $expr 0 [expr $opidx-1]] set eRight [lrange $expr [expr $opidx+1] end] get_near_results $tbl $eLeft $deferred aLeft nullleft get_near_results $tbl $eRight $deferred aRight nullright switch -- [lindex $expr $opidx] { "NOT" { foreach hit [array names aLeft] { if {0==[info exists aRight($hit)]} { set aMatchinfo($hit) $aLeft($hit) } } set nullentry $nullleft } "AND" { foreach hit [array names aLeft] { if {[info exists aRight($hit)]} { set aMatchinfo($hit) [concat $aLeft($hit) $aRight($hit)] } } set nullentry [concat $nullleft $nullright] } "OR" { foreach hit [array names aLeft] { if {[info exists aRight($hit)]} { set aMatchinfo($hit) [concat $aLeft($hit) $aRight($hit)] unset aRight($hit) } else { set aMatchinfo($hit) [concat $aLeft($hit) $nullright] } } foreach hit [array names aRight] { set aMatchinfo($hit) [concat $nullleft $aRight($hit)] } set nullentry [concat $nullleft $nullright] } } } } # End of test procs. Actual tests are below this line. #-------------------------------------------------------------------------- #-------------------------------------------------------------------------- # The following test cases - fts3auto-1.* - focus on testing the Tcl # command [fts3_near_match], which is used by other tests in this file. # proc test_fts3_near_match {tn doc expr res} { fts3_near_match $doc $expr -phrasecountvar p uplevel do_test [list $tn] [list [list set {} $p]] [list $res] } test_fts3_near_match 1.1.1 {a b c a b} a {2} test_fts3_near_match 1.1.2 {a b c a b} {a 5 b 6 c} {2 2 1} test_fts3_near_match 1.1.3 {a b c a b} {"a b"} {2} test_fts3_near_match 1.1.4 {a b c a b} {"b c"} {1} test_fts3_near_match 1.1.5 {a b c a b} {"c c"} {0} test_fts3_near_match 1.2.1 "a b c d e f g" {b 2 f} {0 0} test_fts3_near_match 1.2.2 "a b c d e f g" {b 3 f} {1 1} test_fts3_near_match 1.2.3 "a b c d e f g" {f 2 b} {0 0} test_fts3_near_match 1.2.4 "a b c d e f g" {f 3 b} {1 1} test_fts3_near_match 1.2.5 "a b c d e f g" {"a b" 2 "f g"} {0 0} test_fts3_near_match 1.2.6 "a b c d e f g" {"a b" 3 "f g"} {1 1} set A "a b c d e f g h i j k l m n o p q r s t u v w x y z" test_fts3_near_match 1.3.1 $A {"c d" 5 "i j" 1 "e f"} {0 0 0} test_fts3_near_match 1.3.2 $A {"c d" 5 "i j" 2 "e f"} {1 1 1} #-------------------------------------------------------------------------- # Test cases fts3auto-2.* run some simple tests using the # [do_fts3query_test] proc. # foreach {tn create} { 1 "fts4(a, b)" 2 "fts4(a, b, order=DESC)" 3 "fts4(a, b, order=ASC)" 4 "fts4(a, b, prefix=1)" 5 "fts4(a, b, order=DESC, prefix=1)" 6 "fts4(a, b, order=ASC, prefix=1)" } { do_test 2.$tn.1 { catchsql { DROP TABLE t1 } execsql "CREATE VIRTUAL TABLE t1 USING $create" for {set i 0} {$i<32} {incr i} { set doc [list] if {$i&0x01} {lappend doc one} if {$i&0x02} {lappend doc two} if {$i&0x04} {lappend doc three} if {$i&0x08} {lappend doc four} if {$i&0x10} {lappend doc five} execsql { INSERT INTO t1 VALUES($doc, null) } } } {} foreach {tn2 expr} { 1 {one} 2 {one NEAR/1 five} 3 {t*} 4 {t* NEAR/0 five} 5 {o* NEAR/1 f*} 6 {one NEAR five NEAR two NEAR four NEAR three} 7 {one NEAR xyz} 8 {one OR two} 9 {one AND two} 10 {one NOT two} 11 {one AND two OR three} 12 {three OR one AND two} 13 {(three OR one) AND two} 14 {(three OR one) AND two NOT (five NOT four)} 15 {"one two"} 16 {"one two" NOT "three four"} } { do_fts3query_test 2.$tn.2.$tn2 t1 $expr } } #-------------------------------------------------------------------------- # Some test cases involving deferred tokens. # foreach {tn create} { 1 "fts4(x)" 2 "fts4(x, order=DESC)" } { catchsql { DROP TABLE t1 } execsql "CREATE VIRTUAL TABLE t1 USING $create" do_execsql_test 3.$tn.1 { INSERT INTO t1(docid, x) VALUES(-2, 'a b c d e f g h i j k'); INSERT INTO t1(docid, x) VALUES(-1, 'b c d e f g h i j k a'); INSERT INTO t1(docid, x) VALUES(0, 'c d e f g h i j k a b'); INSERT INTO t1(docid, x) VALUES(1, 'd e f g h i j k a b c'); INSERT INTO t1(docid, x) VALUES(2, 'e f g h i j k a b c d'); INSERT INTO t1(docid, x) VALUES(3, 'f g h i j k a b c d e'); INSERT INTO t1(docid, x) VALUES(4, 'a c e g i k'); INSERT INTO t1(docid, x) VALUES(5, 'a d g j'); INSERT INTO t1(docid, x) VALUES(6, 'c a b'); } set limit [fts3_make_deferrable t1 c] do_fts3query_test 3.$tn.2.1 t1 {a OR c} do_test 3.$tn.3 { fts3_zero_long_segments t1 $limit } {1} foreach {tn2 expr def} { 1 {a NEAR c} {} 2 {a AND c} c 3 {"a c"} c 4 {"c a"} c 5 {"a c" NEAR/1 g} {} 6 {"a c" NEAR/0 g} {} } { do_fts3query_test 3.$tn.4.$tn2 -deferred $def t1 $expr } } #-------------------------------------------------------------------------- # foreach {tn create} { 1 "fts4(x, y)" 2 "fts4(x, y, order=DESC)" 3 "fts4(x, y, order=DESC, prefix=2)" } { execsql [subst { DROP TABLE t1; CREATE VIRTUAL TABLE t1 USING $create; INSERT INTO t1 VALUES('one two five four five', ''); INSERT INTO t1 VALUES('', 'one two five four five'); INSERT INTO t1 VALUES('one two', 'five four five'); }] do_fts3query_test 4.$tn.1.1 t1 {one AND five} do_fts3query_test 4.$tn.1.2 t1 {one NEAR five} do_fts3query_test 4.$tn.1.3 t1 {one NEAR/1 five} do_fts3query_test 4.$tn.1.4 t1 {one NEAR/2 five} do_fts3query_test 4.$tn.1.5 t1 {one NEAR/3 five} do_test 4.$tn.2 { set limit [fts3_make_deferrable t1 five] execsql { INSERT INTO t1(t1) VALUES('optimize') } expr {[fts3_zero_long_segments t1 $limit]>0} } {1} do_fts3query_test 4.$tn.3.1 -deferred five t1 {one AND five} do_fts3query_test 4.$tn.3.2 -deferred five t1 {one NEAR five} do_fts3query_test 4.$tn.3.3 -deferred five t1 {one NEAR/1 five} do_fts3query_test 4.$tn.3.4 -deferred five t1 {one NEAR/2 five} do_fts3query_test 4.$tn.3.5 -deferred five t1 {one NEAR/3 five} do_fts3query_test 4.$tn.4.1 -deferred fi* t1 {on* AND fi*} do_fts3query_test 4.$tn.4.2 -deferred fi* t1 {on* NEAR fi*} do_fts3query_test 4.$tn.4.3 -deferred fi* t1 {on* NEAR/1 fi*} do_fts3query_test 4.$tn.4.4 -deferred fi* t1 {on* NEAR/2 fi*} do_fts3query_test 4.$tn.4.5 -deferred fi* t1 {on* NEAR/3 fi*} } #-------------------------------------------------------------------------- # The following test cases - fts3auto-5.* - focus on using prefix indexes. # set chunkconfig [fts3_configure_incr_load 1 1] foreach {tn create pending} { 2 "fts4(a, b, order=ASC, prefix=1)" 1 1 "fts4(a, b)" 1 3 "fts4(a, b, order=ASC, prefix=1,3)" 0 4 "fts4(a, b, order=DESC, prefix=2,4)" 0 5 "fts4(a, b, order=DESC, prefix=1)" 0 6 "fts4(a, b, order=ASC, prefix=1,3)" 0 } { execsql [subst { DROP TABLE IF EXISTS t1; CREATE VIRTUAL TABLE t1 USING $create; }] if {$pending} {execsql BEGIN} foreach {a b} { "the song of songs which is solomons" "let him kiss me with the kisses of his mouth for thy love is better than wine" "because of the savour of thy good ointments thy name is as ointment poured forth therefore do the virgins love thee" "draw me we will run after thee the king hath brought me into his chambers we will be glad and rejoice in thee we will remember thy love more than wine the upright love thee" "i am black but comely o ye daughters of jerusalem as the tents of kedar as the curtains of solomon" "look not upon me because i am black because the sun hath looked upon me my mothers children were angry with me they made me the keeper of the vineyards but mine own vineyard have i not kept" "tell me o thou whom my soul loveth where thou feedest where thou makest thy flock to rest at noon for why should i be as one that turneth aside by the flocks of thy companions?" "if thou know not o thou fairest among women go thy way forth by the footsteps of the flock and feed thy kids beside the shepherds tents" "i have compared thee o my love to a company of horses in pharaohs chariots" "thy cheeks are comely with rows of jewels thy neck with chains of gold" "we will make thee borders of gold with studs of silver" "while the king sitteth at his table my spikenard sendeth forth the smell thereof" "a bundle of myrrh is my wellbeloved unto me he shall lie all night betwixt my breasts" "my beloved is unto me as a cluster of camphire in the vineyards of en gedi" "behold thou art fair my love behold thou art fair thou hast doves eyes" "behold thou art fair my beloved yea pleasant also our bed is green" "the beams of our house are cedar and our rafters of fir" } { execsql {INSERT INTO t1(a, b) VALUES($a, $b)} } do_fts3query_test 5.$tn.1.1 t1 {s*} do_fts3query_test 5.$tn.1.2 t1 {so*} do_fts3query_test 5.$tn.1.3 t1 {"s* o*"} do_fts3query_test 5.$tn.1.4 t1 {b* NEAR/3 a*} do_fts3query_test 5.$tn.1.5 t1 {a*} do_fts3query_test 5.$tn.1.6 t1 {th* NEAR/5 a* NEAR/5 w*} do_fts3query_test 5.$tn.1.7 t1 {"b* th* art* fair*"} if {$pending} {execsql COMMIT} } eval fts3_configure_incr_load $chunkconfig set sqlite_fts3_enable_parentheses $sfep finish_test |
Changes to test/fts3defer.test.
︙ | ︙ | |||
16 17 18 19 20 21 22 23 24 25 26 27 28 29 | ifcapable !fts3 { finish_test return } set sqlite_fts3_enable_parentheses 1 set ::testprefix fts3defer #-------------------------------------------------------------------------- # Test cases fts3defer-1.* are the "warm body" cases. The database contains # one row with 15000 instances of the token "a". This makes the doclist for # "a" so large that FTS3 will avoid loading it in most cases. # | > > | 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 | ifcapable !fts3 { finish_test return } set sqlite_fts3_enable_parentheses 1 set fts3_simple_deferred_tokens_only 1 set ::testprefix fts3defer #-------------------------------------------------------------------------- # Test cases fts3defer-1.* are the "warm body" cases. The database contains # one row with 15000 instances of the token "a". This makes the doclist for # "a" so large that FTS3 will avoid loading it in most cases. # |
︙ | ︙ | |||
253 254 255 256 257 258 259 | do_select_test 1.1 { SELECT rowid FROM t1 WHERE t1 MATCH 'jk xnxhf' } {13 29 40 47 48 52 63 92} do_select_test 1.2 { SELECT rowid FROM t1 WHERE t1 MATCH 'jk eh' } {100} | < | 255 256 257 258 259 260 261 262 263 264 265 266 267 268 | do_select_test 1.1 { SELECT rowid FROM t1 WHERE t1 MATCH 'jk xnxhf' } {13 29 40 47 48 52 63 92} do_select_test 1.2 { SELECT rowid FROM t1 WHERE t1 MATCH 'jk eh' } {100} do_select_test 1.3 { SELECT rowid FROM t1 WHERE t1 MATCH 'jk ubwrfqnbjf' } {7 70 98} do_select_test 1.4 { SELECT rowid FROM t1 WHERE t1 MATCH 'duszemmzl jk' } {3 5 8 10 13 18 20 23 32 37 41 43 55 60 65 67 72 74 76 81 94 96 97} do_select_test 1.5 { |
︙ | ︙ | |||
278 279 280 281 282 283 284 | } {68 100} do_select_test 1.9 { SELECT rowid FROM t1 WHERE t1 MATCH 'zm ubwrfqnbjf' } {7 70 98} do_select_test 1.10 { SELECT rowid FROM t1 WHERE t1 MATCH 'z* vgsld' } {10 13 17 31 35 51 58 88 89 90 93 100} | > > | | | | | | | > | 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 | } {68 100} do_select_test 1.9 { SELECT rowid FROM t1 WHERE t1 MATCH 'zm ubwrfqnbjf' } {7 70 98} do_select_test 1.10 { SELECT rowid FROM t1 WHERE t1 MATCH 'z* vgsld' } {10 13 17 31 35 51 58 88 89 90 93 100} if { $fts3_simple_deferred_tokens_only==0 } { do_select_test 1.11 { SELECT rowid FROM t1 WHERE t1 MATCH '( zdu OR zexh OR zf OR zhbrzadb OR zidhxhbtv OR zk OR zkhdvkw OR zm OR zsmhnf ) vgsld' } {10 13 17 31 35 51 58 88 89 90 93 100} } do_select_test 2.1 { SELECT rowid FROM t1 WHERE t1 MATCH '"zm agmckuiu"' } {3 24 52 53} do_select_test 2.2 { SELECT rowid FROM t1 WHERE t1 MATCH '"zm zf"' } {33 53 75 88 101} |
︙ | ︙ | |||
360 361 362 363 364 365 366 367 368 369 370 371 372 373 | # The following block of tests runs normally with FTS3 or FTS4 without the # long doclists zeroed. And with OOM-injection for FTS4 with long doclists # zeroed. Change this by messing with the [set dmt_modes] commands above. # foreach DO_MALLOC_TEST $dmt_modes { # Phrase search. do_select_test 5.$DO_MALLOC_TEST.1 { SELECT rowid FROM t1 WHERE t1 MATCH '"jk mjpavjuhw"' } {8 15 36 64 67 72} # Multiple tokens search. do_select_test 5.$DO_MALLOC_TEST.2 { SELECT rowid FROM t1 WHERE t1 MATCH 'duszemmzl zm' | > | 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 | # The following block of tests runs normally with FTS3 or FTS4 without the # long doclists zeroed. And with OOM-injection for FTS4 with long doclists # zeroed. Change this by messing with the [set dmt_modes] commands above. # foreach DO_MALLOC_TEST $dmt_modes { # Phrase search. # do_select_test 5.$DO_MALLOC_TEST.1 { SELECT rowid FROM t1 WHERE t1 MATCH '"jk mjpavjuhw"' } {8 15 36 64 67 72} # Multiple tokens search. do_select_test 5.$DO_MALLOC_TEST.2 { SELECT rowid FROM t1 WHERE t1 MATCH 'duszemmzl zm' |
︙ | ︙ | |||
412 413 414 415 416 417 418 | } {10} do_select_test 6.2.1 { SELECT rowid FROM t1 WHERE t1 MATCH '"jk xduvfhk"' } {8} do_select_test 6.2.2 { SELECT rowid FROM t1 WHERE t1 MATCH '"zm azavwm"' } {15 26 92 96} | > | | | > | 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 | } {10} do_select_test 6.2.1 { SELECT rowid FROM t1 WHERE t1 MATCH '"jk xduvfhk"' } {8} do_select_test 6.2.2 { SELECT rowid FROM t1 WHERE t1 MATCH '"zm azavwm"' } {15 26 92 96} if {$fts3_simple_deferred_tokens_only==0} { do_select_test 6.2.3 { SELECT rowid FROM t1 WHERE t1 MATCH '"jk xduvfhk" OR "zm azavwm"' } {8 15 26 92 96} } } set testprefix fts3defer do_execsql_test 3.1 { CREATE VIRTUAL TABLE x1 USING fts4(a, b); INSERT INTO x1 VALUES('a b c', 'd e f'); |
︙ | ︙ |
Changes to test/fts3defer2.test.
︙ | ︙ | |||
44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 | INSERT INTO t1(t1) VALUES('optimize'); } do_execsql_test 1.1.4 { SELECT count(*) FROM t1_segments WHERE length(block)>10000; UPDATE t1_segments SET block = zeroblob(length(block)) WHERE length(block)>10000; } {2} do_execsql_test 1.2.1 { SELECT content FROM t1 WHERE t1 MATCH 'f (e NEAR/2 a)'; } {{a b c d e f a x y}} do_execsql_test 1.2.2 { SELECT snippet(t1, '[', ']'), offsets(t1), mit(matchinfo(t1, 'pcxnal')) FROM t1 WHERE t1 MATCH 'f (e NEAR/2 a)'; } [list \ {a b c d [e] [f] [a] x y} \ {0 1 8 1 0 0 10 1 0 2 12 1} \ | > > > > | | | 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 | INSERT INTO t1(t1) VALUES('optimize'); } do_execsql_test 1.1.4 { SELECT count(*) FROM t1_segments WHERE length(block)>10000; UPDATE t1_segments SET block = zeroblob(length(block)) WHERE length(block)>10000; } {2} do_execsql_test 1.2.0 { SELECT content FROM t1 WHERE t1 MATCH 'f (e a)'; } {{a b c d e f a x y}} do_execsql_test 1.2.1 { SELECT content FROM t1 WHERE t1 MATCH 'f (e NEAR/2 a)'; } {{a b c d e f a x y}} do_execsql_test 1.2.2 { SELECT snippet(t1, '[', ']'), offsets(t1), mit(matchinfo(t1, 'pcxnal')) FROM t1 WHERE t1 MATCH 'f (e NEAR/2 a)'; } [list \ {a b c d [e] [f] [a] x y} \ {0 1 8 1 0 0 10 1 0 2 12 1} \ [list 3 1 1 1 1 1 1 1 1 1 1 8 5001 9] ] do_execsql_test 1.2.3 { SELECT snippet(t1, '[', ']'), offsets(t1), mit(matchinfo(t1, 'pcxnal')) FROM t1 WHERE t1 MATCH 'f (e NEAR/3 a)'; } [list \ {[a] b c d [e] [f] [a] x y} \ {0 2 0 1 0 1 8 1 0 0 10 1 0 2 12 1} \ [list 3 1 1 1 1 1 1 1 2 2 1 8 5001 9] ] do_execsql_test 1.3.1 { DROP TABLE t1 } #----------------------------------------------------------------------------- # Test cases fts3defer2-2.* focus specifically on the matchinfo function. # |
︙ | ︙ | |||
95 96 97 98 99 100 101 102 | do_execsql_test 2.2.$tn.1 { SELECT mit(matchinfo(t2, 'pcxnal')) FROM t2 WHERE t2 MATCH 'a b'; } [list \ [list 2 1 1 54 54 1 3 3 54 372 8] \ [list 2 1 1 54 54 1 3 3 54 372 7] \ ] set sqlite_fts3_enable_parentheses 1 | > > > > > > | | 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 | do_execsql_test 2.2.$tn.1 { SELECT mit(matchinfo(t2, 'pcxnal')) FROM t2 WHERE t2 MATCH 'a b'; } [list \ [list 2 1 1 54 54 1 3 3 54 372 8] \ [list 2 1 1 54 54 1 3 3 54 372 7] \ ] do_execsql_test 2.2.$tn.2 { SELECT mit(matchinfo(t2, 'x')) FROM t2 WHERE t2 MATCH 'g z'; } [list \ [list 1 2 2 1 54 54] \ ] set sqlite_fts3_enable_parentheses 1 do_execsql_test 2.2.$tn.3 { SELECT mit(matchinfo(t2, 'x')) FROM t2 WHERE t2 MATCH 'g OR (g z)'; } [list \ [list 1 2 2 1 2 2 1 54 54] \ [list 1 2 2 1 2 2 0 54 54] \ ] set sqlite_fts3_enable_parentheses 0 } |
︙ | ︙ |
Changes to test/fts3matchinfo.test.
︙ | ︙ | |||
64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 | CREATE VIRTUAL TABLE t3 USING fts3(mtchinfo=fts3); INSERT INTO t3(mtchinfo) VALUES('Beside the lake, beneath the trees'); SELECT mtchinfo FROM t3; } {{Beside the lake, beneath the trees}} do_execsql_test 3.2 { CREATE VIRTUAL TABLE xx USING FTS4; SELECT * FROM xx WHERE xx MATCH 'abc'; SELECT * FROM xx WHERE xx MATCH 'a b c'; } #-------------------------------------------------------------------------- # Proc [do_matchinfo_test] is used to test the FTSX matchinfo() function. # | > > > > | 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 | CREATE VIRTUAL TABLE t3 USING fts3(mtchinfo=fts3); INSERT INTO t3(mtchinfo) VALUES('Beside the lake, beneath the trees'); SELECT mtchinfo FROM t3; } {{Beside the lake, beneath the trees}} do_execsql_test 3.2 { CREATE VIRTUAL TABLE xx USING FTS4; } do_execsql_test 3.3 { SELECT * FROM xx WHERE xx MATCH 'abc'; } do_execsql_test 3.4 { SELECT * FROM xx WHERE xx MATCH 'a b c'; } #-------------------------------------------------------------------------- # Proc [do_matchinfo_test] is used to test the FTSX matchinfo() function. # |
︙ | ︙ | |||
236 237 238 239 240 241 242 | do_matchinfo_test 4.2.3 t5 {t5 MATCH 'a b a'} { s {3} } do_matchinfo_test 4.2.4 t5 {t5 MATCH 'a a a'} { s {3 1} } do_matchinfo_test 4.2.5 t5 {t5 MATCH '"a b" "a b"'} { s {2} } do_matchinfo_test 4.2.6 t5 {t5 MATCH 'a OR b'} { s {1 2 1} } do_execsql_test 4.3.0 "INSERT INTO t5 VALUES('x y [string repeat {b } 50000]')"; | > > > | | | > | 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 | do_matchinfo_test 4.2.3 t5 {t5 MATCH 'a b a'} { s {3} } do_matchinfo_test 4.2.4 t5 {t5 MATCH 'a a a'} { s {3 1} } do_matchinfo_test 4.2.5 t5 {t5 MATCH '"a b" "a b"'} { s {2} } do_matchinfo_test 4.2.6 t5 {t5 MATCH 'a OR b'} { s {1 2 1} } do_execsql_test 4.3.0 "INSERT INTO t5 VALUES('x y [string repeat {b } 50000]')"; # It used to be that the second 'a' token would be deferred. That doesn't # work any longer. if 0 { do_matchinfo_test 4.3.1 t5 {t5 MATCH 'a a'} { x {{5 8 2 5 5 5} {3 8 2 3 5 5}} s {2 1} } } do_matchinfo_test 4.3.2 t5 {t5 MATCH 'a b'} { s {2} } do_matchinfo_test 4.3.3 t5 {t5 MATCH 'a b a'} { s {3} } do_matchinfo_test 4.3.4 t5 {t5 MATCH 'a a a'} { s {3 1} } do_matchinfo_test 4.3.5 t5 {t5 MATCH '"a b" "a b"'} { s {2} } do_matchinfo_test 4.3.6 t5 {t5 MATCH 'a OR b'} { s {1 2 1 1} } |
︙ | ︙ |
Added test/fts3prefix.test.
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 | # 2011 May 04 # # The author disclaims copyright to this source code. In place of # a legal notice, here is a blessing: # # May you do good and not evil. # May you find forgiveness for yourself and forgive others. # May you share freely, never taking more than you give. # #************************************************************************* # This file implements regression tests for SQLite library. The # focus of this script is testing the FTS3 module. # set testdir [file dirname $argv0] source $testdir/tester.tcl set testprefix fts3prefix ifcapable !fts3 { finish_test return } # This proc tests that the prefixes index appears to represent the same content # as the terms index. # proc fts3_terms_and_prefixes {db tbl prefixlengths} { set iIndex 0 foreach len $prefixlengths { incr iIndex $db eval { DROP TABLE IF EXISTS fts3check1; DROP TABLE IF EXISTS fts3check2; } $db eval "CREATE VIRTUAL TABLE fts3check1 USING fts4term($tbl, 0);" $db eval "CREATE VIRTUAL TABLE fts3check2 USING fts4term($tbl, $iIndex);" $db eval { DROP TABLE IF EXISTS temp.terms; DROP TABLE IF EXISTS temp.prefixes; CREATE TEMP TABLE terms AS SELECT * FROM fts3check1; CREATE TEMP TABLE prefixes AS SELECT * FROM fts3check2; CREATE INDEX temp.idx ON prefixes(term); DROP TABLE fts3check1; DROP TABLE fts3check2; } set nExpect 0 $db eval { SELECT term, docid, col, pos FROM temp.terms } a { if {[string length $a(term)]<$len} continue incr nExpect set prefix [string range $a(term) 0 [expr $len-1]] set r [$db one { SELECT count(*) FROM temp.prefixes WHERE term = $prefix AND docid = $a(docid) AND col = $a(col) AND pos = $a(pos) }] if {$r != 1} { error "$t, $a(docid), $a(col), $a(pos)" } } set nCount [$db one {SELECT count(*) FROM temp.prefixes}] if {$nCount != $nExpect} { error "prefixes.count(*) is $nCount expected $nExpect" } execsql { DROP TABLE temp.prefixes } execsql { DROP TABLE temp.terms } set list [list] $db eval " SELECT sum( 1 << (16*(level%1024)) ) AS total, (level/1024) AS tree FROM ${tbl}_segdir GROUP BY tree " { lappend list [list $total $tree] } if { [lsort -integer -index 0 $list] != [lsort -integer -index 1 $list] } { error "inconsistent tree structures: $list" } } return "" } proc fts3_tap_test {tn db tbl lens} { uplevel [list do_test $tn [list fts3_terms_and_prefixes $db $tbl $lens] ""] } #------------------------------------------------------------------------- # Test cases 1.* are a sanity check. They test that the prefixes index is # being constructed correctly for the simplest possible case. # do_execsql_test 1.1 { CREATE VIRTUAL TABLE t1 USING fts4(prefix='1,3,6'); CREATE VIRTUAL TABLE p1 USING fts4term(t1, 1); CREATE VIRTUAL TABLE p2 USING fts4term(t1, 2); CREATE VIRTUAL TABLE p3 USING fts4term(t1, 3); CREATE VIRTUAL TABLE terms USING fts4term(t1); } do_execsql_test 1.2 { INSERT INTO t1 VALUES('sqlite mysql firebird'); } do_execsql_test 1.3.1 { SELECT term FROM p1 } {f m s} do_execsql_test 1.3.2 { SELECT term FROM p2 } {fir mys sql} do_execsql_test 1.3.3 { SELECT term FROM p3 } {firebi sqlite} do_execsql_test 1.4 { SELECT term FROM terms; } {firebird mysql sqlite} fts3_tap_test 1.5 db t1 {1 3 6} #------------------------------------------------------------------------- # A slightly more complicated dataset. This test also verifies that DELETE # operations do not corrupt the prefixes index. # do_execsql_test 2.1 { INSERT INTO t1 VALUES('FTS3 and FTS4 are an SQLite virtual table modules'); INSERT INTO t1 VALUES('that allows users to perform full-text searches on'); INSERT INTO t1 VALUES('a set of documents. The most common (and'); INSERT INTO t1 VALUES('effective) way to describe full-text searches is'); INSERT INTO t1 VALUES('"what Google, Yahoo and Altavista do with'); INSERT INTO t1 VALUES('documents placed on the World Wide Web". Users'); INSERT INTO t1 VALUES('input a term, or series of terms, perhaps'); INSERT INTO t1 VALUES('connected by a binary operator or grouped together'); INSERT INTO t1 VALUES('into a phrase, and the full-text query system'); INSERT INTO t1 VALUES('finds the set of documents that best matches those'); INSERT INTO t1 VALUES('terms considering the operators and groupings the'); INSERT INTO t1 VALUES('user has specified. This article describes the'); INSERT INTO t1 VALUES('deployment and usage of FTS3 and FTS4.'); INSERT INTO t1 VALUES('FTS1 and FTS2 are obsolete full-text search'); INSERT INTO t1 VALUES('modules for SQLite. There are known issues with'); INSERT INTO t1 VALUES('these older modules and their use should be'); INSERT INTO t1 VALUES('avoided. Portions of the original FTS3 code were'); INSERT INTO t1 VALUES('contributed to the SQLite project by Scott Hess of'); INSERT INTO t1 VALUES('Google. It is now developed and maintained as part'); INSERT INTO t1 VALUES('of SQLite. '); } fts3_tap_test 2.2 db t1 {1 3 6} do_execsql_test 2.3 { DELETE FROM t1 WHERE docid%2; } fts3_tap_test 2.4 db t1 {1 3 6} do_execsql_test 2.5 { INSERT INTO t1(t1) VALUES('optimize') } fts3_tap_test 2.6 db t1 {1 3 6} do_execsql_test 3.1 { CREATE VIRTUAL TABLE t2 USING fts4(prefix='1,2,3'); INSERT INTO t2 VALUES('On 12 September the wind direction turned and'); INSERT INTO t2 VALUES('William''s fleet sailed. A storm blew up and the'); INSERT INTO t2 VALUES('fleet was forced to take shelter at'); INSERT INTO t2 VALUES('Saint-Valery-sur-Somme and again wait for the wind'); INSERT INTO t2 VALUES('to change. On 27 September the Norman fleet'); INSERT INTO t2 VALUES('finally set sail, landing in England at Pevensey'); INSERT INTO t2 VALUES('Bay (Sussex) on 28 September. William then moved'); INSERT INTO t2 VALUES('to Hastings, a few miles to the east, where he'); INSERT INTO t2 VALUES('built a prefabricated wooden castle for a base of'); INSERT INTO t2 VALUES('operations. From there, he ravaged the hinterland'); INSERT INTO t2 VALUES('and waited for Harold''s return from the north.'); INSERT INTO t2 VALUES('On 12 September the wind direction turned and'); INSERT INTO t2 VALUES('William''s fleet sailed. A storm blew up and the'); INSERT INTO t2 VALUES('fleet was forced to take shelter at'); INSERT INTO t2 VALUES('Saint-Valery-sur-Somme and again wait for the wind'); INSERT INTO t2 VALUES('to change. On 27 September the Norman fleet'); INSERT INTO t2 VALUES('finally set sail, landing in England at Pevensey'); INSERT INTO t2 VALUES('Bay (Sussex) on 28 September. William then moved'); INSERT INTO t2 VALUES('to Hastings, a few miles to the east, where he'); INSERT INTO t2 VALUES('built a prefabricated wooden castle for a base of'); INSERT INTO t2 VALUES('operations. From there, he ravaged the hinterland'); INSERT INTO t2 VALUES('and waited for Harold''s return from the north.'); } fts3_tap_test 3.2 db t2 {1 2 3} do_execsql_test 3.3 { SELECT optimize(t2) FROM t2 LIMIT 1 } {{Index optimized}} fts3_tap_test 3.4 db t2 {1 2 3} #------------------------------------------------------------------------- # Simple tests for reading the prefix-index. # do_execsql_test 4.1 { CREATE VIRTUAL TABLE t3 USING fts4(prefix="1,4"); INSERT INTO t3 VALUES('one two three'); INSERT INTO t3 VALUES('four five six'); INSERT INTO t3 VALUES('seven eight nine'); } do_execsql_test 4.2 { SELECT * FROM t3 WHERE t3 MATCH 'f*' } {{four five six}} do_execsql_test 4.3 { SELECT * FROM t3 WHERE t3 MATCH 'four*' } {{four five six}} do_execsql_test 4.4 { SELECT * FROM t3 WHERE t3 MATCH 's*' } {{four five six} {seven eight nine}} do_execsql_test 4.5 { SELECT * FROM t3 WHERE t3 MATCH 'sev*' } {{seven eight nine}} do_execsql_test 4.6 { SELECT * FROM t3 WHERE t3 MATCH 'one*' } {{one two three}} finish_test |
Changes to test/fts3rnd.test.
︙ | ︙ | |||
158 159 160 161 162 163 164 | #lsort -uniq -integer $ret set ret } # This [proc] is used to test the FTS3 matchinfo() function. # | | > > | | 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 | #lsort -uniq -integer $ret set ret } # This [proc] is used to test the FTS3 matchinfo() function. # proc simple_token_matchinfo {zToken bDesc} { set nDoc(0) 0 set nDoc(1) 0 set nDoc(2) 0 set nHit(0) 0 set nHit(1) 0 set nHit(2) 0 set dir -inc if {$bDesc} { set dir -dec } foreach key [array names ::t1] { set value $::t1($key) set a($key) [list] foreach i {0 1 2} col $value { set hit [llength [lsearch -all $col $zToken]] lappend a($key) $hit incr nHit($i) $hit if {$hit>0} { incr nDoc($i) } } } set ret [list] foreach docid [lsort -integer $dir [array names a]] { if { [lindex [lsort -integer $a($docid)] end] } { set matchinfo [list 1 3] foreach i {0 1 2} hit $a($docid) { lappend matchinfo $hit $nHit($i) $nDoc($i) } lappend ret $docid $matchinfo } |
︙ | ︙ | |||
258 259 260 261 262 263 264 265 | proc mit {blob} { set scan(littleEndian) i* set scan(bigEndian) I* binary scan $blob $scan($::tcl_platform(byteOrder)) r return $r } db func mit mit | > > > > > > > | > | > > > > > > > | | > > | 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 | proc mit {blob} { set scan(littleEndian) i* set scan(bigEndian) I* binary scan $blob $scan($::tcl_platform(byteOrder)) r return $r } db func mit mit set sqlite_fts3_enable_parentheses 1 proc do_orderbydocid_test {tn sql res} { uplevel [list do_select_test $tn.asc "$sql ORDER BY docid ASC" $res] uplevel [list do_select_test $tn.desc "$sql ORDER BY docid DESC" \ [lsort -int -dec $res] ] } set NUM_TRIALS 100 foreach {nodesize order} { 50 DESC 50 ASC 500 ASC 1000 DESC 2000 ASC } { catch { array unset ::t1 } set testname "$nodesize/$order" # Create the FTS3 table. Populate it (and the Tcl array) with 100 rows. # db transaction { catchsql { DROP TABLE t1 } execsql "CREATE VIRTUAL TABLE t1 USING fts4(a, b, c, order=$order)" execsql "INSERT INTO t1(t1) VALUES('nodesize=$nodesize')" for {set i 0} {$i < 100} {incr i} { insert_row $i } } for {set iTest 1} {$iTest <= $NUM_TRIALS} {incr iTest} { catchsql COMMIT set DO_MALLOC_TEST 0 set nRep 10 if {$iTest==100 && $nodesize==50} { set DO_MALLOC_TEST 1 set nRep 2 } set ::testprefix fts3rnd-1.$testname.$iTest # Delete one row, update one row and insert one row. # set rows [array names ::t1] set nRow [llength $rows] set iUpdate [lindex $rows [expr {int(rand()*$nRow)}]] set iDelete $iUpdate |
︙ | ︙ | |||
303 304 305 306 307 308 309 | execsql BEGIN insert_row $iInsert update_row $iUpdate delete_row $iDelete if {0==($iTest%2)} { execsql COMMIT } if {0==($iTest%2)} { | | | > | > > > > | | | | | > > | | | | | | | | 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 | execsql BEGIN insert_row $iInsert update_row $iUpdate delete_row $iDelete if {0==($iTest%2)} { execsql COMMIT } if {0==($iTest%2)} { #do_test 0 { fts3_integrity_check t1 } ok } # Pick 10 terms from the vocabulary. Check that the results of querying # the database for the set of documents containing each of these terms # is the same as the result obtained by scanning the contents of the Tcl # array for each term. # for {set i 0} {$i < 10} {incr i} { set term [random_term] do_select_test 1.$i.asc { SELECT docid, mit(matchinfo(t1)) FROM t1 WHERE t1 MATCH $term ORDER BY docid ASC } [simple_token_matchinfo $term 0] do_select_test 1.$i.desc { SELECT docid, mit(matchinfo(t1)) FROM t1 WHERE t1 MATCH $term ORDER BY docid DESC } [simple_token_matchinfo $term 1] } # This time, use the first two characters of each term as a term prefix # to query for. Test that querying the Tcl array produces the same results # as querying the FTS3 table for the prefix. # for {set i 0} {$i < $nRep} {incr i} { set prefix [string range [random_term] 0 end-1] set match "${prefix}*" do_orderbydocid_test 2.$i { SELECT docid FROM t1 WHERE t1 MATCH $match } [simple_phrase $match] } # Similar to the above, except for phrase queries. # for {set i 0} {$i < $nRep} {incr i} { set term [list [random_term] [random_term]] set match "\"$term\"" do_orderbydocid_test 3.$i { SELECT docid FROM t1 WHERE t1 MATCH $match } [simple_phrase $term] } # Three word phrases. # for {set i 0} {$i < $nRep} {incr i} { set term [list [random_term] [random_term] [random_term]] set match "\"$term\"" do_orderbydocid_test 4.$i { SELECT docid FROM t1 WHERE t1 MATCH $match } [simple_phrase $term] } # Three word phrases made up of term-prefixes. # for {set i 0} {$i < $nRep} {incr i} { set query "[string range [random_term] 0 end-1]* " append query "[string range [random_term] 0 end-1]* " append query "[string range [random_term] 0 end-1]*" set match "\"$query\"" do_orderbydocid_test 5.$i { SELECT docid FROM t1 WHERE t1 MATCH $match } [simple_phrase $query] } # A NEAR query with terms as the arguments: # # ... MATCH '$term1 NEAR $term2' ... # for {set i 0} {$i < $nRep} {incr i} { set terms [list [random_term] [random_term]] set match [join $terms " NEAR "] do_orderbydocid_test 6.$i { SELECT docid FROM t1 WHERE t1 MATCH $match } [simple_near $terms 10] } # A 3-way NEAR query with terms as the arguments. # for {set i 0} {$i < $nRep} {incr i} { set terms [list [random_term] [random_term] [random_term]] set nNear 11 set match [join $terms " NEAR/$nNear "] do_orderbydocid_test 7.$i { SELECT docid FROM t1 WHERE t1 MATCH $match } [simple_near $terms $nNear] } # Set operations on simple term queries. # foreach {tn op proc} { 8 OR setop_or 9 NOT setop_not 10 AND setop_and } { for {set i 0} {$i < $nRep} {incr i} { set term1 [random_term] set term2 [random_term] set match "$term1 $op $term2" do_orderbydocid_test $tn.$i { SELECT docid FROM t1 WHERE t1 MATCH $match } [$proc [simple_phrase $term1] [simple_phrase $term2]] } } # Set operations on NEAR queries. # foreach {tn op proc} { 11 OR setop_or 12 NOT setop_not 13 AND setop_and } { for {set i 0} {$i < $nRep} {incr i} { set term1 [random_term] set term2 [random_term] set term3 [random_term] set term4 [random_term] set match "$term1 NEAR $term2 $op $term3 NEAR $term4" do_orderbydocid_test $tn.$i { SELECT docid FROM t1 WHERE t1 MATCH $match } [$proc \ [simple_near [list $term1 $term2] 10] \ [simple_near [list $term3 $term4] 10] ] } } catchsql COMMIT } } finish_test |
Changes to test/fts3sort.test.
︙ | ︙ | |||
17 18 19 20 21 22 23 | # If SQLITE_ENABLE_FTS3 is defined, omit this file. ifcapable !fts3 { finish_test return } | < | | > > > > > > > > > > > | | | | | | | > > > > | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | > > > > | > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > | 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 | # If SQLITE_ENABLE_FTS3 is defined, omit this file. ifcapable !fts3 { finish_test return } proc build_database {nRow param} { db close forcedelete test.db sqlite3 db test.db set vocab [list aa ab ac ba bb bc ca cb cc da] expr srand(0) execsql "CREATE VIRTUAL TABLE t1 USING fts4($param)" for {set i 0} {$i < $nRow} {incr i} { set v [expr int(rand()*1000000)] set doc [list] for {set div 1} {$div < 1000000} {set div [expr $div*10]} { lappend doc [lindex $vocab [expr ($v/$div) % 10]] } execsql { INSERT INTO t1 VALUES($doc) } } } set testprefix fts3sort unset -nocomplain CONTROL foreach {t param} { 1 "" 2 "order=asc" 3 "order=desc" } { set testprefix fts3sort-1.$t set nRow 1000 do_test 1.0 { build_database $nRow $param execsql { SELECT count(*) FROM t1 } } $nRow foreach {tn query} { 1 "SELECT docid, * FROM t1" 2 "SELECT docid, * FROM t1 WHERE t1 MATCH 'aa'" 3 "SELECT docid, * FROM t1 WHERE t1 MATCH 'a*'" 4 "SELECT docid, quote(matchinfo(t1)) FROM t1 WHERE t1 MATCH 'a*'" 5 "SELECT docid, quote(matchinfo(t1,'pcnxals')) FROM t1 WHERE t1 MATCH 'b*'" 6 "SELECT docid, * FROM t1 WHERE t1 MATCH 'a* b* c*'" 7 "SELECT docid, * FROM t1 WHERE t1 MATCH 'aa OR da'" 8 "SELECT docid, * FROM t1 WHERE t1 MATCH 'nosuchtoken'" 9 "SELECT docid, snippet(t1) FROM t1 WHERE t1 MATCH 'aa OR da'" 10 "SELECT docid, snippet(t1) FROM t1 WHERE t1 MATCH 'aa OR nosuchtoken'" 11 "SELECT docid, snippet(t1) FROM t1 WHERE t1 MATCH 'aa NEAR bb'" 12 "SELECT docid, snippet(t1) FROM t1 WHERE t1 MATCH '\"aa bb\"'" 13 "SELECT docid, content FROM t1 WHERE t1 MATCH 'aa NEAR/2 bb NEAR/3 cc'" 14 "SELECT docid, content FROM t1 WHERE t1 MATCH '\"aa bb cc\"'" } { unset -nocomplain A B C D set A_list [list] set B_list [list] set C_list [list] set D_list [list] unset -nocomplain X db eval "$query ORDER BY rowid ASC" X { set A($X(docid)) [array get X] lappend A_list $X(docid) } unset -nocomplain X db eval "$query ORDER BY rowid DESC" X { set B($X(docid)) [array get X] lappend B_list $X(docid) } unset -nocomplain X db eval "$query ORDER BY docid ASC" X { set C($X(docid)) [array get X] lappend C_list $X(docid) } unset -nocomplain X db eval "$query ORDER BY docid DESC" X { set D($X(docid)) [array get X] lappend D_list $X(docid) } do_test $tn.1 { set A_list } [lsort -integer -increasing $A_list] do_test $tn.2 { set B_list } [lsort -integer -decreasing $B_list] do_test $tn.3 { set C_list } [lsort -integer -increasing $C_list] do_test $tn.4 { set D_list } [lsort -integer -decreasing $D_list] unset -nocomplain DATA unset -nocomplain X db eval "$query" X { set DATA($X(docid)) [array get X] } do_test $tn.5 { lsort [array get A] } [lsort [array get DATA]] do_test $tn.6 { lsort [array get B] } [lsort [array get DATA]] do_test $tn.7 { lsort [array get C] } [lsort [array get DATA]] do_test $tn.8 { lsort [array get D] } [lsort [array get DATA]] if {[info exists CONTROL($tn)]} { do_test $tn.9 { set CONTROL($tn) } [lsort [array get DATA]] } else { set CONTROL($tn) [lsort [array get DATA]] } } } unset -nocomplain CONTROL set testprefix fts3sort #------------------------------------------------------------------------- # Tests for parsing the "order=asc" and "order=desc" directives. # foreach {tn param res} { 1 "order=asc" {0 {}} 2 "order=desc" {0 {}} 3 "order=dec" {1 {unrecognized order: dec}} 4 "order=xxx, order=asc" {1 {unrecognized order: xxx}} 5 "order=desc, order=asc" {0 {}} } { execsql { DROP TABLE IF EXISTS t1 } do_catchsql_test 2.1.$tn " CREATE VIRTUAL TABLE t1 USING fts4(a, b, $param) " $res } do_execsql_test 2.2 { BEGIN; CREATE VIRTUAL TABLE t2 USING fts4(order=desc); INSERT INTO t2 VALUES('aa bb'); INSERT INTO t2 VALUES('bb cc'); INSERT INTO t2 VALUES('cc aa'); SELECT docid FROM t2 WHERE t2 MATCH 'aa'; END; } {3 1} do_execsql_test 2.3 { SELECT docid FROM t2 WHERE t2 MATCH 'aa'; } {3 1} finish_test |
Changes to test/hook.test.
︙ | ︙ | |||
271 272 273 274 275 276 277 278 279 280 281 282 283 284 | SELECT * FROM t1 EXCEPT SELECT * FROM t3; SELECT * FROM t1 ORDER BY b; SELECT * FROM t1 GROUP BY b; } set ::update_hook } [list] } db update_hook {} # #---------------------------------------------------------------------------- #---------------------------------------------------------------------------- # Test the rollback-hook. The rollback-hook is a bit more complicated than # either the commit or update hooks because a rollback can happen | > > > > > > > > > > > > > > > > > > > > > > > > > > > > | 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 | SELECT * FROM t1 EXCEPT SELECT * FROM t3; SELECT * FROM t1 ORDER BY b; SELECT * FROM t1 GROUP BY b; } set ::update_hook } [list] } do_test hook-4.4 { execsql { CREATE TABLE t4(a UNIQUE, b); INSERT INTO t4 VALUES(1, 'a'); INSERT INTO t4 VALUES(2, 'b'); } set ::update_hook [list] execsql { REPLACE INTO t4 VALUES(1, 'c'); } set ::update_hook } [list INSERT main t4 3 ] do_execsql_test hook-4.4.1 { SELECT * FROM t4 ORDER BY a; } {1 c 2 b} do_test hook-4.4.2 { set ::update_hook [list] execsql { PRAGMA recursive_triggers = on; REPLACE INTO t4 VALUES(1, 'd'); } set ::update_hook } [list INSERT main t4 4 ] do_execsql_test hook-4.4.3 { SELECT * FROM t4 ORDER BY a; } {1 d 2 b} db update_hook {} # #---------------------------------------------------------------------------- #---------------------------------------------------------------------------- # Test the rollback-hook. The rollback-hook is a bit more complicated than # either the commit or update hooks because a rollback can happen |
︙ | ︙ |
Changes to test/lock_common.tcl.
︙ | ︙ | |||
51 52 53 54 55 56 57 | proc csql1 {sql} { list [catch { sql1 $sql } msg] $msg } proc csql2 {sql} { list [catch { sql2 $sql } msg] $msg } proc csql3 {sql} { list [catch { sql3 $sql } msg] $msg } uplevel set $varname $tn uplevel $script | | | | 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 | proc csql1 {sql} { list [catch { sql1 $sql } msg] $msg } proc csql2 {sql} { list [catch { sql2 $sql } msg] $msg } proc csql3 {sql} { list [catch { sql3 $sql } msg] $msg } uplevel set $varname $tn uplevel $script catch { code2 { db2 close } } catch { code3 { db3 close } } catch { close $::code2_chan } catch { close $::code3_chan } catch { db close } } } # Launch another testfixture process to be controlled by this one. A |
︙ | ︙ |
Changes to test/malloc.test.
︙ | ︙ | |||
889 890 891 892 893 894 895 896 897 898 899 900 901 902 | sqlite3 db test.db sqlite3_db_config_lookaside db 0 0 0 add_test_collate db 0 0 1 } -sqlbody { SELECT * FROM t4 AS t41, t4 AS t42 WHERE t41.x>'ddd' AND t42.x>'ccc' } } # Ensure that no file descriptors were leaked. do_test malloc-99.X { catch {db close} set sqlite_open_file_count } {0} | > > > > > > > > > > > > | 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 | sqlite3 db test.db sqlite3_db_config_lookaside db 0 0 0 add_test_collate db 0 0 1 } -sqlbody { SELECT * FROM t4 AS t41, t4 AS t42 WHERE t41.x>'ddd' AND t42.x>'ccc' } } # Test that if an OOM error occurs, aux-data is still correctly destroyed. # This test case was causing either a memory-leak or an assert() failure # at one point, depending on the configuration. # do_malloc_test 39 -tclprep { sqlite3 db test.db } -sqlbody { SELECT test_auxdata('abc', 'def'); } -cleanup { db close } # Ensure that no file descriptors were leaked. do_test malloc-99.X { catch {db close} set sqlite_open_file_count } {0} |
︙ | ︙ |
Changes to test/multiplex.test.
︙ | ︙ | |||
568 569 570 571 572 573 574 575 576 577 578 579 580 581 | } #------------------------------------------------------------------------- # Test that you can vacuum a multiplex'ed DB. ifcapable vacuum { do_test multiplex-6.0.0 { multiplex_delete test.db sqlite3_multiplex_initialize "" 1 sqlite3 db test.db multiplex_set db main 4096 16 } {SQLITE_OK} | > | 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 | } #------------------------------------------------------------------------- # Test that you can vacuum a multiplex'ed DB. ifcapable vacuum { sqlite3_multiplex_shutdown do_test multiplex-6.0.0 { multiplex_delete test.db sqlite3_multiplex_initialize "" 1 sqlite3 db test.db multiplex_set db main 4096 16 } {SQLITE_OK} |
︙ | ︙ |
Changes to test/pager1.test.
︙ | ︙ | |||
437 438 439 440 441 442 443 444 445 446 447 448 449 450 | INSERT INTO t1 SELECT a_string(204), a_string(304) FROM t1; REPLACE INTO t2 SELECT * FROM t1; COMMIT; } db close tstvfs delete } {} do_test pager1.4.2.2 { faultsim_restore_and_reopen execsql { SELECT count(*) FROM t1; PRAGMA integrity_check; } } {4 ok} | > > | 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 | INSERT INTO t1 SELECT a_string(204), a_string(304) FROM t1; REPLACE INTO t2 SELECT * FROM t1; COMMIT; } db close tstvfs delete } {} if {$::tcl_platform(platform)!="windows"} { do_test pager1.4.2.2 { faultsim_restore_and_reopen execsql { SELECT count(*) FROM t1; PRAGMA integrity_check; } } {4 ok} |
︙ | ︙ | |||
469 470 471 472 473 474 475 476 477 478 479 480 481 482 | hexio_write test.db-journal [expr [file size test.db-journal]-20] 123456 foreach f [glob test.db-mj*] { file delete -force $f } execsql { SELECT count(*) FROM t1; PRAGMA integrity_check; } } {4 ok} do_test pager1.4.3.1 { testvfs tstvfs -default 1 tstvfs filter xSync tstvfs script xSyncCallback proc xSyncCallback {method file args} { set file [file tail $file] | > | 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 | hexio_write test.db-journal [expr [file size test.db-journal]-20] 123456 foreach f [glob test.db-mj*] { file delete -force $f } execsql { SELECT count(*) FROM t1; PRAGMA integrity_check; } } {4 ok} } do_test pager1.4.3.1 { testvfs tstvfs -default 1 tstvfs filter xSync tstvfs script xSyncCallback proc xSyncCallback {method file args} { set file [file tail $file] |
︙ | ︙ | |||
1544 1545 1546 1547 1548 1549 1550 1551 1552 1553 1554 1555 1556 1557 | INSERT INTO t1 SELECT NULL, a_string(400) FROM t1; /* 32 */ INSERT INTO t1 SELECT NULL, a_string(400) FROM t1; /* 64 */ INSERT INTO t1 SELECT NULL, a_string(400) FROM t1; /* 128 */ COMMIT; UPDATE t1 SET b = a_string(400); } {persist} # Run transactions of increasing sizes. Eventually, one (or more than one) # of these will write just enough content that one of the old headers created # by the transaction in the block above lies immediately after the content # journalled by the current transaction. # for {set nUp 1} {$nUp<64} {incr nUp} { do_execsql_test pager1-13.1.2.$nUp.1 { | > | 1547 1548 1549 1550 1551 1552 1553 1554 1555 1556 1557 1558 1559 1560 1561 | INSERT INTO t1 SELECT NULL, a_string(400) FROM t1; /* 32 */ INSERT INTO t1 SELECT NULL, a_string(400) FROM t1; /* 64 */ INSERT INTO t1 SELECT NULL, a_string(400) FROM t1; /* 128 */ COMMIT; UPDATE t1 SET b = a_string(400); } {persist} if {$::tcl_platform(platform)!="windows"} { # Run transactions of increasing sizes. Eventually, one (or more than one) # of these will write just enough content that one of the old headers created # by the transaction in the block above lies immediately after the content # journalled by the current transaction. # for {set nUp 1} {$nUp<64} {incr nUp} { do_execsql_test pager1-13.1.2.$nUp.1 { |
︙ | ︙ | |||
1566 1567 1568 1569 1570 1571 1572 | execsql { SELECT sum(length(b)) FROM t1 } db2 } [expr {128*400 - ($nUp-1)}] do_test pager1-13.1.2.$nUp.4 { execsql { PRAGMA integrity_check } db2 } {ok} db2 close } | | > > > | 1570 1571 1572 1573 1574 1575 1576 1577 1578 1579 1580 1581 1582 1583 1584 1585 1586 1587 1588 1589 1590 1591 1592 1593 1594 1595 1596 1597 1598 1599 1600 1601 1602 1603 1604 1605 1606 | execsql { SELECT sum(length(b)) FROM t1 } db2 } [expr {128*400 - ($nUp-1)}] do_test pager1-13.1.2.$nUp.4 { execsql { PRAGMA integrity_check } db2 } {ok} db2 close } } if {$::tcl_platform(platform)!="windows"} { # Same test as above. But this time with an index on the table. # do_execsql_test pager1-13.2.1 { CREATE INDEX i1 ON t1(b); UPDATE t1 SET b = a_string(400); } {} for {set nUp 1} {$nUp<64} {incr nUp} { do_execsql_test pager1-13.2.2.$nUp.1 { UPDATE t1 SET b = a_string(399) WHERE a <= $nUp } {} do_execsql_test pager1-13.2.2.$nUp.2 { PRAGMA integrity_check } {ok} sqlite3 db2 sv_test.db do_test pager1-13.2.2.$nUp.3 { execsql { SELECT sum(length(b)) FROM t1 } db2 } [expr {128*400 - ($nUp-1)}] do_test pager1-13.2.2.$nUp.4 { execsql { PRAGMA integrity_check } db2 } {ok} db2 close } } db close tv delete #------------------------------------------------------------------------- # Test specal "PRAGMA journal_mode=OFF" test cases. |
︙ | ︙ |
Changes to test/pagerfault.test.
︙ | ︙ | |||
15 16 17 18 19 20 21 22 23 24 25 26 27 28 | source $testdir/lock_common.tcl source $testdir/malloc_common.tcl if {[permutation] == "inmemory_journal"} { finish_test return } set a_string_counter 1 proc a_string {n} { global a_string_counter incr a_string_counter string range [string repeat "${a_string_counter}." $n] 1 $n } | > > > > > | 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 | source $testdir/lock_common.tcl source $testdir/malloc_common.tcl if {[permutation] == "inmemory_journal"} { finish_test return } if {$::tcl_platform(platform)=="windows"} { finish_test return } set a_string_counter 1 proc a_string {n} { global a_string_counter incr a_string_counter string range [string repeat "${a_string_counter}." $n] 1 $n } |
︙ | ︙ |
Changes to test/permutations.test.
︙ | ︙ | |||
178 179 180 181 182 183 184 185 186 187 | } -files { fts3aa.test fts3ab.test fts3ac.test fts3ad.test fts3ae.test fts3af.test fts3ag.test fts3ah.test fts3ai.test fts3aj.test fts3ak.test fts3al.test fts3am.test fts3an.test fts3ao.test fts3atoken.test fts3b.test fts3c.test fts3cov.test fts3d.test fts3defer.test fts3defer2.test fts3e.test fts3expr.test fts3expr2.test fts3near.test fts3query.test fts3shared.test fts3snippet.test fts3fault.test fts3malloc.test fts3matchinfo.test | > | | 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 | } -files { fts3aa.test fts3ab.test fts3ac.test fts3ad.test fts3ae.test fts3af.test fts3ag.test fts3ah.test fts3ai.test fts3aj.test fts3ak.test fts3al.test fts3am.test fts3an.test fts3ao.test fts3atoken.test fts3b.test fts3c.test fts3cov.test fts3d.test fts3defer.test fts3defer2.test fts3e.test fts3expr.test fts3expr2.test fts3near.test fts3query.test fts3shared.test fts3snippet.test fts3sort.test fts3fault.test fts3malloc.test fts3matchinfo.test fts3aux1.test fts3comp1.test fts3auto.test } lappend ::testsuitelist xxx #------------------------------------------------------------------------- # Define the coverage related test suites: # |
︙ | ︙ | |||
527 528 529 530 531 532 533 | pager1.test async4.test corrupt.test filefmt.test pager2.test corrupt5.test corruptA.test pageropt.test # Exclude stmt.test, which expects sub-journals to use temporary files. stmt.test # WAL mode is different. | | | 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 | pager1.test async4.test corrupt.test filefmt.test pager2.test corrupt5.test corruptA.test pageropt.test # Exclude stmt.test, which expects sub-journals to use temporary files. stmt.test # WAL mode is different. wal* tkt-2d1a5c67d.test }] ifcapable mem3 { test_suite "memsys3" -description { Run tests using the allocator in mem3.c. } -files [test_set $::allquicktests -exclude { autovacuum.test delete3.test manydb.test |
︙ | ︙ |
Changes to test/tester.tcl.
︙ | ︙ | |||
376 377 378 379 380 381 382 | set L2 [list] foreach l $L {lappend L2 $l} set L2 } proc do_execsql_test {testname sql {result {}}} { fix_testname testname | | | | 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 | set L2 [list] foreach l $L {lappend L2 $l} set L2 } proc do_execsql_test {testname sql {result {}}} { fix_testname testname uplevel do_test [list $testname] [list "execsql {$sql}"] [list [list {*}$result]] } proc do_catchsql_test {testname sql result} { fix_testname testname uplevel do_test [list $testname] [list "catchsql {$sql}"] [list $result] } proc do_eqp_test {name sql res} { uplevel do_execsql_test $name [list "EXPLAIN QUERY PLAN $sql"] [list $res] } #------------------------------------------------------------------------- # Usage: do_select_tests PREFIX ?SWITCHES? TESTLIST |
︙ | ︙ |
Changes to test/trace2.test.
︙ | ︙ | |||
137 138 139 140 141 142 143 | "-- INSERT INTO 'main'.'x1_segdir' VALUES(?,?,?,?,?,?)" } do_trace_test 2.3 { INSERT INTO x1(x1) VALUES('optimize'); } { "INSERT INTO x1(x1) VALUES('optimize');" | | | | | 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 | "-- INSERT INTO 'main'.'x1_segdir' VALUES(?,?,?,?,?,?)" } do_trace_test 2.3 { INSERT INTO x1(x1) VALUES('optimize'); } { "INSERT INTO x1(x1) VALUES('optimize');" "-- SELECT idx, start_block, leaves_end_block, end_block, root FROM 'main'.'x1_segdir' WHERE level BETWEEN ? AND ?ORDER BY level DESC, idx ASC" "-- SELECT max(level) FROM 'main'.'x1_segdir' WHERE level BETWEEN ? AND ?" "-- SELECT coalesce((SELECT max(blockid) FROM 'main'.'x1_segments') + 1, 1)" "-- DELETE FROM 'main'.'x1_segdir' WHERE level BETWEEN ? AND ?" "-- INSERT INTO 'main'.'x1_segdir' VALUES(?,?,?,?,?,?)" } } finish_test |
Changes to test/wal7.test.
︙ | ︙ | |||
57 58 59 60 61 62 63 | PRAGMA wal_autocheckpoint=50; -- 50 pages PRAGMA journal_size_limit=25000; CREATE TABLE t1(x, y UNIQUE); INSERT INTO t1 VALUES(1,2); INSERT INTO t1 VALUES(zeroblob(200000),4); CREATE TABLE t2(z); DELETE FROM t1; | | | 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 | PRAGMA wal_autocheckpoint=50; -- 50 pages PRAGMA journal_size_limit=25000; CREATE TABLE t1(x, y UNIQUE); INSERT INTO t1 VALUES(1,2); INSERT INTO t1 VALUES(zeroblob(200000),4); CREATE TABLE t2(z); DELETE FROM t1; INSERT INTO t2 VALUES(1); } file size test.db-wal } 25000 # Case 3: Size limit of zero. # |
︙ | ︙ | |||
79 80 81 82 83 84 85 | PRAGMA wal_autocheckpoint=50; -- 50 pages PRAGMA journal_size_limit=0; CREATE TABLE t1(x, y UNIQUE); INSERT INTO t1 VALUES(1,2); INSERT INTO t1 VALUES(zeroblob(200000),4); CREATE TABLE t2(z); DELETE FROM t1; | | | | | 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 | PRAGMA wal_autocheckpoint=50; -- 50 pages PRAGMA journal_size_limit=0; CREATE TABLE t1(x, y UNIQUE); INSERT INTO t1 VALUES(1,2); INSERT INTO t1 VALUES(zeroblob(200000),4); CREATE TABLE t2(z); DELETE FROM t1; INSERT INTO t2 VALUES(1); } set sz [file size test.db-wal] expr {$sz>0 && $sz<13700} } 1 # Case 4: Size limit set before going WAL # do_test wal7-4.0 { db close forcedelete test.db sqlite3 db test.db db eval { PRAGMA page_size=1024; PRAGMA journal_size_limit=25000; PRAGMA journal_mode=WAL; PRAGMA wal_autocheckpoint=50; -- 50 pages CREATE TABLE t1(x, y UNIQUE); INSERT INTO t1 VALUES(1,2); INSERT INTO t1 VALUES(zeroblob(200000),4); CREATE TABLE t2(z); DELETE FROM t1; INSERT INTO t2 VALUES(1); } set sz [file size test.db-wal] } 25000 finish_test |
Added test/walro.test.
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 | # 2011 May 09 # # The author disclaims copyright to this source code. In place of # a legal notice, here is a blessing: # # May you do good and not evil. # May you find forgiveness for yourself and forgive others. # May you share freely, never taking more than you give. # #*********************************************************************** # # This file contains tests for using WAL databases in read-only mode. # set testdir [file dirname $argv0] source $testdir/tester.tcl source $testdir/lock_common.tcl set ::testprefix walro # These tests are only going to work on unix. # if {$::tcl_platform(platform) != "unix"} { finish_test return } do_multiclient_test tn { # Do not run tests with the connections in the same process. # if {$tn==2} continue # Close all connections and delete the database. # code1 { db close } code2 { db2 close } code3 { db3 close } forcedelete test.db forcedelete walro foreach c {code1 code2 code3} { $c { sqlite3_shutdown sqlite3_config_uri 1 } } file mkdir walro do_test 1.1.1 { code2 { sqlite3 db2 test.db } sql2 { PRAGMA journal_mode = WAL; CREATE TABLE t1(x, y); INSERT INTO t1 VALUES('a', 'b'); } file exists test.db-shm } {1} do_test 1.1.2 { file attributes test.db-shm -permissions r--r--r-- code1 { sqlite3 db file:test.db?readonly_shm=1 } } {} do_test 1.1.3 { sql1 "SELECT * FROM t1" } {a b} do_test 1.1.4 { sql2 "INSERT INTO t1 VALUES('c', 'd')" } {} do_test 1.1.5 { sql1 "SELECT * FROM t1" } {a b c d} # Check that the read-only connection cannot write or checkpoint the db. # do_test 1.1.6 { csql1 "INSERT INTO t1 VALUES('e', 'f')" } {1 {attempt to write a readonly database}} do_test 1.1.7 { csql1 "PRAGMA wal_checkpoint" } {1 {attempt to write a readonly database}} do_test 1.1.9 { sql2 "INSERT INTO t1 VALUES('e', 'f')" } {} do_test 1.1.10 { sql1 "SELECT * FROM t1" } {a b c d e f} do_test 1.1.11 { sql2 { INSERT INTO t1 VALUES('g', 'h'); PRAGMA wal_checkpoint; } set {} {} } {} do_test 1.1.12 { sql1 "SELECT * FROM t1" } {a b c d e f g h} do_test 1.1.13 { sql2 "INSERT INTO t1 VALUES('i', 'j')" } {} do_test 1.2.1 { code2 { db2 close } code1 { db close } list [file exists test.db-wal] [file exists test.db-shm] } {1 1} do_test 1.2.2 { code1 { sqlite3 db file:test.db?readonly_shm=1 } sql1 { SELECT * FROM t1 } } {a b c d e f g h i j} do_test 1.2.3 { code1 { db close } file attributes test.db-shm -permissions rw-r--r-- hexio_write test.db-shm 0 01020304 file attributes test.db-shm -permissions r--r--r-- code1 { sqlite3 db file:test.db?readonly_shm=1 } csql1 { SELECT * FROM t1 } } {1 {attempt to write a readonly database}} do_test 1.2.4 { code1 { sqlite3_extended_errcode db } } {SQLITE_READONLY_RECOVERY} do_test 1.2.5 { file attributes test.db-shm -permissions rw-r--r-- code2 { sqlite3 db2 test.db } sql2 "SELECT * FROM t1" } {a b c d e f g h i j} file attributes test.db-shm -permissions r--r--r-- do_test 1.2.6 { sql1 "SELECT * FROM t1" } {a b c d e f g h i j} do_test 1.2.7 { sql2 { PRAGMA wal_checkpoint; INSERT INTO t1 VALUES('k', 'l'); } set {} {} } {} do_test 1.2.8 { sql1 "SELECT * FROM t1" } {a b c d e f g h i j k l} # Now check that if the readonly_shm option is not supplied, or if it # is set to zero, it is not possible to connect to the database without # read-write access to the shm. do_test 1.3.1 { code1 { db close } code1 { sqlite3 db test.db } csql1 { SELECT * FROM t1 } } {1 {unable to open database file}} # Also test that if the -shm file can be opened for read/write access, # it is, even if readonly_shm=1 is present in the URI. do_test 1.3.2.1 { code1 { db close } code2 { db2 close } file exists test.db-shm } {0} do_test 1.3.2.2 { code1 { sqlite3 db file:test.db?readonly_shm=1 } sql1 { SELECT * FROM t1 } } {a b c d e f g h i j k l} do_test 1.3.2.3 { code1 { db close } close [open test.db-shm w] file attributes test.db-shm -permissions r--r--r-- code1 { sqlite3 db file:test.db?readonly_shm=1 } csql1 { SELECT * FROM t1 } } {1 {attempt to write a readonly database}} do_test 1.3.2.4 { code1 { sqlite3_extended_errcode db } } {SQLITE_READONLY_RECOVERY} } finish_test |
Added tool/build-shell.sh.
> > > > > > > > > > > > > > > > > > > > > | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 | #!/bin/sh # # This script demonstrates how to do a full-featured build of the sqlite3 # command-line shell on Linux. # # SQLite source code should be in a sibling directory named "sqlite". For # example, put SQLite sources in ~/sqlite/sqlite and run this script from # ~/sqlite/bld. There should be an appropriate Makefile in the current # directory as well. # make sqlite3.c gcc -o sqlite3 -g -Os -I. \ -DSQLITE_THREADSAFE=0 \ -DSQLITE_ENABLE_VFSTRACE \ -DSQLITE_ENABLE_STAT2 \ -DSQLITE_ENABLE_FTS3 \ -DSQLITE_ENABLE_RTREE \ -DHAVE_READLINE \ -DHAVE_USLEEP=1 \ ../sqlite/src/shell.c ../sqlite/src/test_vfstrace.c \ sqlite3.c -ldl -lreadline -lncurses |
Changes to tool/lemon.c.
︙ | ︙ | |||
3430 3431 3432 3433 3434 3435 3436 3437 3438 3439 3440 3441 3442 3443 3444 3445 3446 3447 3448 3449 | int i,j; /* Loop counters */ int hash; /* For hashing the name of a type */ const char *name; /* Name of the parser */ /* Allocate and initialize types[] and allocate stddt[] */ arraysize = lemp->nsymbol * 2; types = (char**)calloc( arraysize, sizeof(char*) ); for(i=0; i<arraysize; i++) types[i] = 0; maxdtlength = 0; if( lemp->vartype ){ maxdtlength = lemonStrlen(lemp->vartype); } for(i=0; i<lemp->nsymbol; i++){ int len; struct symbol *sp = lemp->symbols[i]; if( sp->datatype==0 ) continue; len = lemonStrlen(sp->datatype); if( len>maxdtlength ) maxdtlength = len; } stddt = (char*)malloc( maxdtlength*2 + 1 ); | > > > > | | 3430 3431 3432 3433 3434 3435 3436 3437 3438 3439 3440 3441 3442 3443 3444 3445 3446 3447 3448 3449 3450 3451 3452 3453 3454 3455 3456 3457 3458 3459 3460 3461 | int i,j; /* Loop counters */ int hash; /* For hashing the name of a type */ const char *name; /* Name of the parser */ /* Allocate and initialize types[] and allocate stddt[] */ arraysize = lemp->nsymbol * 2; types = (char**)calloc( arraysize, sizeof(char*) ); if( types==0 ){ fprintf(stderr,"Out of memory.\n"); exit(1); } for(i=0; i<arraysize; i++) types[i] = 0; maxdtlength = 0; if( lemp->vartype ){ maxdtlength = lemonStrlen(lemp->vartype); } for(i=0; i<lemp->nsymbol; i++){ int len; struct symbol *sp = lemp->symbols[i]; if( sp->datatype==0 ) continue; len = lemonStrlen(sp->datatype); if( len>maxdtlength ) maxdtlength = len; } stddt = (char*)malloc( maxdtlength*2 + 1 ); if( stddt==0 ){ fprintf(stderr,"Out of memory.\n"); exit(1); } /* Build a hash table of datatypes. The ".dtnum" field of each symbol ** is filled in with the hash index plus 1. A ".dtnum" value of 0 is ** used for terminal symbols. If there is no %default_type defined then |
︙ | ︙ |
Changes to tool/mksqlite3c.tcl.
︙ | ︙ | |||
42 43 44 45 46 47 48 49 50 51 52 53 54 55 | } close $in # Open the output file and write a header comment at the beginning # of the file. # set out [open sqlite3.c w] set today [clock format [clock seconds] -format "%Y-%m-%d %H:%M:%S UTC" -gmt 1] puts $out [subst \ {/****************************************************************************** ** This file is an amalgamation of many separate C source files from SQLite ** version $VERSION. By combining all the individual C code files into this ** single large file, the entire code can be compiled as a single translation ** unit. This allows many compilers to do optimizations that would not be | > > | 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 | } close $in # Open the output file and write a header comment at the beginning # of the file. # set out [open sqlite3.c w] # Force the output to use unix line endings, even on Windows. # fconfigure $out -translation lf set today [clock format [clock seconds] -format "%Y-%m-%d %H:%M:%S UTC" -gmt 1] puts $out [subst \ {/****************************************************************************** ** This file is an amalgamation of many separate C source files from SQLite ** version $VERSION. By combining all the individual C code files into this ** single large file, the entire code can be compiled as a single translation ** unit. This allows many compilers to do optimizations that would not be |
︙ | ︙ |
Changes to tool/mksqlite3h.tcl.
︙ | ︙ | |||
60 61 62 63 64 65 66 67 68 69 70 71 72 73 | } close $in # Set up patterns for recognizing API declarations. # set varpattern {^[a-zA-Z][a-zA-Z_0-9 *]+sqlite3_[_a-zA-Z0-9]+(\[|;| =)} set declpattern {^ *[a-zA-Z][a-zA-Z_0-9 ]+ \**sqlite3_[_a-zA-Z0-9]+\(} # Process the src/sqlite.h.in ext/rtree/sqlite3rtree.h files. # foreach file [list $TOP/src/sqlite.h.in $TOP/ext/rtree/sqlite3rtree.h] { set in [open $file] while {![eof $in]} { | > > > | 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 | } close $in # Set up patterns for recognizing API declarations. # set varpattern {^[a-zA-Z][a-zA-Z_0-9 *]+sqlite3_[_a-zA-Z0-9]+(\[|;| =)} set declpattern {^ *[a-zA-Z][a-zA-Z_0-9 ]+ \**sqlite3_[_a-zA-Z0-9]+\(} # Force the output to use unix line endings, even on Windows. fconfigure stdout -translation lf # Process the src/sqlite.h.in ext/rtree/sqlite3rtree.h files. # foreach file [list $TOP/src/sqlite.h.in $TOP/ext/rtree/sqlite3rtree.h] { set in [open $file] while {![eof $in]} { |
︙ | ︙ |
Changes to tool/shell1.test.
︙ | ︙ | |||
195 196 197 198 199 200 201 | set rc [lindex $res 0] list $rc \ [regexp {Error: missing argument for option: -nullvalue} $res] } {1 1} # -version show SQLite version do_test shell1-1.16.1 { | | > | | 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 | set rc [lindex $res 0] list $rc \ [regexp {Error: missing argument for option: -nullvalue} $res] } {1 1} # -version show SQLite version do_test shell1-1.16.1 { set x [catchcmd "-version test.db" ""] regexp {0 \{3.\d.\d+ 20\d\d-[01]\d-\d\d \d\d:\d\d:\d\d [0-9a-f]+\}} $x } 1 #---------------------------------------------------------------------------- # Test cases shell1-2.*: Basic "dot" command token parsing. # # check first token handling do_test shell1-2.1.1 { |
︙ | ︙ |
Added tool/symbols.sh.
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 | #!/bin/sh # # Run this script in a directory that contains a valid SQLite makefile in # order to verify that unintentionally exported symbols. # make sqlite3.c echo '****** Exported symbols from a build including RTREE, FTS4 & ICU ******' gcc -c -DSQLITE_ENABLE_FTS3 -DSQLITE_ENABLE_RTREE \ -DSQLITE_ENABLE_MEMORY_MANAGEMENT -DSQLITE_ENABLE_STAT2 \ -DSQLITE_ENABLE_MEMSYS5 -DSQLITE_ENABLE_UNLOCK_NOTIFY \ -DSQLITE_ENABLE_COLUMN_METADATA -DSQLITE_ENABLE_ATOMIC_WRITE \ -DSQLITE_ENABLE_ICU \ sqlite3.c nm sqlite3.o | grep ' T ' | sort -k 3 echo '****** Surplus symbols from a build including RTREE, FTS4 & ICU ******' nm sqlite3.o | grep ' T ' | grep -v ' sqlite3_' echo '****** Dependencies of the core. No extensions. No OS interface *******' gcc -c -DSQLITE_ENABLE_MEMORY_MANAGEMENT -DSQLITE_ENABLE_STAT2 \ -DSQLITE_ENABLE_MEMSYS5 -DSQLITE_ENABLE_UNLOCK_NOTIFY \ -DSQLITE_ENABLE_COLUMN_METADATA -DSQLITE_ENABLE_ATOMIC_WRITE \ -DSQLITE_OS_OTHER -DSQLITE_THREADSAFE=0 \ sqlite3.c nm sqlite3.o | grep ' U ' | sort -k 3 echo '****** Dependencies including RTREE & FTS4 *******' gcc -c -DSQLITE_ENABLE_FTS3 -DSQLITE_ENABLE_RTREE \ -DSQLITE_ENABLE_MEMORY_MANAGEMENT -DSQLITE_ENABLE_STAT2 \ -DSQLITE_ENABLE_MEMSYS5 -DSQLITE_ENABLE_UNLOCK_NOTIFY \ -DSQLITE_ENABLE_COLUMN_METADATA -DSQLITE_ENABLE_ATOMIC_WRITE \ sqlite3.c nm sqlite3.o | grep ' U ' | sort -k 3 |
Added tool/warnings.sh.
> > > > > > > > > > > > > > | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 | #/bin/sh # # Run this script in a directory with a working makefile to check for # compiler warnings in SQLite. # make sqlite3.c echo '********** No optimizations. Includes FTS4 and RTREE *********' gcc -c -Wshadow -Wall -Wextra -pedantic-errors -Wno-long-long -std=c89 \ -ansi -DHAVE_STDINT_H -DSQLITE_ENABLE_FTS4 -DSQLITE_ENABLE_RTREE \ sqlite3.c echo '********** Optimized -O3. Includes FTS4 and RTREE *********' gcc -O3 -c -Wshadow -Wall -Wextra -pedantic-errors -Wno-long-long -std=c89 \ -ansi -DHAVE_STDINT_H -DSQLITE_ENABLE_FTS4 -DSQLITE_ENABLE_RTREE \ sqlite3.c |