Many hyperlinks are disabled.
Use anonymous login
to enable hyperlinks.
Changes In Branch dbdata Excluding Merge-Ins
This is equivalent to a diff from 7be6222c to 425d708c
2019-04-27
| ||
20:30 | Add the ".recover" command to the shell tool. For recovering as much data as possible from corrupt databases. (check-in: 50fe4845 user: dan tags: trunk) | |
20:15 | Fix building the shell with SQLITE_OMIT_VIRTUAL_TABLE. And without SQLITE_ENABLE_DBPAGE_VTAB. (Closed-Leaf check-in: 425d708c user: dan tags: dbdata) | |
19:36 | Add comments and fix formatting issues in new code in shell.c.in. (check-in: b91d819b user: dan tags: dbdata) | |
2019-04-26
| ||
17:20 | An experimental interface for retrieving the estimated cost and estimated number of output rows for a query. (check-in: 1b25fa10 user: drh tags: cost-est) | |
17:08 | Omit tests of the LIKE optimization in like3.test when SQLITE_ENABLE_ICU is defined. (check-in: af53c41a user: dan tags: trunk) | |
2019-04-25
| ||
20:06 | Merge latest trunk changes into this branch. (check-in: 1da302d8 user: dan tags: dbdata) | |
18:15 | Add the sqlite3_hard_heap_limit64() interface and the corresponding "PRAGMA hard_heap_limit=N" command. (check-in: b0ccef61 user: drh tags: hard-heap-limit) | |
2019-04-24
| ||
17:04 | New test cases in test/fuzzdata8.db. (check-in: 7be6222c user: drh tags: trunk) | |
16:13 | Fix another instance in fts3 where a corrupt record can cause a buffer overflow. (check-in: e1724f1d user: dan tags: trunk) | |
Added ext/misc/dbdata.c.
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 | /* ** 2019-04-17 ** ** The author disclaims copyright to this source code. In place of ** a legal notice, here is a blessing: ** ** May you do good and not evil. ** May you find forgiveness for yourself and forgive others. ** May you share freely, never taking more than you give. ** ****************************************************************************** ** ** This file contains an implementation of two eponymous virtual tables, ** "sqlite_dbdata" and "sqlite_dbptr". Both modules require that the ** "sqlite_dbpage" eponymous virtual table be available. ** ** SQLITE_DBDATA: ** sqlite_dbdata is used to extract data directly from a database b-tree ** page and its associated overflow pages, bypassing the b-tree layer. ** The table schema is equivalent to: ** ** CREATE TABLE sqlite_dbdata( ** pgno INTEGER, ** cell INTEGER, ** field INTEGER, ** value ANY, ** schema TEXT HIDDEN ** ); ** ** IMPORTANT: THE VIRTUAL TABLE SCHEMA ABOVE IS SUBJECT TO CHANGE. IN THE ** FUTURE NEW NON-HIDDEN COLUMNS MAY BE ADDED BETWEEN "value" AND ** "schema". ** ** Each page of the database is inspected. If it cannot be interpreted as ** a b-tree page, or if it is a b-tree page containing 0 entries, the ** sqlite_dbdata table contains no rows for that page. Otherwise, the ** table contains one row for each field in the record associated with ** each cell on the page. For intkey b-trees, the key value is stored in ** field -1. ** ** For example, for the database: ** ** CREATE TABLE t1(a, b); -- root page is page 2 ** INSERT INTO t1(rowid, a, b) VALUES(5, 'v', 'five'); ** INSERT INTO t1(rowid, a, b) VALUES(10, 'x', 'ten'); ** ** the sqlite_dbdata table contains, as well as from entries related to ** page 1, content equivalent to: ** ** INSERT INTO sqlite_dbdata(pgno, cell, field, value) VALUES ** (2, 0, -1, 5 ), ** (2, 0, 0, 'v' ), ** (2, 0, 1, 'five'), ** (2, 1, -1, 10 ), ** (2, 1, 0, 'x' ), ** (2, 1, 1, 'ten' ); ** ** If database corruption is encountered, this module does not report an ** error. Instead, it attempts to extract as much data as possible and ** ignores the corruption. ** ** SQLITE_DBPTR: ** The sqlite_dbptr table has the following schema: ** ** CREATE TABLE sqlite_dbptr( ** pgno INTEGER, ** child INTEGER, ** schema TEXT HIDDEN ** ); ** ** It contains one entry for each b-tree pointer between a parent and ** child page in the database. */ #if !defined(SQLITEINT_H) #include "sqlite3ext.h" typedef unsigned char u8; #endif SQLITE_EXTENSION_INIT1 #include <string.h> #include <assert.h> typedef struct DbdataTable DbdataTable; typedef struct DbdataCursor DbdataCursor; /* Cursor object */ struct DbdataCursor { sqlite3_vtab_cursor base; /* Base class. Must be first */ sqlite3_stmt *pStmt; /* For fetching database pages */ int iPgno; /* Current page number */ u8 *aPage; /* Buffer containing page */ int nPage; /* Size of aPage[] in bytes */ int nCell; /* Number of cells on aPage[] */ int iCell; /* Current cell number */ int bOnePage; /* True to stop after one page */ int szDb; sqlite3_int64 iRowid; /* Only for the sqlite_dbdata table */ u8 *pRec; /* Buffer containing current record */ int nRec; /* Size of pRec[] in bytes */ int nHdr; /* Size of header in bytes */ int iField; /* Current field number */ u8 *pHdrPtr; u8 *pPtr; sqlite3_int64 iIntkey; /* Integer key value */ }; /* Table object */ struct DbdataTable { sqlite3_vtab base; /* Base class. Must be first */ sqlite3 *db; /* The database connection */ sqlite3_stmt *pStmt; /* For fetching database pages */ int bPtr; /* True for sqlite3_dbptr table */ }; /* Column and schema definitions for sqlite_dbdata */ #define DBDATA_COLUMN_PGNO 0 #define DBDATA_COLUMN_CELL 1 #define DBDATA_COLUMN_FIELD 2 #define DBDATA_COLUMN_VALUE 3 #define DBDATA_COLUMN_SCHEMA 4 #define DBDATA_SCHEMA \ "CREATE TABLE x(" \ " pgno INTEGER," \ " cell INTEGER," \ " field INTEGER," \ " value ANY," \ " schema TEXT HIDDEN" \ ")" /* Column and schema definitions for sqlite_dbptr */ #define DBPTR_COLUMN_PGNO 0 #define DBPTR_COLUMN_CHILD 1 #define DBPTR_COLUMN_SCHEMA 2 #define DBPTR_SCHEMA \ "CREATE TABLE x(" \ " pgno INTEGER," \ " child INTEGER," \ " schema TEXT HIDDEN" \ ")" /* ** Connect to an sqlite_dbdata (pAux==0) or sqlite_dbptr (pAux!=0) virtual ** table. */ static int dbdataConnect( sqlite3 *db, void *pAux, int argc, const char *const*argv, sqlite3_vtab **ppVtab, char **pzErr ){ DbdataTable *pTab = 0; int rc = sqlite3_declare_vtab(db, pAux ? DBPTR_SCHEMA : DBDATA_SCHEMA); if( rc==SQLITE_OK ){ pTab = (DbdataTable*)sqlite3_malloc64(sizeof(DbdataTable)); if( pTab==0 ){ rc = SQLITE_NOMEM; }else{ memset(pTab, 0, sizeof(DbdataTable)); pTab->db = db; pTab->bPtr = (pAux!=0); } } *ppVtab = (sqlite3_vtab*)pTab; return rc; } /* ** Disconnect from or destroy a sqlite_dbdata or sqlite_dbptr virtual table. */ static int dbdataDisconnect(sqlite3_vtab *pVtab){ DbdataTable *pTab = (DbdataTable*)pVtab; if( pTab ){ sqlite3_finalize(pTab->pStmt); sqlite3_free(pVtab); } return SQLITE_OK; } /* ** This function interprets two types of constraints: ** ** schema=? ** pgno=? ** ** If neither are present, idxNum is set to 0. If schema=? is present, ** the 0x01 bit in idxNum is set. If pgno=? is present, the 0x02 bit ** in idxNum is set. ** ** If both parameters are present, schema is in position 0 and pgno in ** position 1. */ static int dbdataBestIndex(sqlite3_vtab *tab, sqlite3_index_info *pIdx){ DbdataTable *pTab = (DbdataTable*)tab; int i; int iSchema = -1; int iPgno = -1; int colSchema = (pTab->bPtr ? DBPTR_COLUMN_SCHEMA : DBDATA_COLUMN_SCHEMA); for(i=0; i<pIdx->nConstraint; i++){ struct sqlite3_index_constraint *p = &pIdx->aConstraint[i]; if( p->op==SQLITE_INDEX_CONSTRAINT_EQ ){ if( p->iColumn==colSchema ){ if( p->usable==0 ) return SQLITE_CONSTRAINT; iSchema = i; } if( p->iColumn==DBDATA_COLUMN_PGNO && p->usable ){ iPgno = i; } } } if( iSchema>=0 ){ pIdx->aConstraintUsage[iSchema].argvIndex = 1; pIdx->aConstraintUsage[iSchema].omit = 1; } if( iPgno>=0 ){ pIdx->aConstraintUsage[iPgno].argvIndex = 1 + (iSchema>=0); pIdx->aConstraintUsage[iPgno].omit = 1; pIdx->estimatedCost = 100; pIdx->estimatedRows = 50; if( pTab->bPtr==0 && pIdx->nOrderBy && pIdx->aOrderBy[0].desc==0 ){ int iCol = pIdx->aOrderBy[0].iColumn; if( pIdx->nOrderBy==1 ){ pIdx->orderByConsumed = (iCol==0 || iCol==1); }else if( pIdx->nOrderBy==2 && pIdx->aOrderBy[1].desc==0 && iCol==0 ){ pIdx->orderByConsumed = (pIdx->aOrderBy[1].iColumn==1); } } }else{ pIdx->estimatedCost = 100000000; pIdx->estimatedRows = 1000000000; } pIdx->idxNum = (iSchema>=0 ? 0x01 : 0x00) | (iPgno>=0 ? 0x02 : 0x00); return SQLITE_OK; } /* ** Open a new sqlite_dbdata or sqlite_dbptr cursor. */ static int dbdataOpen(sqlite3_vtab *pVTab, sqlite3_vtab_cursor **ppCursor){ DbdataCursor *pCsr; pCsr = (DbdataCursor*)sqlite3_malloc64(sizeof(DbdataCursor)); if( pCsr==0 ){ return SQLITE_NOMEM; }else{ memset(pCsr, 0, sizeof(DbdataCursor)); pCsr->base.pVtab = pVTab; } *ppCursor = (sqlite3_vtab_cursor *)pCsr; return SQLITE_OK; } /* ** Restore a cursor object to the state it was in when first allocated ** by dbdataOpen(). */ static void dbdataResetCursor(DbdataCursor *pCsr){ DbdataTable *pTab = (DbdataTable*)(pCsr->base.pVtab); if( pTab->pStmt==0 ){ pTab->pStmt = pCsr->pStmt; }else{ sqlite3_finalize(pCsr->pStmt); } pCsr->pStmt = 0; pCsr->iPgno = 1; pCsr->iCell = 0; pCsr->iField = 0; pCsr->bOnePage = 0; } /* ** Close an sqlite_dbdata or sqlite_dbptr cursor. */ static int dbdataClose(sqlite3_vtab_cursor *pCursor){ DbdataCursor *pCsr = (DbdataCursor*)pCursor; dbdataResetCursor(pCsr); sqlite3_free(pCsr); return SQLITE_OK; } /* ** Utility methods to decode 16 and 32-bit big-endian unsigned integers. */ static unsigned int get_uint16(unsigned char *a){ return (a[0]<<8)|a[1]; } static unsigned int get_uint32(unsigned char *a){ return (a[0]<<24)|(a[1]<<16)|(a[2]<<8)|a[3]; } /* ** Load page pgno from the database via the sqlite_dbpage virtual table. ** If successful, set (*ppPage) to point to a buffer containing the page ** data, (*pnPage) to the size of that buffer in bytes and return ** SQLITE_OK. In this case it is the responsibility of the caller to ** eventually free the buffer using sqlite3_free(). ** ** Or, if an error occurs, set both (*ppPage) and (*pnPage) to 0 and ** return an SQLite error code. */ static int dbdataLoadPage( DbdataCursor *pCsr, /* Cursor object */ unsigned int pgno, /* Page number of page to load */ u8 **ppPage, /* OUT: pointer to page buffer */ int *pnPage /* OUT: Size of (*ppPage) in bytes */ ){ int rc2; int rc = SQLITE_OK; sqlite3_stmt *pStmt = pCsr->pStmt; *ppPage = 0; *pnPage = 0; sqlite3_bind_int64(pStmt, 2, pgno); if( SQLITE_ROW==sqlite3_step(pStmt) ){ int nCopy = sqlite3_column_bytes(pStmt, 0); if( nCopy>0 ){ u8 *pPage; pPage = (u8*)sqlite3_malloc64(nCopy); if( pPage==0 ){ rc = SQLITE_NOMEM; }else{ const u8 *pCopy = sqlite3_column_blob(pStmt, 0); memcpy(pPage, pCopy, nCopy); } *ppPage = pPage; *pnPage = nCopy; } } rc2 = sqlite3_reset(pStmt); if( rc==SQLITE_OK ) rc = rc2; return rc; } /* ** Read a varint. Put the value in *pVal and return the number of bytes. */ static int dbdataGetVarint(const u8 *z, sqlite3_int64 *pVal){ sqlite3_int64 v = 0; int i; for(i=0; i<8; i++){ v = (v<<7) + (z[i]&0x7f); if( (z[i]&0x80)==0 ){ *pVal = v; return i+1; } } v = (v<<8) + (z[i]&0xff); *pVal = v; return 9; } /* ** Return the number of bytes of space used by an SQLite value of type ** eType. */ static int dbdataValueBytes(int eType){ switch( eType ){ case 0: case 8: case 9: case 10: case 11: return 0; case 1: return 1; case 2: return 2; case 3: return 3; case 4: return 4; case 5: return 6; case 6: case 7: return 8; default: return ((eType-12) / 2); } } /* ** Load a value of type eType from buffer pData and use it to set the ** result of context object pCtx. */ static void dbdataValue(sqlite3_context *pCtx, int eType, u8 *pData){ switch( eType ){ case 0: case 10: case 11: sqlite3_result_null(pCtx); break; case 8: sqlite3_result_int(pCtx, 0); break; case 9: sqlite3_result_int(pCtx, 1); break; case 1: case 2: case 3: case 4: case 5: case 6: case 7: { sqlite3_uint64 v = (signed char)pData[0]; pData++; switch( eType ){ case 7: case 6: v = (v<<16) + (pData[0]<<8) + pData[1]; pData += 2; case 5: v = (v<<16) + (pData[0]<<8) + pData[1]; pData += 2; case 4: v = (v<<8) + pData[0]; pData++; case 3: v = (v<<8) + pData[0]; pData++; case 2: v = (v<<8) + pData[0]; pData++; } if( eType==7 ){ double r; memcpy(&r, &v, sizeof(r)); sqlite3_result_double(pCtx, r); }else{ sqlite3_result_int64(pCtx, (sqlite3_int64)v); } break; } default: { int n = ((eType-12) / 2); if( eType % 2 ){ sqlite3_result_text(pCtx, (const char*)pData, n, SQLITE_TRANSIENT); }else{ sqlite3_result_blob(pCtx, pData, n, SQLITE_TRANSIENT); } } } } /* ** Move an sqlite_dbdata or sqlite_dbptr cursor to the next entry. */ static int dbdataNext(sqlite3_vtab_cursor *pCursor){ DbdataCursor *pCsr = (DbdataCursor*)pCursor; DbdataTable *pTab = (DbdataTable*)pCursor->pVtab; pCsr->iRowid++; while( 1 ){ int rc; int iOff = (pCsr->iPgno==1 ? 100 : 0); if( pCsr->aPage==0 ){ while( 1 ){ if( pCsr->bOnePage==0 && pCsr->iPgno>pCsr->szDb ) return SQLITE_OK; rc = dbdataLoadPage(pCsr, pCsr->iPgno, &pCsr->aPage, &pCsr->nPage); if( rc!=SQLITE_OK ) return rc; if( pCsr->aPage ) break; pCsr->iPgno++; } pCsr->iCell = pTab->bPtr ? -2 : 0; pCsr->nCell = get_uint16(&pCsr->aPage[iOff+3]); } if( pTab->bPtr ){ if( pCsr->aPage[iOff]!=0x02 && pCsr->aPage[iOff]!=0x05 ){ pCsr->iCell = pCsr->nCell; } pCsr->iCell++; if( pCsr->iCell>=pCsr->nCell ){ sqlite3_free(pCsr->aPage); pCsr->aPage = 0; if( pCsr->bOnePage ) return SQLITE_OK; pCsr->iPgno++; }else{ return SQLITE_OK; } }else{ /* If there is no record loaded, load it now. */ if( pCsr->pRec==0 ){ int bHasRowid = 0; int nPointer = 0; sqlite3_int64 nPayload = 0; sqlite3_int64 nHdr = 0; int iHdr; int U, X; int nLocal; switch( pCsr->aPage[iOff] ){ case 0x02: nPointer = 4; break; case 0x0a: break; case 0x0d: bHasRowid = 1; break; default: /* This is not a b-tree page with records on it. Continue. */ pCsr->iCell = pCsr->nCell; break; } if( pCsr->iCell>=pCsr->nCell ){ sqlite3_free(pCsr->aPage); pCsr->aPage = 0; if( pCsr->bOnePage ) return SQLITE_OK; pCsr->iPgno++; continue; } iOff += 8 + nPointer + pCsr->iCell*2; iOff = get_uint16(&pCsr->aPage[iOff]); /* For an interior node cell, skip past the child-page number */ iOff += nPointer; /* Load the "byte of payload including overflow" field */ iOff += dbdataGetVarint(&pCsr->aPage[iOff], &nPayload); /* If this is a leaf intkey cell, load the rowid */ if( bHasRowid ){ iOff += dbdataGetVarint(&pCsr->aPage[iOff], &pCsr->iIntkey); } /* Allocate space for payload */ pCsr->pRec = (u8*)sqlite3_malloc64(nPayload); if( pCsr->pRec==0 ) return SQLITE_NOMEM; pCsr->nRec = nPayload; U = pCsr->nPage; if( bHasRowid ){ X = U-35; }else{ X = ((U-12)*64/255)-23; } if( nPayload<=X ){ nLocal = nPayload; }else{ int M, K; M = ((U-12)*32/255)-23; K = M+((nPayload-M)%(U-4)); if( K<=X ){ nLocal = K; }else{ nLocal = M; } } /* Load the nLocal bytes of payload */ memcpy(pCsr->pRec, &pCsr->aPage[iOff], nLocal); iOff += nLocal; /* Load content from overflow pages */ if( nPayload>nLocal ){ sqlite3_int64 nRem = nPayload - nLocal; unsigned int pgnoOvfl = get_uint32(&pCsr->aPage[iOff]); while( nRem>0 ){ u8 *aOvfl = 0; int nOvfl = 0; int nCopy; rc = dbdataLoadPage(pCsr, pgnoOvfl, &aOvfl, &nOvfl); assert( rc!=SQLITE_OK || nOvfl==pCsr->nPage ); if( rc!=SQLITE_OK ) return rc; nCopy = U-4; if( nCopy>nRem ) nCopy = nRem; memcpy(&pCsr->pRec[nPayload-nRem], &aOvfl[4], nCopy); nRem -= nCopy; pgnoOvfl = get_uint32(aOvfl); sqlite3_free(aOvfl); } } iHdr = dbdataGetVarint(pCsr->pRec, &nHdr); pCsr->nHdr = nHdr; pCsr->pHdrPtr = &pCsr->pRec[iHdr]; pCsr->pPtr = &pCsr->pRec[pCsr->nHdr]; pCsr->iField = (bHasRowid ? -1 : 0); }else{ pCsr->iField++; if( pCsr->iField>0 ){ sqlite3_int64 iType; pCsr->pHdrPtr += dbdataGetVarint(pCsr->pHdrPtr, &iType); pCsr->pPtr += dbdataValueBytes(iType); } } if( pCsr->iField<0 || pCsr->pHdrPtr<&pCsr->pRec[pCsr->nHdr] ){ return SQLITE_OK; } /* Advance to the next cell. The next iteration of the loop will load ** the record and so on. */ sqlite3_free(pCsr->pRec); pCsr->pRec = 0; pCsr->iCell++; } } assert( !"can't get here" ); return SQLITE_OK; } /* ** Return true if the cursor is at EOF. */ static int dbdataEof(sqlite3_vtab_cursor *pCursor){ DbdataCursor *pCsr = (DbdataCursor*)pCursor; return pCsr->aPage==0; } /* ** Determine the size in pages of database zSchema (where zSchema is ** "main", "temp" or the name of an attached database) and set ** pCsr->szDb accordingly. If successful, return SQLITE_OK. Otherwise, ** an SQLite error code. */ static int dbdataDbsize(DbdataCursor *pCsr, const char *zSchema){ DbdataTable *pTab = (DbdataTable*)pCsr->base.pVtab; char *zSql = 0; int rc, rc2; sqlite3_stmt *pStmt = 0; zSql = sqlite3_mprintf("PRAGMA %Q.page_count", zSchema); if( zSql==0 ) return SQLITE_NOMEM; rc = sqlite3_prepare_v2(pTab->db, zSql, -1, &pStmt, 0); sqlite3_free(zSql); if( rc==SQLITE_OK && sqlite3_step(pStmt)==SQLITE_ROW ){ pCsr->szDb = sqlite3_column_int(pStmt, 0); } rc2 = sqlite3_finalize(pStmt); if( rc==SQLITE_OK ) rc = rc2; return rc; } /* ** xFilter method for sqlite_dbdata and sqlite_dbptr. */ static int dbdataFilter( sqlite3_vtab_cursor *pCursor, int idxNum, const char *idxStr, int argc, sqlite3_value **argv ){ DbdataCursor *pCsr = (DbdataCursor*)pCursor; DbdataTable *pTab = (DbdataTable*)pCursor->pVtab; int rc = SQLITE_OK; const char *zSchema = "main"; dbdataResetCursor(pCsr); assert( pCsr->iPgno==1 ); if( idxNum & 0x01 ){ zSchema = (const char*)sqlite3_value_text(argv[0]); } if( idxNum & 0x02 ){ pCsr->iPgno = sqlite3_value_int(argv[(idxNum & 0x01)]); pCsr->bOnePage = 1; }else{ pCsr->nPage = dbdataDbsize(pCsr, zSchema); rc = dbdataDbsize(pCsr, zSchema); } if( rc==SQLITE_OK ){ if( pTab->pStmt ){ pCsr->pStmt = pTab->pStmt; pTab->pStmt = 0; }else{ rc = sqlite3_prepare_v2(pTab->db, "SELECT data FROM sqlite_dbpage(?) WHERE pgno=?", -1, &pCsr->pStmt, 0 ); } } if( rc==SQLITE_OK ){ rc = sqlite3_bind_text(pCsr->pStmt, 1, zSchema, -1, SQLITE_TRANSIENT); }else{ pTab->base.zErrMsg = sqlite3_mprintf("%s", sqlite3_errmsg(pTab->db)); } if( rc==SQLITE_OK ){ rc = dbdataNext(pCursor); } return rc; } /* ** Return a column for the sqlite_dbdata or sqlite_dbptr table. */ static int dbdataColumn( sqlite3_vtab_cursor *pCursor, sqlite3_context *ctx, int i ){ DbdataCursor *pCsr = (DbdataCursor*)pCursor; DbdataTable *pTab = (DbdataTable*)pCursor->pVtab; if( pTab->bPtr ){ switch( i ){ case DBPTR_COLUMN_PGNO: sqlite3_result_int64(ctx, pCsr->iPgno); break; case DBPTR_COLUMN_CHILD: { int iOff = pCsr->iPgno==1 ? 100 : 0; if( pCsr->iCell<0 ){ iOff += 8; }else{ iOff += 12 + pCsr->iCell*2; iOff = get_uint16(&pCsr->aPage[iOff]); } sqlite3_result_int64(ctx, get_uint32(&pCsr->aPage[iOff])); break; } } }else{ switch( i ){ case DBDATA_COLUMN_PGNO: sqlite3_result_int64(ctx, pCsr->iPgno); break; case DBDATA_COLUMN_CELL: sqlite3_result_int(ctx, pCsr->iCell); break; case DBDATA_COLUMN_FIELD: sqlite3_result_int(ctx, pCsr->iField); break; case DBDATA_COLUMN_VALUE: { if( pCsr->iField<0 ){ sqlite3_result_int64(ctx, pCsr->iIntkey); }else{ sqlite3_int64 iType; dbdataGetVarint(pCsr->pHdrPtr, &iType); dbdataValue(ctx, iType, pCsr->pPtr); } break; } } } return SQLITE_OK; } /* ** Return the rowid for an sqlite_dbdata or sqlite_dptr table. */ static int dbdataRowid(sqlite3_vtab_cursor *pCursor, sqlite_int64 *pRowid){ DbdataCursor *pCsr = (DbdataCursor*)pCursor; *pRowid = pCsr->iRowid; return SQLITE_OK; } /* ** Invoke this routine to register the "sqlite_dbdata" virtual table module */ static int sqlite3DbdataRegister(sqlite3 *db){ static sqlite3_module dbdata_module = { 0, /* iVersion */ 0, /* xCreate */ dbdataConnect, /* xConnect */ dbdataBestIndex, /* xBestIndex */ dbdataDisconnect, /* xDisconnect */ 0, /* xDestroy */ dbdataOpen, /* xOpen - open a cursor */ dbdataClose, /* xClose - close a cursor */ dbdataFilter, /* xFilter - configure scan constraints */ dbdataNext, /* xNext - advance a cursor */ dbdataEof, /* xEof - check for end of scan */ dbdataColumn, /* xColumn - read data */ dbdataRowid, /* xRowid - read data */ 0, /* xUpdate */ 0, /* xBegin */ 0, /* xSync */ 0, /* xCommit */ 0, /* xRollback */ 0, /* xFindMethod */ 0, /* xRename */ 0, /* xSavepoint */ 0, /* xRelease */ 0, /* xRollbackTo */ 0 /* xShadowName */ }; int rc = sqlite3_create_module(db, "sqlite_dbdata", &dbdata_module, 0); if( rc==SQLITE_OK ){ rc = sqlite3_create_module(db, "sqlite_dbptr", &dbdata_module, (void*)1); } return rc; } #ifdef _WIN32 __declspec(dllexport) #endif int sqlite3_dbdata_init( sqlite3 *db, char **pzErrMsg, const sqlite3_api_routines *pApi ){ SQLITE_EXTENSION_INIT2(pApi); return sqlite3DbdataRegister(db); } |
Changes to main.mk.
︙ | ︙ | |||
734 735 736 737 738 739 740 741 742 743 744 745 746 747 | $(TOP)/ext/misc/fileio.c \ $(TOP)/ext/misc/completion.c \ $(TOP)/ext/misc/sqlar.c \ $(TOP)/ext/expert/sqlite3expert.c \ $(TOP)/ext/expert/sqlite3expert.h \ $(TOP)/ext/misc/zipfile.c \ $(TOP)/ext/misc/memtrace.c \ $(TOP)/src/test_windirent.c shell.c: $(SHELL_SRC) $(TOP)/tool/mkshellc.tcl tclsh $(TOP)/tool/mkshellc.tcl >shell.c | > | 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 | $(TOP)/ext/misc/fileio.c \ $(TOP)/ext/misc/completion.c \ $(TOP)/ext/misc/sqlar.c \ $(TOP)/ext/expert/sqlite3expert.c \ $(TOP)/ext/expert/sqlite3expert.h \ $(TOP)/ext/misc/zipfile.c \ $(TOP)/ext/misc/memtrace.c \ $(TOP)/ext/misc/dbdata.c \ $(TOP)/src/test_windirent.c shell.c: $(SHELL_SRC) $(TOP)/tool/mkshellc.tcl tclsh $(TOP)/tool/mkshellc.tcl >shell.c |
︙ | ︙ |
Changes to src/shell.c.in.
︙ | ︙ | |||
943 944 945 946 947 948 949 950 951 952 953 954 955 956 | INCLUDE ../ext/misc/memtrace.c #ifdef SQLITE_HAVE_ZLIB INCLUDE ../ext/misc/zipfile.c INCLUDE ../ext/misc/sqlar.c #endif INCLUDE ../ext/expert/sqlite3expert.h INCLUDE ../ext/expert/sqlite3expert.c #if defined(SQLITE_ENABLE_SESSION) /* ** State information for a single open session */ typedef struct OpenSession OpenSession; struct OpenSession { | > > > > | 943 944 945 946 947 948 949 950 951 952 953 954 955 956 957 958 959 960 | INCLUDE ../ext/misc/memtrace.c #ifdef SQLITE_HAVE_ZLIB INCLUDE ../ext/misc/zipfile.c INCLUDE ../ext/misc/sqlar.c #endif INCLUDE ../ext/expert/sqlite3expert.h INCLUDE ../ext/expert/sqlite3expert.c #if !defined(SQLITE_OMIT_VIRTUALTABLE) && defined(SQLITE_ENABLE_DBPAGE_VTAB) INCLUDE ../ext/misc/dbdata.c #endif #if defined(SQLITE_ENABLE_SESSION) /* ** State information for a single open session */ typedef struct OpenSession OpenSession; struct OpenSession { |
︙ | ︙ | |||
3570 3571 3572 3573 3574 3575 3576 3577 3578 3579 3580 3581 3582 3583 | " --once Do no more than one progress interrupt", " --quiet|-q No output except at interrupts", " --reset Reset the count for each input and interrupt", #endif ".prompt MAIN CONTINUE Replace the standard prompts", ".quit Exit this program", ".read FILE Read input from FILE", ".restore ?DB? FILE Restore content of DB (default \"main\") from FILE", ".save FILE Write in-memory database into FILE", ".scanstats on|off Turn sqlite3_stmt_scanstatus() metrics on or off", ".schema ?PATTERN? Show the CREATE statements matching PATTERN", " Options:", " --indent Try to pretty-print the schema", ".selftest ?OPTIONS? Run tests defined in the SELFTEST table", | > > > | 3574 3575 3576 3577 3578 3579 3580 3581 3582 3583 3584 3585 3586 3587 3588 3589 3590 | " --once Do no more than one progress interrupt", " --quiet|-q No output except at interrupts", " --reset Reset the count for each input and interrupt", #endif ".prompt MAIN CONTINUE Replace the standard prompts", ".quit Exit this program", ".read FILE Read input from FILE", #if !defined(SQLITE_OMIT_VIRTUALTABLE) && defined(SQLITE_ENABLE_DBPAGE_VTAB) ".recover Recover as much data as possible from corrupt db.", #endif ".restore ?DB? FILE Restore content of DB (default \"main\") from FILE", ".save FILE Write in-memory database into FILE", ".scanstats on|off Turn sqlite3_stmt_scanstatus() metrics on or off", ".schema ?PATTERN? Show the CREATE statements matching PATTERN", " Options:", " --indent Try to pretty-print the schema", ".selftest ?OPTIONS? Run tests defined in the SELFTEST table", |
︙ | ︙ | |||
3926 3927 3928 3929 3930 3931 3932 3933 3934 3935 3936 3937 3938 3939 | p->lineno = nLine; } sqlite3_free(a); utf8_printf(stderr,"Error on line %d of --hexdb input\n", nLine); return 0; } #endif /* SQLITE_ENABLE_DESERIALIZE */ /* Flags for open_db(). ** ** The default behavior of open_db() is to exit(1) if the database fails to ** open. The OPEN_DB_KEEPALIVE flag changes that so that it prints an error ** but still returns without calling exit. ** | > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > | 3933 3934 3935 3936 3937 3938 3939 3940 3941 3942 3943 3944 3945 3946 3947 3948 3949 3950 3951 3952 3953 3954 3955 3956 3957 3958 3959 3960 3961 3962 3963 3964 3965 3966 3967 3968 3969 3970 3971 3972 3973 3974 3975 3976 3977 3978 3979 3980 3981 3982 3983 3984 3985 3986 3987 3988 3989 3990 3991 3992 3993 3994 3995 3996 3997 3998 3999 4000 4001 4002 4003 4004 4005 4006 4007 4008 4009 4010 4011 4012 4013 4014 4015 4016 4017 4018 4019 4020 4021 4022 4023 4024 4025 4026 4027 4028 4029 4030 4031 4032 4033 4034 4035 4036 4037 4038 4039 4040 4041 4042 4043 4044 4045 4046 4047 4048 4049 4050 4051 4052 4053 4054 4055 4056 4057 4058 4059 4060 4061 4062 4063 4064 4065 | p->lineno = nLine; } sqlite3_free(a); utf8_printf(stderr,"Error on line %d of --hexdb input\n", nLine); return 0; } #endif /* SQLITE_ENABLE_DESERIALIZE */ /* ** Scalar function "shell_int32". The first argument to this function ** must be a blob. The second a non-negative integer. This function ** reads and returns a 32-bit big-endian integer from byte ** offset (4*<arg2>) of the blob. */ static void shellInt32( sqlite3_context *context, int argc, sqlite3_value **argv ){ const unsigned char *pBlob; int nBlob; int iInt; nBlob = sqlite3_value_bytes(argv[0]); pBlob = (const unsigned char*)sqlite3_value_blob(argv[0]); iInt = sqlite3_value_int(argv[1]); if( iInt>=0 && (iInt+1)*4<=nBlob ){ const unsigned char *a = &pBlob[iInt*4]; sqlite3_int64 iVal = ((sqlite3_int64)a[0]<<24) + ((sqlite3_int64)a[1]<<16) + ((sqlite3_int64)a[2]<< 8) + ((sqlite3_int64)a[3]<< 0); sqlite3_result_int64(context, iVal); } } /* ** Scalar function "shell_escape_crnl" used by the .recover command. ** The argument passed to this function is the output of built-in ** function quote(). If the first character of the input is "'", ** indicating that the value passed to quote() was a text value, ** then this function searches the input for "\n" and "\r" characters ** and adds a wrapper similar to the following: ** ** replace(replace(<input>, '\n', char(10), '\r', char(13)); ** ** Or, if the first character of the input is not "'", then a copy ** of the input is returned. */ static void shellEscapeCrnl( sqlite3_context *context, int argc, sqlite3_value **argv ){ const char *zText = (const char*)sqlite3_value_text(argv[0]); if( zText[0]=='\'' ){ int nText = sqlite3_value_bytes(argv[0]); int i; char zBuf1[20]; char zBuf2[20]; const char *zNL = 0; const char *zCR = 0; int nCR = 0; int nNL = 0; for(i=0; zText[i]; i++){ if( zNL==0 && zText[i]=='\n' ){ zNL = unused_string(zText, "\\n", "\\012", zBuf1); nNL = (int)strlen(zNL); } if( zCR==0 && zText[i]=='\r' ){ zCR = unused_string(zText, "\\r", "\\015", zBuf2); nCR = (int)strlen(zCR); } } if( zNL || zCR ){ int iOut = 0; i64 nMax = (nNL > nCR) ? nNL : nCR; i64 nAlloc = nMax * nText + (nMax+12)*2; char *zOut = (char*)sqlite3_malloc64(nAlloc); if( zOut==0 ){ sqlite3_result_error_nomem(context); return; } if( zNL && zCR ){ memcpy(&zOut[iOut], "replace(replace(", 16); iOut += 16; }else{ memcpy(&zOut[iOut], "replace(", 8); iOut += 8; } for(i=0; zText[i]; i++){ if( zText[i]=='\n' ){ memcpy(&zOut[iOut], zNL, nNL); iOut += nNL; }else if( zText[i]=='\r' ){ memcpy(&zOut[iOut], zCR, nCR); iOut += nCR; }else{ zOut[iOut] = zText[i]; iOut++; } } if( zNL ){ memcpy(&zOut[iOut], ",'", 2); iOut += 2; memcpy(&zOut[iOut], zNL, nNL); iOut += nNL; memcpy(&zOut[iOut], "', char(10))", 12); iOut += 12; } if( zCR ){ memcpy(&zOut[iOut], ",'", 2); iOut += 2; memcpy(&zOut[iOut], zCR, nCR); iOut += nCR; memcpy(&zOut[iOut], "', char(13))", 12); iOut += 12; } sqlite3_result_text(context, zOut, iOut, SQLITE_TRANSIENT); sqlite3_free(zOut); return; } } sqlite3_result_value(context, argv[0]); } /* Flags for open_db(). ** ** The default behavior of open_db() is to exit(1) if the database fails to ** open. The OPEN_DB_KEEPALIVE flag changes that so that it prints an error ** but still returns without calling exit. ** |
︙ | ︙ | |||
3995 3996 3997 3998 3999 4000 4001 4002 4003 4004 4005 4006 4007 4008 4009 4010 4011 4012 4013 4014 4015 4016 4017 4018 | } #ifndef SQLITE_OMIT_LOAD_EXTENSION sqlite3_enable_load_extension(p->db, 1); #endif sqlite3_fileio_init(p->db, 0, 0); sqlite3_shathree_init(p->db, 0, 0); sqlite3_completion_init(p->db, 0, 0); #ifdef SQLITE_HAVE_ZLIB sqlite3_zipfile_init(p->db, 0, 0); sqlite3_sqlar_init(p->db, 0, 0); #endif sqlite3_create_function(p->db, "shell_add_schema", 3, SQLITE_UTF8, 0, shellAddSchemaName, 0, 0); sqlite3_create_function(p->db, "shell_module_schema", 1, SQLITE_UTF8, 0, shellModuleSchema, 0, 0); sqlite3_create_function(p->db, "shell_putsnl", 1, SQLITE_UTF8, p, shellPutsFunc, 0, 0); #ifndef SQLITE_NOHAVE_SYSTEM sqlite3_create_function(p->db, "edit", 1, SQLITE_UTF8, 0, editFunc, 0, 0); sqlite3_create_function(p->db, "edit", 2, SQLITE_UTF8, 0, editFunc, 0, 0); #endif if( p->openMode==SHELL_OPEN_ZIPFILE ){ | > > > > > > > | 4121 4122 4123 4124 4125 4126 4127 4128 4129 4130 4131 4132 4133 4134 4135 4136 4137 4138 4139 4140 4141 4142 4143 4144 4145 4146 4147 4148 4149 4150 4151 | } #ifndef SQLITE_OMIT_LOAD_EXTENSION sqlite3_enable_load_extension(p->db, 1); #endif sqlite3_fileio_init(p->db, 0, 0); sqlite3_shathree_init(p->db, 0, 0); sqlite3_completion_init(p->db, 0, 0); #if !defined(SQLITE_OMIT_VIRTUALTABLE) && defined(SQLITE_ENABLE_DBPAGE_VTAB) sqlite3_dbdata_init(p->db, 0, 0); #endif #ifdef SQLITE_HAVE_ZLIB sqlite3_zipfile_init(p->db, 0, 0); sqlite3_sqlar_init(p->db, 0, 0); #endif sqlite3_create_function(p->db, "shell_add_schema", 3, SQLITE_UTF8, 0, shellAddSchemaName, 0, 0); sqlite3_create_function(p->db, "shell_module_schema", 1, SQLITE_UTF8, 0, shellModuleSchema, 0, 0); sqlite3_create_function(p->db, "shell_putsnl", 1, SQLITE_UTF8, p, shellPutsFunc, 0, 0); sqlite3_create_function(p->db, "shell_escape_crnl", 1, SQLITE_UTF8, 0, shellEscapeCrnl, 0, 0); sqlite3_create_function(p->db, "shell_int32", 2, SQLITE_UTF8, 0, shellInt32, 0, 0); #ifndef SQLITE_NOHAVE_SYSTEM sqlite3_create_function(p->db, "edit", 1, SQLITE_UTF8, 0, editFunc, 0, 0); sqlite3_create_function(p->db, "edit", 2, SQLITE_UTF8, 0, editFunc, 0, 0); #endif if( p->openMode==SHELL_OPEN_ZIPFILE ){ |
︙ | ︙ | |||
5259 5260 5261 5262 5263 5264 5265 | usage: raw_printf(stderr, "Usage %s sub-command ?switches...?\n", azArg[0]); raw_printf(stderr, "Where sub-commands are:\n"); raw_printf(stderr, " fkey-indexes\n"); return SQLITE_ERROR; } | | < < < | 5392 5393 5394 5395 5396 5397 5398 5399 5400 5401 5402 5403 5404 5405 5406 | usage: raw_printf(stderr, "Usage %s sub-command ?switches...?\n", azArg[0]); raw_printf(stderr, "Where sub-commands are:\n"); raw_printf(stderr, " fkey-indexes\n"); return SQLITE_ERROR; } #if !defined SQLITE_OMIT_VIRTUALTABLE static void shellPrepare( sqlite3 *db, int *pRc, const char *zSql, sqlite3_stmt **ppStmt ){ *ppStmt = 0; |
︙ | ︙ | |||
5333 5334 5335 5336 5337 5338 5339 5340 5341 5342 5343 5344 5345 5346 | if( rc!=SQLITE_OK ){ sqlite3 *db = sqlite3_db_handle(pStmt); raw_printf(stderr, "SQL error: %s\n", sqlite3_errmsg(db)); } *pRc = rc; } } /* ** Structure representing a single ".ar" command. */ typedef struct ArCommand ArCommand; struct ArCommand { u8 eCmd; /* An AR_CMD_* value */ u8 bVerbose; /* True if --verbose */ | > > > > > > | 5463 5464 5465 5466 5467 5468 5469 5470 5471 5472 5473 5474 5475 5476 5477 5478 5479 5480 5481 5482 | if( rc!=SQLITE_OK ){ sqlite3 *db = sqlite3_db_handle(pStmt); raw_printf(stderr, "SQL error: %s\n", sqlite3_errmsg(db)); } *pRc = rc; } } #endif /* !defined SQLITE_OMIT_VIRTUALTABLE */ #if !defined(SQLITE_OMIT_VIRTUALTABLE) && defined(SQLITE_HAVE_ZLIB) /********************************************************************************* ** The ".archive" or ".ar" command. */ /* ** Structure representing a single ".ar" command. */ typedef struct ArCommand ArCommand; struct ArCommand { u8 eCmd; /* An AR_CMD_* value */ u8 bVerbose; /* True if --verbose */ |
︙ | ︙ | |||
6022 6023 6024 6025 6026 6027 6028 6029 6030 6031 6032 6033 6034 6035 | return rc; } /* End of the ".archive" or ".ar" command logic **********************************************************************************/ #endif /* !defined(SQLITE_OMIT_VIRTUALTABLE) && defined(SQLITE_HAVE_ZLIB) */ /* ** If an input line begins with "." then invoke this routine to ** process that line. ** ** Return 1 on error, 2 to exit, and 0 otherwise. */ | > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > | 6158 6159 6160 6161 6162 6163 6164 6165 6166 6167 6168 6169 6170 6171 6172 6173 6174 6175 6176 6177 6178 6179 6180 6181 6182 6183 6184 6185 6186 6187 6188 6189 6190 6191 6192 6193 6194 6195 6196 6197 6198 6199 6200 6201 6202 6203 6204 6205 6206 6207 6208 6209 6210 6211 6212 6213 6214 6215 6216 6217 6218 6219 6220 6221 6222 6223 6224 6225 6226 6227 6228 6229 6230 6231 6232 6233 6234 6235 6236 6237 6238 6239 6240 6241 6242 6243 6244 6245 6246 6247 6248 6249 6250 6251 6252 6253 6254 6255 6256 6257 6258 6259 6260 6261 6262 6263 6264 6265 6266 6267 6268 6269 6270 6271 6272 6273 6274 6275 6276 6277 6278 6279 6280 6281 6282 6283 6284 6285 6286 6287 6288 6289 6290 6291 6292 6293 6294 6295 6296 6297 6298 6299 6300 6301 6302 6303 6304 6305 6306 6307 6308 6309 6310 6311 6312 6313 6314 6315 6316 6317 6318 6319 6320 6321 6322 6323 6324 6325 6326 6327 6328 6329 6330 6331 6332 6333 6334 6335 6336 6337 6338 6339 6340 6341 6342 6343 6344 6345 6346 6347 6348 6349 6350 6351 6352 6353 6354 6355 6356 6357 6358 6359 6360 6361 6362 6363 6364 6365 6366 6367 6368 6369 6370 6371 6372 6373 6374 6375 6376 6377 6378 6379 6380 6381 6382 6383 6384 6385 6386 6387 6388 6389 6390 6391 6392 6393 6394 6395 6396 6397 6398 6399 6400 6401 6402 6403 6404 6405 6406 6407 6408 6409 6410 6411 6412 6413 6414 6415 6416 6417 6418 6419 6420 6421 6422 6423 6424 6425 6426 6427 6428 6429 6430 6431 6432 6433 6434 6435 6436 6437 6438 6439 6440 6441 6442 6443 6444 6445 6446 6447 6448 6449 6450 6451 6452 6453 6454 6455 6456 6457 6458 6459 6460 6461 6462 6463 6464 6465 6466 6467 6468 6469 6470 6471 6472 6473 6474 6475 6476 6477 6478 6479 6480 6481 6482 6483 6484 6485 6486 6487 6488 6489 6490 6491 6492 6493 6494 6495 6496 6497 6498 6499 6500 6501 6502 6503 6504 6505 6506 6507 6508 6509 6510 6511 6512 6513 6514 6515 6516 6517 6518 6519 6520 6521 6522 6523 6524 6525 6526 6527 6528 6529 6530 6531 6532 6533 6534 6535 6536 6537 6538 6539 6540 6541 6542 6543 6544 6545 6546 6547 6548 6549 6550 6551 6552 6553 6554 6555 6556 6557 6558 6559 6560 6561 6562 6563 6564 6565 6566 6567 6568 6569 6570 6571 6572 6573 6574 6575 6576 6577 6578 6579 6580 6581 6582 6583 6584 6585 6586 6587 6588 6589 6590 6591 6592 6593 6594 6595 6596 6597 6598 6599 6600 6601 6602 6603 6604 6605 6606 6607 6608 6609 6610 6611 6612 6613 6614 6615 6616 6617 6618 6619 6620 6621 6622 6623 6624 6625 6626 6627 6628 6629 6630 6631 6632 6633 6634 6635 6636 6637 6638 6639 6640 6641 6642 6643 6644 6645 6646 6647 6648 6649 6650 6651 6652 6653 6654 6655 6656 6657 6658 6659 6660 6661 6662 6663 6664 6665 6666 6667 6668 6669 6670 6671 6672 6673 6674 6675 6676 6677 6678 6679 6680 6681 6682 6683 6684 6685 6686 6687 6688 6689 6690 6691 6692 6693 6694 6695 6696 6697 6698 6699 6700 6701 6702 6703 6704 6705 6706 6707 6708 6709 6710 6711 6712 6713 6714 6715 6716 6717 6718 6719 6720 6721 6722 6723 6724 6725 6726 6727 6728 6729 6730 6731 6732 6733 6734 6735 6736 6737 6738 6739 6740 6741 6742 6743 6744 6745 6746 6747 6748 6749 6750 6751 6752 6753 6754 6755 6756 6757 6758 6759 6760 6761 6762 6763 6764 6765 6766 6767 6768 6769 6770 6771 6772 6773 6774 6775 6776 6777 6778 6779 6780 6781 6782 6783 6784 6785 6786 6787 6788 6789 6790 6791 6792 6793 6794 6795 6796 | return rc; } /* End of the ".archive" or ".ar" command logic **********************************************************************************/ #endif /* !defined(SQLITE_OMIT_VIRTUALTABLE) && defined(SQLITE_HAVE_ZLIB) */ #if !defined(SQLITE_OMIT_VIRTUALTABLE) && defined(SQLITE_ENABLE_DBPAGE_VTAB) /* ** If (*pRc) is not SQLITE_OK when this function is called, it is a no-op. ** Otherwise, the SQL statement or statements in zSql are executed using ** database connection db and the error code written to *pRc before ** this function returns. */ static void shellExec(sqlite3 *db, int *pRc, const char *zSql){ int rc = *pRc; if( rc==SQLITE_OK ){ char *zErr = 0; rc = sqlite3_exec(db, zSql, 0, 0, &zErr); if( rc!=SQLITE_OK ){ raw_printf(stderr, "SQL error: %s\n", zErr); } *pRc = rc; } } /* ** Like shellExec(), except that zFmt is a printf() style format string. */ static void shellExecPrintf(sqlite3 *db, int *pRc, const char *zFmt, ...){ char *z = 0; if( *pRc==SQLITE_OK ){ va_list ap; va_start(ap, zFmt); z = sqlite3_vmprintf(zFmt, ap); va_end(ap); if( z==0 ){ *pRc = SQLITE_NOMEM; }else{ shellExec(db, pRc, z); } sqlite3_free(z); } } /* ** If *pRc is not SQLITE_OK when this function is called, it is a no-op. ** Otherwise, an attempt is made to allocate, zero and return a pointer ** to a buffer nByte bytes in size. If an OOM error occurs, *pRc is set ** to SQLITE_NOMEM and NULL returned. */ static void *shellMalloc(int *pRc, sqlite3_int64 nByte){ void *pRet = 0; if( *pRc==SQLITE_OK ){ pRet = sqlite3_malloc64(nByte); if( pRet==0 ){ *pRc = SQLITE_NOMEM; }else{ memset(pRet, 0, nByte); } } return pRet; } /* ** If *pRc is not SQLITE_OK when this function is called, it is a no-op. ** Otherwise, zFmt is treated as a printf() style string. The result of ** formatting it along with any trailing arguments is written into a ** buffer obtained from sqlite3_malloc(), and pointer to which is returned. ** It is the responsibility of the caller to eventually free this buffer ** using a call to sqlite3_free(). ** ** If an OOM error occurs, (*pRc) is set to SQLITE_NOMEM and a NULL ** pointer returned. */ static char *shellMPrintf(int *pRc, const char *zFmt, ...){ char *z = 0; if( *pRc==SQLITE_OK ){ va_list ap; va_start(ap, zFmt); z = sqlite3_vmprintf(zFmt, ap); va_end(ap); if( z==0 ){ *pRc = SQLITE_NOMEM; } } return z; } /* ** When running the ".recover" command, each output table, and the special ** orphaned row table if it is required, is represented by an instance ** of the following struct. */ typedef struct RecoverTable RecoverTable; struct RecoverTable { char *zQuoted; /* Quoted version of table name */ int nCol; /* Number of columns in table */ char **azlCol; /* Array of column lists */ int iPk; /* Index of IPK column */ }; /* ** Free a RecoverTable object allocated by recoverFindTable() or ** recoverOrphanTable(). */ static void recoverFreeTable(RecoverTable *pTab){ if( pTab ){ sqlite3_free(pTab->zQuoted); if( pTab->azlCol ){ int i; for(i=0; i<=pTab->nCol; i++){ sqlite3_free(pTab->azlCol[i]); } sqlite3_free(pTab->azlCol); } sqlite3_free(pTab); } } /* ** This function is a no-op if (*pRc) is not SQLITE_OK when it is called. ** Otherwise, it allocates and returns a RecoverTable object based on the ** final four arguments passed to this function. It is the responsibility ** of the caller to eventually free the returned object using ** recoverFreeTable(). */ static RecoverTable *recoverNewTable( int *pRc, /* IN/OUT: Error code */ const char *zName, /* Name of table */ const char *zSql, /* CREATE TABLE statement */ int bIntkey, int nCol ){ sqlite3 *dbtmp = 0; /* sqlite3 handle for testing CREATE TABLE */ int rc = *pRc; RecoverTable *pTab = 0; pTab = (RecoverTable*)shellMalloc(&rc, sizeof(RecoverTable)); if( rc==SQLITE_OK ){ int nSqlCol = 0; int bSqlIntkey = 0; sqlite3_stmt *pStmt = 0; rc = sqlite3_open("", &dbtmp); if( rc==SQLITE_OK ){ rc = sqlite3_exec(dbtmp, "PRAGMA writable_schema = on", 0, 0, 0); } if( rc==SQLITE_OK ){ rc = sqlite3_exec(dbtmp, zSql, 0, 0, 0); if( rc==SQLITE_ERROR ){ rc = SQLITE_OK; goto finished; } } shellPreparePrintf(dbtmp, &rc, &pStmt, "SELECT count(*) FROM pragma_table_info(%Q)", zName ); if( rc==SQLITE_OK && SQLITE_ROW==sqlite3_step(pStmt) ){ nSqlCol = sqlite3_column_int(pStmt, 0); } shellFinalize(&rc, pStmt); if( rc!=SQLITE_OK || nSqlCol<nCol ){ goto finished; } shellPreparePrintf(dbtmp, &rc, &pStmt, "SELECT (" " SELECT substr(data,1,1)==X'0D' FROM sqlite_dbpage WHERE pgno=rootpage" ") FROM sqlite_master WHERE name = %Q", zName ); if( rc==SQLITE_OK && SQLITE_ROW==sqlite3_step(pStmt) ){ bSqlIntkey = sqlite3_column_int(pStmt, 0); } shellFinalize(&rc, pStmt); if( bIntkey==bSqlIntkey ){ int i; const char *zPk = "_rowid_"; sqlite3_stmt *pPkFinder = 0; /* If this is an intkey table and there is an INTEGER PRIMARY KEY, ** set zPk to the name of the PK column, and pTab->iPk to the index ** of the column, where columns are 0-numbered from left to right. ** Or, if this is a WITHOUT ROWID table or if there is no IPK column, ** leave zPk as "_rowid_" and pTab->iPk at -2. */ pTab->iPk = -2; if( bIntkey ){ shellPreparePrintf(dbtmp, &rc, &pPkFinder, "SELECT cid, name FROM pragma_table_info(%Q) " " WHERE pk=1 AND type='integer' COLLATE nocase" " AND NOT EXISTS (SELECT cid FROM pragma_table_info(%Q) WHERE pk=2)" , zName, zName ); if( rc==SQLITE_OK && SQLITE_ROW==sqlite3_step(pPkFinder) ){ pTab->iPk = sqlite3_column_int(pPkFinder, 0); zPk = (const char*)sqlite3_column_text(pPkFinder, 1); } } pTab->zQuoted = shellMPrintf(&rc, "%Q", zName); pTab->azlCol = (char**)shellMalloc(&rc, sizeof(char*) * (nSqlCol+1)); pTab->nCol = nSqlCol; if( bIntkey ){ pTab->azlCol[0] = shellMPrintf(&rc, "%Q", zPk); }else{ pTab->azlCol[0] = shellMPrintf(&rc, ""); } i = 1; shellPreparePrintf(dbtmp, &rc, &pStmt, "SELECT %Q || group_concat(name, ', ') " " FILTER (WHERE cid!=%d) OVER (ORDER BY %s cid) " "FROM pragma_table_info(%Q)", bIntkey ? ", " : "", pTab->iPk, bIntkey ? "" : "(CASE WHEN pk=0 THEN 1000000 ELSE pk END), ", zName ); while( rc==SQLITE_OK && SQLITE_ROW==sqlite3_step(pStmt) ){ const char *zText = (const char*)sqlite3_column_text(pStmt, 0); pTab->azlCol[i] = shellMPrintf(&rc, "%s%s", pTab->azlCol[0], zText); i++; } shellFinalize(&rc, pStmt); shellFinalize(&rc, pPkFinder); } } finished: sqlite3_close(dbtmp); *pRc = rc; if( rc!=SQLITE_OK ){ recoverFreeTable(pTab); pTab = 0; } return pTab; } /* ** This function is called to search the schema recovered from the ** sqlite_master table of the (possibly) corrupt database as part ** of a ".recover" command. Specifically, for a table with root page ** iRoot and at least nCol columns. Additionally, if bIntkey is 0, the ** table must be a WITHOUT ROWID table, or if non-zero, not one of ** those. ** ** If a table is found, a (RecoverTable*) object is returned. Or, if ** no such table is found, but bIntkey is false and iRoot is the ** root page of an index in the recovered schema, then (*pbNoop) is ** set to true and NULL returned. Or, if there is no such table or ** index, NULL is returned and (*pbNoop) set to 0, indicating that ** the caller should write data to the orphans table. */ static RecoverTable *recoverFindTable( ShellState *pState, /* Shell state object */ int *pRc, /* IN/OUT: Error code */ int iRoot, /* Root page of table */ int bIntkey, /* True for an intkey table */ int nCol, /* Number of columns in table */ int *pbNoop /* OUT: True if iRoot is root of index */ ){ sqlite3_stmt *pStmt = 0; RecoverTable *pRet = 0; int bNoop = 0; const char *zSql = 0; const char *zName = 0; /* Search the recovered schema for an object with root page iRoot. */ shellPreparePrintf(pState->db, pRc, &pStmt, "SELECT type, name, sql FROM recovery.schema WHERE rootpage=%d", iRoot ); while( *pRc==SQLITE_OK && SQLITE_ROW==sqlite3_step(pStmt) ){ const char *zType = (const char*)sqlite3_column_text(pStmt, 0); if( bIntkey==0 && sqlite3_stricmp(zType, "index")==0 ){ bNoop = 1; break; } if( sqlite3_stricmp(zType, "table")==0 ){ zName = (const char*)sqlite3_column_text(pStmt, 1); zSql = (const char*)sqlite3_column_text(pStmt, 2); pRet = recoverNewTable(pRc, zName, zSql, bIntkey, nCol); break; } } shellFinalize(pRc, pStmt); *pbNoop = bNoop; return pRet; } /* ** Return a RecoverTable object representing the orphans table. */ static RecoverTable *recoverOrphanTable( ShellState *pState, /* Shell state object */ int *pRc, /* IN/OUT: Error code */ const char *zLostAndFound, /* Base name for orphans table */ int nCol /* Number of user data columns */ ){ RecoverTable *pTab = 0; if( nCol>=0 && *pRc==SQLITE_OK ){ int i; /* This block determines the name of the orphan table. The prefered ** name is zLostAndFound. But if that clashes with another name ** in the recovered schema, try zLostAndFound_0, zLostAndFound_1 ** and so on until a non-clashing name is found. */ int iTab = 0; char *zTab = shellMPrintf(pRc, "%s", zLostAndFound); sqlite3_stmt *pTest = 0; shellPrepare(pState->db, pRc, "SELECT 1 FROM recovery.schema WHERE name=?", &pTest ); if( pTest ) sqlite3_bind_text(pTest, 1, zTab, -1, SQLITE_TRANSIENT); while( *pRc==SQLITE_OK && SQLITE_ROW==sqlite3_step(pTest) ){ shellReset(pRc, pTest); sqlite3_free(zTab); zTab = shellMPrintf(pRc, "%s_%d", zLostAndFound, iTab++); sqlite3_bind_text(pTest, 1, zTab, -1, SQLITE_TRANSIENT); } shellFinalize(pRc, pTest); pTab = (RecoverTable*)shellMalloc(pRc, sizeof(RecoverTable)); if( pTab ){ pTab->zQuoted = shellMPrintf(pRc, "%Q", zTab); pTab->nCol = nCol; pTab->iPk = -2; if( nCol>0 ){ pTab->azlCol = (char**)shellMalloc(pRc, sizeof(char*) * (nCol+1)); if( pTab->azlCol ){ pTab->azlCol[nCol] = shellMPrintf(pRc, ""); for(i=nCol-1; i>=0; i--){ pTab->azlCol[i] = shellMPrintf(pRc, "%s, NULL", pTab->azlCol[i+1]); } } } if( *pRc!=SQLITE_OK ){ recoverFreeTable(pTab); pTab = 0; }else{ raw_printf(pState->out, "CREATE TABLE %s(rootpgno INTEGER, " "pgno INTEGER, nfield INTEGER, id INTEGER", pTab->zQuoted ); for(i=0; i<nCol; i++){ raw_printf(pState->out, ", c%d", i); } raw_printf(pState->out, ");\n"); } } sqlite3_free(zTab); } return pTab; } /* ** This function is called to recover data from the database. A script ** to construct a new database containing all recovered data is output ** on stream pState->out. */ static int recoverDatabaseCmd(ShellState *pState, int nArg, char **azArg){ int rc = SQLITE_OK; sqlite3_stmt *pLoop = 0; /* Loop through all root pages */ sqlite3_stmt *pPages = 0; /* Loop through all pages in a group */ sqlite3_stmt *pCells = 0; /* Loop through all cells in a page */ const char *zRecoveryDb = ""; /* Name of "recovery" database */ const char *zLostAndFound = "lost_and_found"; int i; int nOrphan = -1; RecoverTable *pOrphan = 0; int bFreelist = 1; /* 0 if --freelist-corrupt is specified */ for(i=1; i<nArg; i++){ char *z = azArg[i]; int n; if( z[0]=='-' && z[1]=='-' ) z++; n = strlen(z); if( n<=17 && memcmp("-freelist-corrupt", z, n)==0 ){ bFreelist = 0; }else if( n<=12 && memcmp("-recovery-db", z, n)==0 && i<(nArg-1) ){ i++; zRecoveryDb = azArg[i]; }else if( n<=15 && memcmp("-lost-and-found", z, n)==0 && i<(nArg-1) ){ i++; zLostAndFound = azArg[i]; } else{ raw_printf(stderr, "unexpected option: %s\n", azArg[i]); raw_printf(stderr, "options are:\n"); raw_printf(stderr, " --freelist-corrupt\n"); raw_printf(stderr, " --recovery-db DATABASE\n"); raw_printf(stderr, " --lost-and-found TABLE-NAME\n"); return 1; } } shellExecPrintf(pState->db, &rc, /* Attach an in-memory database named 'recovery'. Create an indexed ** cache of the sqlite_dbptr virtual table. */ "ATTACH %Q AS recovery;" "DROP TABLE IF EXISTS recovery.dbptr;" "DROP TABLE IF EXISTS recovery.freelist;" "DROP TABLE IF EXISTS recovery.map;" "DROP TABLE IF EXISTS recovery.schema;" "CREATE TABLE recovery.freelist(pgno INTEGER PRIMARY KEY);", zRecoveryDb ); if( bFreelist ){ shellExec(pState->db, &rc, "WITH trunk(pgno) AS (" " SELECT shell_int32(" " (SELECT data FROM sqlite_dbpage WHERE pgno=1), 8) AS x " " WHERE x>0" " UNION" " SELECT shell_int32(" " (SELECT data FROM sqlite_dbpage WHERE pgno=trunk.pgno), 0) AS x " " FROM trunk WHERE x>0" ")," "freelist(data, n, freepgno) AS (" " SELECT data, shell_int32(data, 1)-1, t.pgno " " FROM trunk t, sqlite_dbpage s WHERE s.pgno=t.pgno" " UNION ALL" " SELECT data, n-1, shell_int32(data, 2+n) " " FROM freelist WHERE n>=0" ")" "REPLACE INTO recovery.freelist SELECT freepgno FROM freelist;" ); } shellExec(pState->db, &rc, "CREATE TABLE recovery.dbptr(" " pgno, child, PRIMARY KEY(child, pgno)" ") WITHOUT ROWID;" "INSERT OR IGNORE INTO recovery.dbptr(pgno, child) " " SELECT * FROM sqlite_dbptr" " WHERE pgno NOT IN freelist AND child NOT IN freelist;" /* Delete any pointer to page 1. This ensures that page 1 is considered ** a root page, regardless of how corrupt the db is. */ "DELETE FROM recovery.dbptr WHERE child = 1;" /* Delete all pointers to any pages that have more than one pointer ** to them. Such pages will be treated as root pages when recovering ** data. */ "DELETE FROM recovery.dbptr WHERE child IN (" " SELECT child FROM recovery.dbptr GROUP BY child HAVING count(*)>1" ");" /* Create the "map" table that will (eventually) contain instructions ** for dealing with each page in the db that contains one or more ** records. */ "CREATE TABLE recovery.map(" "pgno INTEGER PRIMARY KEY, maxlen INT, intkey, root INT" ");" /* Populate table [map]. If there are circular loops of pages in the ** database, the following adds all pages in such a loop to the map ** as individual root pages. This could be handled better. */ "WITH pages(i, maxlen) AS (" " SELECT page_count, (" " SELECT max(field+1) FROM sqlite_dbdata WHERE pgno=page_count" " ) FROM pragma_page_count" " UNION ALL" " SELECT i-1, (" " SELECT max(field+1) FROM sqlite_dbdata WHERE pgno=i-1" " ) FROM pages WHERE i>=2" ")" "INSERT INTO recovery.map(pgno, maxlen, intkey, root) " " SELECT i, maxlen, NULL, (" " WITH p(orig, pgno, parent) AS (" " SELECT 0, i, (SELECT pgno FROM recovery.dbptr WHERE child=i)" " UNION ALL" " SELECT i, p.parent, " " (SELECT pgno FROM recovery.dbptr WHERE child=p.parent) FROM p" " )" " SELECT pgno FROM p WHERE (parent IS NULL OR pgno = orig)" ") " "FROM pages WHERE maxlen > 0 AND i NOT IN freelist;" "UPDATE recovery.map AS o SET intkey = (" " SELECT substr(data, 1, 1)==X'0D' FROM sqlite_dbpage WHERE pgno=o.pgno" ");" /* Extract data from page 1 and any linked pages into table ** recovery.schema. With the same schema as an sqlite_master table. */ "CREATE TABLE recovery.schema(type, name, tbl_name, rootpage, sql);" "INSERT INTO recovery.schema SELECT " " max(CASE WHEN field=0 THEN value ELSE NULL END)," " max(CASE WHEN field=1 THEN value ELSE NULL END)," " max(CASE WHEN field=2 THEN value ELSE NULL END)," " max(CASE WHEN field=3 THEN value ELSE NULL END)," " max(CASE WHEN field=4 THEN value ELSE NULL END)" "FROM sqlite_dbdata WHERE pgno IN (" " SELECT pgno FROM recovery.map WHERE root=1" ")" "GROUP BY pgno, cell;" "CREATE INDEX recovery.schema_rootpage ON schema(rootpage);" ); /* Open a transaction, then print out all non-virtual, non-"sqlite_%" ** CREATE TABLE statements that extracted from the existing schema. */ if( rc==SQLITE_OK ){ sqlite3_stmt *pStmt = 0; raw_printf(pState->out, "BEGIN;\n"); raw_printf(pState->out, "PRAGMA writable_schema = on;\n"); shellPrepare(pState->db, &rc, "SELECT sql FROM recovery.schema " "WHERE type='table' AND sql LIKE 'create table%'", &pStmt ); while( rc==SQLITE_OK && SQLITE_ROW==sqlite3_step(pStmt) ){ const char *zCreateTable = (const char*)sqlite3_column_text(pStmt, 0); raw_printf(pState->out, "CREATE TABLE IF NOT EXISTS %s;\n", &zCreateTable[12] ); } shellFinalize(&rc, pStmt); } /* Figure out if an orphan table will be required. And if so, how many ** user columns it should contain */ shellPrepare(pState->db, &rc, "SELECT coalesce(max(maxlen), -2) FROM recovery.map" " WHERE root>1 AND root NOT IN (SELECT rootpage FROM recovery.schema)" , &pLoop ); if( rc==SQLITE_OK && SQLITE_ROW==sqlite3_step(pLoop) ){ nOrphan = sqlite3_column_int(pLoop, 0); } shellFinalize(&rc, pLoop); pLoop = 0; pOrphan = recoverOrphanTable(pState, &rc, zLostAndFound, nOrphan); shellPrepare(pState->db, &rc, "SELECT pgno FROM recovery.map WHERE root=?", &pPages ); shellPrepare(pState->db, &rc, "SELECT max(field), group_concat(shell_escape_crnl(quote(value)), ', ')" "FROM sqlite_dbdata WHERE pgno = ? AND field != ?" "GROUP BY cell", &pCells ); /* Loop through each root page. */ shellPrepare(pState->db, &rc, "SELECT root, intkey, max(maxlen) FROM recovery.map" " WHERE root>1 GROUP BY root, intkey ORDER BY root=(" " SELECT rootpage FROM recovery.schema WHERE name='sqlite_sequence'" ")", &pLoop ); while( rc==SQLITE_OK && SQLITE_ROW==sqlite3_step(pLoop) ){ int iRoot = sqlite3_column_int(pLoop, 0); int bIntkey = sqlite3_column_int(pLoop, 1); int nCol = sqlite3_column_int(pLoop, 2); int bNoop = 0; RecoverTable *pTab; pTab = recoverFindTable(pState, &rc, iRoot, bIntkey, nCol, &bNoop); if( bNoop || rc ) continue; if( pTab==0 ) pTab = pOrphan; if( 0==sqlite3_stricmp(pTab->zQuoted, "'sqlite_sequence'") ){ raw_printf(pState->out, "DELETE FROM sqlite_sequence;\n"); } sqlite3_bind_int(pPages, 1, iRoot); sqlite3_bind_int(pCells, 2, pTab->iPk); while( rc==SQLITE_OK && SQLITE_ROW==sqlite3_step(pPages) ){ int iPgno = sqlite3_column_int(pPages, 0); sqlite3_bind_int(pCells, 1, iPgno); while( rc==SQLITE_OK && SQLITE_ROW==sqlite3_step(pCells) ){ int nField = sqlite3_column_int(pCells, 0); const char *zVal = (const char*)sqlite3_column_text(pCells, 1); nField = nField+1; if( pTab==pOrphan ){ raw_printf(pState->out, "INSERT INTO %s VALUES(%d, %d, %d, %s%s%s);\n", pTab->zQuoted, iRoot, iPgno, nField, bIntkey ? "" : "NULL, ", zVal, pTab->azlCol[nField] ); }else{ raw_printf(pState->out, "INSERT INTO %s(%s) VALUES( %s );\n", pTab->zQuoted, pTab->azlCol[nField], zVal ); } } shellReset(&rc, pCells); } shellReset(&rc, pPages); if( pTab!=pOrphan ) recoverFreeTable(pTab); } shellFinalize(&rc, pLoop); shellFinalize(&rc, pPages); shellFinalize(&rc, pCells); recoverFreeTable(pOrphan); /* The rest of the schema */ if( rc==SQLITE_OK ){ sqlite3_stmt *pStmt = 0; shellPrepare(pState->db, &rc, "SELECT sql, name FROM recovery.schema " "WHERE sql NOT LIKE 'create table%'", &pStmt ); while( rc==SQLITE_OK && SQLITE_ROW==sqlite3_step(pStmt) ){ const char *zSql = (const char*)sqlite3_column_text(pStmt, 0); if( sqlite3_strnicmp(zSql, "create virt", 11)==0 ){ const char *zName = (const char*)sqlite3_column_text(pStmt, 1); char *zPrint = shellMPrintf(&rc, "INSERT INTO sqlite_master VALUES('table', %Q, %Q, 0, %Q)", zName, zName, zSql ); raw_printf(pState->out, "%s;\n", zPrint); sqlite3_free(zPrint); }else{ raw_printf(pState->out, "%s;\n", zSql); } } shellFinalize(&rc, pStmt); } if( rc==SQLITE_OK ){ raw_printf(pState->out, "PRAGMA writable_schema = off;\n"); raw_printf(pState->out, "COMMIT;\n"); } sqlite3_exec(pState->db, "DETACH recovery", 0, 0, 0); return rc; } #endif /* !(SQLITE_OMIT_VIRTUALTABLE) && defined(SQLITE_ENABLE_DBPAGE_VTAB) */ /* ** If an input line begins with "." then invoke this routine to ** process that line. ** ** Return 1 on error, 2 to exit, and 0 otherwise. */ |
︙ | ︙ | |||
6309 6310 6311 6312 6313 6314 6315 6316 6317 6318 6319 6320 6321 6322 | } }else if( c=='d' && n>=3 && strncmp(azArg[0], "dbinfo", n)==0 ){ rc = shell_dbinfo_command(p, nArg, azArg); }else if( c=='d' && strncmp(azArg[0], "dump", n)==0 ){ const char *zLike = 0; int i; int savedShowHeader = p->showHeader; int savedShellFlags = p->shellFlgs; ShellClearFlag(p, SHFLG_PreserveRowid|SHFLG_Newlines|SHFLG_Echo); for(i=1; i<nArg; i++){ | > > > > > > > | 7070 7071 7072 7073 7074 7075 7076 7077 7078 7079 7080 7081 7082 7083 7084 7085 7086 7087 7088 7089 7090 | } }else if( c=='d' && n>=3 && strncmp(azArg[0], "dbinfo", n)==0 ){ rc = shell_dbinfo_command(p, nArg, azArg); }else #if !defined(SQLITE_OMIT_VIRTUALTABLE) && defined(SQLITE_ENABLE_DBPAGE_VTAB) if( c=='r' && strncmp(azArg[0], "recover", n)==0 ){ open_db(p, 0); rc = recoverDatabaseCmd(p, nArg, azArg); }else #endif /* !(SQLITE_OMIT_VIRTUALTABLE) && defined(SQLITE_ENABLE_DBPAGE_VTAB) */ if( c=='d' && strncmp(azArg[0], "dump", n)==0 ){ const char *zLike = 0; int i; int savedShowHeader = p->showHeader; int savedShellFlags = p->shellFlgs; ShellClearFlag(p, SHFLG_PreserveRowid|SHFLG_Newlines|SHFLG_Echo); for(i=1; i<nArg; i++){ |
︙ | ︙ | |||
6346 6347 6348 6349 6350 6351 6352 6353 6354 6355 6356 6357 6358 6359 6360 | "?--newlines? ?LIKE-PATTERN?\n"); rc = 1; goto meta_command_exit; }else{ zLike = azArg[i]; } } open_db(p, 0); /* When playing back a "dump", the content might appear in an order ** which causes immediate foreign key constraints to be violated. ** So disable foreign-key constraint enforcement to prevent problems. */ raw_printf(p->out, "PRAGMA foreign_keys=OFF;\n"); raw_printf(p->out, "BEGIN TRANSACTION;\n"); p->writableSchema = 0; p->showHeader = 0; | > > | 7114 7115 7116 7117 7118 7119 7120 7121 7122 7123 7124 7125 7126 7127 7128 7129 7130 | "?--newlines? ?LIKE-PATTERN?\n"); rc = 1; goto meta_command_exit; }else{ zLike = azArg[i]; } } open_db(p, 0); /* When playing back a "dump", the content might appear in an order ** which causes immediate foreign key constraints to be violated. ** So disable foreign-key constraint enforcement to prevent problems. */ raw_printf(p->out, "PRAGMA foreign_keys=OFF;\n"); raw_printf(p->out, "BEGIN TRANSACTION;\n"); p->writableSchema = 0; p->showHeader = 0; |
︙ | ︙ | |||
6394 6395 6396 6397 6398 6399 6400 | } if( p->writableSchema ){ raw_printf(p->out, "PRAGMA writable_schema=OFF;\n"); p->writableSchema = 0; } sqlite3_exec(p->db, "PRAGMA writable_schema=OFF;", 0, 0, 0); sqlite3_exec(p->db, "RELEASE dump;", 0, 0, 0); | | | 7164 7165 7166 7167 7168 7169 7170 7171 7172 7173 7174 7175 7176 7177 7178 | } if( p->writableSchema ){ raw_printf(p->out, "PRAGMA writable_schema=OFF;\n"); p->writableSchema = 0; } sqlite3_exec(p->db, "PRAGMA writable_schema=OFF;", 0, 0, 0); sqlite3_exec(p->db, "RELEASE dump;", 0, 0, 0); raw_printf(p->out, p->nErr?"ROLLBACK; -- due to errors\n":"COMMIT;\n"); p->showHeader = savedShowHeader; p->shellFlgs = savedShellFlags; }else if( c=='e' && strncmp(azArg[0], "echo", n)==0 ){ if( nArg==2 ){ setOrClearFlag(p, SHFLG_Echo, azArg[1]); |
︙ | ︙ |
Added test/dbdata.test.
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 | # 2019-04-11 # # The author disclaims copyright to this source code. In place of # a legal notice, here is a blessing: # # May you do good and not evil. # May you find forgiveness for yourself and forgive others. # May you share freely, never taking more than you give. # #*********************************************************************** # This file implements regression tests for SQLite library. The # focus of this file is testing the sqlite_dbpage virtual table. # set testdir [file dirname $argv0] source $testdir/tester.tcl set testprefix dbdata ifcapable !vtab||!compound { finish_test return } db enable_load_extension 1 if { [catch { db eval { SELECT load_extension('../dbdata') } }] } { finish_test return } do_execsql_test 1.0 { CREATE TABLE T1(a, b); INSERT INTO t1(rowid, a ,b) VALUES(5, 'v', 'five'); INSERT INTO t1(rowid, a, b) VALUES(10, 'x', 'ten'); } do_execsql_test 1.1 { SELECT pgno, cell, field, quote(value) FROM sqlite_dbdata WHERE pgno=2; } { 2 0 -1 5 2 0 0 'v' 2 0 1 'five' 2 1 -1 10 2 1 0 'x' 2 1 1 'ten' } breakpoint do_execsql_test 1.2 { SELECT pgno, cell, field, quote(value) FROM sqlite_dbdata; } { 1 0 -1 1 1 0 0 'table' 1 0 1 'T1' 1 0 2 'T1' 1 0 3 2 1 0 4 {'CREATE TABLE T1(a, b)'} 2 0 -1 5 2 0 0 'v' 2 0 1 'five' 2 1 -1 10 2 1 0 'x' 2 1 1 'ten' } set big [string repeat big 2000] do_execsql_test 1.3 { INSERT INTO t1 VALUES(NULL, $big); SELECT value FROM sqlite_dbdata WHERE pgno=2 AND cell=2 AND field=1; } $big do_execsql_test 1.4 { DELETE FROM t1; INSERT INTO t1 VALUES(NULL, randomblob(5050)); } do_test 1.5 { execsql { SELECT quote(value) FROM sqlite_dbdata WHERE pgno=2 AND cell=0 AND field=1; } } [db one {SELECT quote(b) FROM t1}] #------------------------------------------------------------------------- reset_db db enable_load_extension 1 db eval { SELECT load_extension('../dbdata') } do_execsql_test 2.0 { CREATE TABLE t1(a); CREATE INDEX i1 ON t1(a); WITH s(i) AS ( SELECT 1 UNION ALL SELECT i+1 FROM s WHERE i<10 ) INSERT INTO t1 SELECT randomblob(900) FROM s; } do_execsql_test 2.1 { SELECT * FROM sqlite_dbptr WHERE pgno=2; } { 2 25 2 6 2 7 2 9 2 11 2 13 2 15 2 17 2 19 2 21 } do_execsql_test 2.2 { SELECT * FROM sqlite_dbptr WHERE pgno=3; } { 3 24 3 23 } do_execsql_test 2.3 { SELECT * FROM sqlite_dbptr } { 2 25 2 6 2 7 2 9 2 11 2 13 2 15 2 17 2 19 2 21 3 24 3 23 } finish_test |
Added test/recover.test.
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 | # 2019 April 23 # # The author disclaims copyright to this source code. In place of # a legal notice, here is a blessing: # # May you do good and not evil. # May you find forgiveness for yourself and forgive others. # May you share freely, never taking more than you give. # #*********************************************************************** # # Test the shell tool ".ar" command. # set testdir [file dirname $argv0] source $testdir/tester.tcl set testprefix recover ifcapable !vtab { finish_test; return } set CLI [test_find_cli] proc compare_result {db1 db2 sql} { set r1 [$db1 eval $sql] set r2 [$db2 eval $sql] if {$r1 != $r2} { puts "r1: $r1" puts "r2: $r2" error "mismatch for $sql" } return "" } proc compare_dbs {db1 db2} { compare_result $db1 $db2 "SELECT sql FROM sqlite_master ORDER BY 1" foreach tbl [$db1 eval {SELECT name FROM sqlite_master WHERE type='table'}] { compare_result $db1 $db2 "SELECT * FROM $tbl" } } proc do_recover_test {tn {tsql {}} {res {}}} { set fd [open "|$::CLI test.db .recover"] fconfigure $fd -encoding binary fconfigure $fd -translation binary set sql [read $fd] close $fd forcedelete test.db2 sqlite3 db2 test.db2 execsql $sql db2 if {$tsql==""} { uplevel [list do_test $tn [list compare_dbs db db2] {}] } else { uplevel [list do_execsql_test -db db2 $tn $tsql $res] } db2 close } set doc { hello world } do_execsql_test 1.1.1 { CREATE TABLE t1(a INTEGER PRIMARY KEY, b, c); INSERT INTO t1 VALUES(1, 4, X'1234567800'); INSERT INTO t1 VALUES(2, 'test', 8.1); INSERT INTO t1 VALUES(3, $doc, 8.4); } do_recover_test 1.1.2 do_execsql_test 1.2.1 " DELETE FROM t1; INSERT INTO t1 VALUES(13, 'hello\r\nworld', 13); " do_recover_test 1.2.2 do_execsql_test 1.3.1 " CREATE TABLE t2(i INTEGER PRIMARY KEY AUTOINCREMENT, b, c); INSERT INTO t2 VALUES(NULL, 1, 2); INSERT INTO t2 VALUES(NULL, 3, 4); INSERT INTO t2 VALUES(NULL, 5, 6); CREATE TABLE t3(i INTEGER PRIMARY KEY AUTOINCREMENT, b, c); INSERT INTO t3 VALUES(NULL, 1, 2); INSERT INTO t3 VALUES(NULL, 3, 4); INSERT INTO t3 VALUES(NULL, 5, 6); DELETE FROM t2; " do_recover_test 1.3.2 #------------------------------------------------------------------------- reset_db do_execsql_test 2.1.0 { CREATE TABLE t1(a, b, c, PRIMARY KEY(b, c)) WITHOUT ROWID; INSERT INTO t1 VALUES(1, 2, 3); INSERT INTO t1 VALUES(4, 5, 6); INSERT INTO t1 VALUES(7, 8, 9); } do_recover_test 2.1.1 do_execsql_test 2.2.0 { PRAGMA writable_schema = 1; DELETE FROM sqlite_master WHERE name='t1'; } do_recover_test 2.2.1 { SELECT name FROM sqlite_master } {lost_and_found} do_execsql_test 2.3.0 { CREATE TABLE lost_and_found(a, b, c); } do_recover_test 2.3.1 { SELECT name FROM sqlite_master } {lost_and_found lost_and_found_0} do_execsql_test 2.4.0 { CREATE TABLE lost_and_found_0(a, b, c); } do_recover_test 2.4.1 { SELECT name FROM sqlite_master; SELECT * FROM lost_and_found_1; } {lost_and_found lost_and_found_0 lost_and_found_1 2 2 3 {} 2 3 1 2 2 3 {} 5 6 4 2 2 3 {} 8 9 7 } finish_test |
Changes to tool/mkshellc.tcl.
︙ | ︙ | |||
36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 | if {[info exists typedef_seen($line)]} { return "/* $line */" } set typedef_seen($line) 1 } return $line } while {1} { set lx [omit_redundant_typedefs [gets $in]] if {[eof $in]} break; if {[regexp {^INCLUDE } $lx]} { set cfile [lindex $lx 1] puts $out "/************************* Begin $cfile ******************/" set in2 [open $topdir/src/$cfile rb] while {![eof $in2]} { set lx [omit_redundant_typedefs [gets $in2]] | > > > | > > > | 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 | if {[info exists typedef_seen($line)]} { return "/* $line */" } set typedef_seen($line) 1 } return $line } set iLine 0 while {1} { set lx [omit_redundant_typedefs [gets $in]] if {[eof $in]} break; incr iLine if {[regexp {^INCLUDE } $lx]} { set cfile [lindex $lx 1] puts $out "/************************* Begin $cfile ******************/" puts $out "#line 1 \"$cfile\"" set in2 [open $topdir/src/$cfile rb] while {![eof $in2]} { set lx [omit_redundant_typedefs [gets $in2]] if {[regexp {^#include "sqlite} $lx]} { set lx "/* $lx */" } if {[regexp {^# *include "test_windirent.h"} $lx]} { set lx "/* $lx */" } set lx [string map [list __declspec(dllexport) {}] $lx] puts $out $lx } close $in2 puts $out "/************************* End $cfile ********************/" puts $out "#line [expr $iLine+1] \"shell.c.in\"" continue } puts $out $lx } close $in close $out |