/ Check-in [c981ab2a]
Login

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

Overview
Comment:Add the "scrub" utility program that simultaneously backs-up a live database and erases all deleted content.
Downloads: Tarball | ZIP archive | SQL archive
Timelines: family | ancestors | descendants | both | trunk
Files: files | file ages | folders
SHA1: c981ab2a4771f8c5e95e2e85e37a02a4bd0e36ee
User & Date: drh 2016-05-18 21:03:27
Context
2016-05-19
11:12
Small size reduction and performance improvement in the OP_Column opcode. check-in: 0d773061 user: drh tags: trunk
2016-05-18
21:03
Add the "scrub" utility program that simultaneously backs-up a live database and erases all deleted content. check-in: c981ab2a user: drh tags: trunk
21:01
Omit the unnecessary WHERE_REOPEN_IDX flag in the WHERE interface. check-in: 915416d1 user: drh tags: trunk
2016-05-17
21:17
Enhance the scrub utility program so that it does a FULL checkpoint prior to starting the backup, to ensure that the database file content matches what needs to be backed up without having to look at the WAL file. Closed-Leaf check-in: ab1c5ce5 user: drh tags: scrub-backup
Changes
Hide Diffs Side-by-Side Diffs Show Whitespace Changes Patch

Changes to Makefile.in.

   584    584   sqlite3$(TEXE):	$(TOP)/src/shell.c sqlite3.c
   585    585   	$(LTLINK) $(READLINE_FLAGS) $(SHELL_OPT) -o $@ \
   586    586   		$(TOP)/src/shell.c sqlite3.c \
   587    587   		$(LIBREADLINE) $(TLIBS) -rpath "$(libdir)"
   588    588   
   589    589   sqldiff$(TEXE):	$(TOP)/tool/sqldiff.c sqlite3.c sqlite3.h
   590    590   	$(LTLINK) -o $@ $(TOP)/tool/sqldiff.c sqlite3.c $(TLIBS)
          591  +
          592  +scrub$(TEXE):	$(TOP)/ext/misc/scrub.c sqlite3.o
          593  +	$(LTLINK) -o $@ -I. -DSCRUB_STANDALONE \
          594  +		$(TOP)/ext/misc/scrub.c sqlite3.o $(TLIBS)
   591    595   
   592    596   srcck1$(BEXE):	$(TOP)/tool/srcck1.c
   593    597   	$(BCC) -o srcck1$(BEXE) $(TOP)/tool/srcck1.c
   594    598   
   595    599   sourcetest:	srcck1$(BEXE) sqlite3.c
   596    600   	./srcck1 sqlite3.c
   597    601   

Changes to Makefile.msc.

  1451   1451   	$(LTLINK) $(SHELL_COMPILE_OPTS) $(READLINE_FLAGS) $(TOP)\src\shell.c $(SHELL_CORE_SRC) \
  1452   1452   		/link $(SQLITE3EXEPDB) $(LDFLAGS) $(LTLINKOPTS) $(SHELL_LINK_OPTS) $(LTLIBPATHS) $(LIBRESOBJS) $(LIBREADLINE) $(LTLIBS) $(TLIBS)
  1453   1453   
  1454   1454   # <<mark>>
  1455   1455   sqldiff.exe:	$(TOP)\tool\sqldiff.c $(SQLITE3C) $(SQLITE3H)
  1456   1456   	$(LTLINK) $(NO_WARN) $(TOP)\tool\sqldiff.c $(SQLITE3C) /link $(LDFLAGS) $(LTLINKOPTS)
  1457   1457   
         1458  +scrub.exe:	$(TOP)\ext\misc\scrub.c $(SQLITE3C) $(SQLITE3H)
         1459  +	$(LTLINK) $(NO_WARN) $(TOP)\ext\misc\scrub.c $(SQLITE3C) /link $(LDFLAGS) $(LTLINKOPTS)
         1460  +
  1458   1461   srcck1.exe:	$(TOP)\tool\srcck1.c
  1459   1462   	$(BCC) $(NO_WARN) -Fe$@ $(TOP)\tool\srcck1.c
  1460   1463   
  1461   1464   sourcetest:	srcck1.exe sqlite3.c
  1462   1465   	srcck1.exe sqlite3.c
  1463   1466   
  1464   1467   fuzzershell.exe:	$(TOP)\tool\fuzzershell.c $(SQLITE3C) $(SQLITE3H)

Added ext/misc/scrub.c.

            1  +/*
            2  +** 2016-05-05
            3  +**
            4  +** The author disclaims copyright to this source code.  In place of
            5  +** a legal notice, here is a blessing:
            6  +**
            7  +**    May you do good and not evil.
            8  +**    May you find forgiveness for yourself and forgive others.
            9  +**    May you share freely, never taking more than you give.
           10  +**
           11  +******************************************************************************
           12  +**
           13  +** This file implements a utility function (and a utility program) that
           14  +** makes a copy of an SQLite database while simultaneously zeroing out all
           15  +** deleted content.
           16  +**
           17  +** Normally (when PRAGMA secure_delete=OFF, which is the default) when SQLite
           18  +** deletes content, it does not overwrite the deleted content but rather marks
           19  +** the region of the file that held that content as being reusable.  This can
           20  +** cause deleted content to recoverable from the database file.  This stale
           21  +** content is removed by the VACUUM command, but VACUUM can be expensive for
           22  +** large databases.  When in PRAGMA secure_delete=ON mode, the deleted content
           23  +** is zeroed, but secure_delete=ON has overhead as well.
           24  +**
           25  +** This utility attempts to make a copy of a complete SQLite database where
           26  +** all of the deleted content is zeroed out in the copy, and it attempts to
           27  +** do so while being faster than running VACUUM.
           28  +**
           29  +** Usage:
           30  +**
           31  +**   int sqlite3_scrub_backup(
           32  +**       const char *zSourceFile,   // Source database filename
           33  +**       const char *zDestFile,     // Destination database filename
           34  +**       char **pzErrMsg            // Write error message here
           35  +**   );
           36  +**
           37  +** Simply call the API above specifying the filename of the source database
           38  +** and the name of the backup copy.  The source database must already exist
           39  +** and can be in active use. (A read lock is held during the backup.)  The
           40  +** destination file should not previously exist.  If the pzErrMsg parameter
           41  +** is non-NULL and if an error occurs, then an error message might be written
           42  +** into memory obtained from sqlite3_malloc() and *pzErrMsg made to point to
           43  +** that error message.  But if the error is an OOM, the error might not be
           44  +** reported.  The routine always returns non-zero if there is an error.
           45  +**
           46  +** If compiled with -DSCRUB_STANDALONE then a main() procedure is added and
           47  +** this file becomes a standalone program that can be run as follows:
           48  +**
           49  +**      ./sqlite3scrub SOURCE DEST
           50  +*/
           51  +#include "sqlite3.h"
           52  +#include <assert.h>
           53  +#include <stdio.h>
           54  +#include <stdlib.h>
           55  +#include <stdarg.h>
           56  +#include <string.h>
           57  +
           58  +typedef struct ScrubState ScrubState;
           59  +typedef unsigned char u8;
           60  +typedef unsigned short u16;
           61  +typedef unsigned int u32;
           62  +
           63  +
           64  +/* State information for a scrub-and-backup operation */
           65  +struct ScrubState {
           66  +  const char *zSrcFile;    /* Name of the source file */
           67  +  const char *zDestFile;   /* Name of the destination file */
           68  +  int rcErr;               /* Error code */
           69  +  char *zErr;              /* Error message text */
           70  +  sqlite3 *dbSrc;          /* Source database connection */
           71  +  sqlite3_file *pSrc;      /* Source file handle */
           72  +  sqlite3 *dbDest;         /* Destination database connection */
           73  +  sqlite3_file *pDest;     /* Destination file handle */
           74  +  u32 szPage;              /* Page size */
           75  +  u32 szUsable;            /* Usable bytes on each page */
           76  +  u32 nPage;               /* Number of pages */
           77  +  u8 *page1;               /* Content of page 1 */
           78  +};
           79  +
           80  +/* Store an error message */
           81  +static void scrubBackupErr(ScrubState *p, const char *zFormat, ...){
           82  +  va_list ap;
           83  +  sqlite3_free(p->zErr);
           84  +  va_start(ap, zFormat);
           85  +  p->zErr = sqlite3_vmprintf(zFormat, ap);
           86  +  va_end(ap);
           87  +  if( p->rcErr==0 ) p->rcErr = SQLITE_ERROR;
           88  +}
           89  +
           90  +/* Allocate memory to hold a single page of content */
           91  +static u8 *scrubBackupAllocPage(ScrubState *p){
           92  +  u8 *pPage;
           93  +  if( p->rcErr ) return 0;
           94  +  pPage = sqlite3_malloc( p->szPage );
           95  +  if( pPage==0 ) p->rcErr = SQLITE_NOMEM;
           96  +  return pPage;
           97  +}
           98  +
           99  +/* Read a page from the source database into memory.  Use the memory
          100  +** provided by pBuf if not NULL or allocate a new page if pBuf==NULL.
          101  +*/
          102  +static u8 *scrubBackupRead(ScrubState *p, int pgno, u8 *pBuf){
          103  +  int rc;
          104  +  sqlite3_int64 iOff;
          105  +  u8 *pOut = pBuf;
          106  +  if( p->rcErr ) return 0;
          107  +  if( pOut==0 ){
          108  +    pOut = scrubBackupAllocPage(p);
          109  +    if( pOut==0 ) return 0;
          110  +  }
          111  +  iOff = (pgno-1)*(sqlite3_int64)p->szPage;
          112  +  rc = p->pSrc->pMethods->xRead(p->pSrc, pOut, p->szPage, iOff);
          113  +  if( rc!=SQLITE_OK ){
          114  +    if( pBuf==0 ) sqlite3_free(pOut);
          115  +    pOut = 0;
          116  +    scrubBackupErr(p, "read failed for page %d", pgno);
          117  +    p->rcErr = SQLITE_IOERR;
          118  +  }
          119  +  return pOut;  
          120  +}
          121  +
          122  +/* Write a page to the destination database */
          123  +static void scrubBackupWrite(ScrubState *p, int pgno, const u8 *pData){
          124  +  int rc;
          125  +  sqlite3_int64 iOff;
          126  +  if( p->rcErr ) return;
          127  +  iOff = (pgno-1)*(sqlite3_int64)p->szPage;
          128  +  rc = p->pDest->pMethods->xWrite(p->pDest, pData, p->szPage, iOff);
          129  +  if( rc!=SQLITE_OK ){
          130  +    scrubBackupErr(p, "write failed for page %d", pgno);
          131  +    p->rcErr = SQLITE_IOERR;
          132  +  }
          133  +}
          134  +
          135  +/* Prepare a statement against the "db" database. */
          136  +static sqlite3_stmt *scrubBackupPrepare(
          137  +  ScrubState *p,      /* Backup context */
          138  +  sqlite3 *db,        /* Database to prepare against */
          139  +  const char *zSql    /* SQL statement */
          140  +){
          141  +  sqlite3_stmt *pStmt;
          142  +  if( p->rcErr ) return 0;
          143  +  p->rcErr = sqlite3_prepare_v2(db, zSql, -1, &pStmt, 0);
          144  +  if( p->rcErr ){
          145  +    scrubBackupErr(p, "SQL error \"%s\" on \"%s\"",
          146  +                   sqlite3_errmsg(db), zSql);
          147  +    sqlite3_finalize(pStmt);
          148  +    return 0;
          149  +  }
          150  +  return pStmt;
          151  +}
          152  +
          153  +
          154  +/* Open the source database file */
          155  +static void scrubBackupOpenSrc(ScrubState *p){
          156  +  sqlite3_stmt *pStmt;
          157  +  int rc;
          158  +  /* Open the source database file */
          159  +  p->rcErr = sqlite3_open_v2(p->zSrcFile, &p->dbSrc,
          160  +                 SQLITE_OPEN_READWRITE |
          161  +                 SQLITE_OPEN_URI | SQLITE_OPEN_PRIVATECACHE, 0);
          162  +  if( p->rcErr ){
          163  +    scrubBackupErr(p, "cannot open source database: %s",
          164  +                      sqlite3_errmsg(p->dbSrc));
          165  +    return;
          166  +  }
          167  +  p->rcErr = sqlite3_exec(p->dbSrc, "SELECT 1 FROM sqlite_master; BEGIN;",
          168  +                          0, 0, 0);
          169  +  if( p->rcErr ){
          170  +    scrubBackupErr(p,
          171  +       "cannot start a read transaction on the source database: %s",
          172  +       sqlite3_errmsg(p->dbSrc));
          173  +    return;
          174  +  }
          175  +  rc = sqlite3_wal_checkpoint_v2(p->dbSrc, "main", SQLITE_CHECKPOINT_FULL,
          176  +                                 0, 0);
          177  +  if( rc ){
          178  +    scrubBackupErr(p, "cannot checkpoint the source database");
          179  +    return;
          180  +  }
          181  +  pStmt = scrubBackupPrepare(p, p->dbSrc, "PRAGMA page_size");
          182  +  if( pStmt==0 ) return;
          183  +  rc = sqlite3_step(pStmt);
          184  +  if( rc==SQLITE_ROW ){
          185  +    p->szPage = sqlite3_column_int(pStmt, 0);
          186  +  }else{
          187  +    scrubBackupErr(p, "unable to determine the page size");
          188  +  }
          189  +  sqlite3_finalize(pStmt);
          190  +  if( p->rcErr ) return;
          191  +  pStmt = scrubBackupPrepare(p, p->dbSrc, "PRAGMA page_count");
          192  +  if( pStmt==0 ) return;
          193  +  rc = sqlite3_step(pStmt);
          194  +  if( rc==SQLITE_ROW ){
          195  +    p->nPage = sqlite3_column_int(pStmt, 0);
          196  +  }else{
          197  +    scrubBackupErr(p, "unable to determine the size of the source database");
          198  +  }
          199  +  sqlite3_finalize(pStmt);
          200  +  sqlite3_file_control(p->dbSrc, "main", SQLITE_FCNTL_FILE_POINTER, &p->pSrc);
          201  +  if( p->pSrc==0 || p->pSrc->pMethods==0 ){
          202  +    scrubBackupErr(p, "cannot get the source file handle");
          203  +    p->rcErr = SQLITE_ERROR;
          204  +  }
          205  +}
          206  +
          207  +/* Create and open the destination file */
          208  +static void scrubBackupOpenDest(ScrubState *p){
          209  +  sqlite3_stmt *pStmt;
          210  +  int rc;
          211  +  char *zSql;
          212  +  if( p->rcErr ) return;
          213  +  p->rcErr = sqlite3_open_v2(p->zDestFile, &p->dbDest,
          214  +                 SQLITE_OPEN_READWRITE | SQLITE_OPEN_CREATE |
          215  +                 SQLITE_OPEN_URI | SQLITE_OPEN_PRIVATECACHE, 0);
          216  +  if( p->rcErr ){
          217  +    scrubBackupErr(p, "cannot open destination database: %s",
          218  +                      sqlite3_errmsg(p->dbDest));
          219  +    return;
          220  +  }
          221  +  zSql = sqlite3_mprintf("PRAGMA page_size(%u);", p->szPage);
          222  +  if( zSql==0 ){
          223  +    p->rcErr = SQLITE_NOMEM;
          224  +    return;
          225  +  }
          226  +  p->rcErr = sqlite3_exec(p->dbDest, zSql, 0, 0, 0);
          227  +  sqlite3_free(zSql);
          228  +  if( p->rcErr ){
          229  +    scrubBackupErr(p,
          230  +       "cannot set the page size on the destination database: %s",
          231  +       sqlite3_errmsg(p->dbDest));
          232  +    return;
          233  +  }
          234  +  sqlite3_exec(p->dbDest, "PRAGMA journal_mode=OFF;", 0, 0, 0);
          235  +  p->rcErr = sqlite3_exec(p->dbDest, "BEGIN EXCLUSIVE;", 0, 0, 0);
          236  +  if( p->rcErr ){
          237  +    scrubBackupErr(p,
          238  +       "cannot start a write transaction on the destination database: %s",
          239  +       sqlite3_errmsg(p->dbDest));
          240  +    return;
          241  +  }
          242  +  pStmt = scrubBackupPrepare(p, p->dbDest, "PRAGMA page_count;");
          243  +  if( pStmt==0 ) return;
          244  +  rc = sqlite3_step(pStmt);
          245  +  if( rc!=SQLITE_ROW ){
          246  +    scrubBackupErr(p, "cannot measure the size of the destination");
          247  +  }else if( sqlite3_column_int(pStmt, 0)>1 ){
          248  +    scrubBackupErr(p, "destination database is not empty - holds %d pages",
          249  +                   sqlite3_column_int(pStmt, 0));
          250  +  }
          251  +  sqlite3_finalize(pStmt);
          252  +  sqlite3_file_control(p->dbDest, "main", SQLITE_FCNTL_FILE_POINTER, &p->pDest);
          253  +  if( p->pDest==0 || p->pDest->pMethods==0 ){
          254  +    scrubBackupErr(p, "cannot get the destination file handle");
          255  +    p->rcErr = SQLITE_ERROR;
          256  +  }
          257  +}
          258  +
          259  +/* Read a 32-bit big-endian integer */
          260  +static u32 scrubBackupInt32(const u8 *a){
          261  +  u32 v = a[3];
          262  +  v += ((u32)a[2])<<8;
          263  +  v += ((u32)a[1])<<16;
          264  +  v += ((u32)a[0])<<24;
          265  +  return v;
          266  +}
          267  +
          268  +/* Read a 16-bit big-endian integer */
          269  +static u32 scrubBackupInt16(const u8 *a){
          270  +  return (a[0]<<8) + a[1];
          271  +}
          272  +
          273  +/*
          274  +** Read a varint.  Put the value in *pVal and return the number of bytes.
          275  +*/
          276  +static int scrubBackupVarint(const u8 *z, sqlite3_int64 *pVal){
          277  +  sqlite3_int64 v = 0;
          278  +  int i;
          279  +  for(i=0; i<8; i++){
          280  +    v = (v<<7) + (z[i]&0x7f);
          281  +    if( (z[i]&0x80)==0 ){ *pVal = v; return i+1; }
          282  +  }
          283  +  v = (v<<8) + (z[i]&0xff);
          284  +  *pVal = v;
          285  +  return 9;
          286  +}
          287  +
          288  +/*
          289  +** Return the number of bytes in a varint.
          290  +*/
          291  +static int scrubBackupVarintSize(const u8 *z){
          292  +  int i;
          293  +  for(i=0; i<8; i++){
          294  +    if( (z[i]&0x80)==0 ){ return i+1; }
          295  +  }
          296  +  return 9;
          297  +}
          298  +
          299  +/*
          300  +** Copy the freelist trunk page given, and all its descendents,
          301  +** zeroing out as much as possible in the process.
          302  +*/
          303  +static void scrubBackupFreelist(ScrubState *p, int pgno, u32 nFree){
          304  +  u8 *a, *aBuf;
          305  +  u32 n, mx;
          306  +
          307  +  if( p->rcErr ) return;
          308  +  aBuf = scrubBackupAllocPage(p);
          309  +  if( aBuf==0 ) return;
          310  + 
          311  +  while( pgno && nFree){
          312  +    a = scrubBackupRead(p, pgno, aBuf);
          313  +    if( a==0 ) break;
          314  +    n = scrubBackupInt32(&a[4]);
          315  +    mx = p->szUsable/4 - 2;
          316  +    if( n<mx ){
          317  +      memset(&a[n*4+8], 0, 4*(mx-n));
          318  +    }
          319  +    scrubBackupWrite(p, pgno, a);
          320  +    pgno = scrubBackupInt32(a);
          321  +#if 0
          322  +    /* There is really no point in copying the freelist leaf pages.
          323  +    ** Simply leave them uninitialized in the destination database.  The
          324  +    ** OS filesystem should zero those pages for us automatically.
          325  +    */
          326  +    for(i=0; i<n && nFree; i++){
          327  +      u32 iLeaf = scrubBackupInt32(&a[i*4+8]);
          328  +      if( aZero==0 ){
          329  +        aZero = scrubBackupAllocPage(p);
          330  +        if( aZero==0 ){ pgno = 0; break; }
          331  +        memset(aZero, 0, p->szPage);
          332  +      }
          333  +      scrubBackupWrite(p, iLeaf, aZero);
          334  +      nFree--;
          335  +    }
          336  +#endif
          337  +  }
          338  +  sqlite3_free(aBuf);
          339  +}
          340  +
          341  +/*
          342  +** Copy an overflow chain from source to destination.  Zero out any
          343  +** unused tail at the end of the overflow chain.
          344  +*/
          345  +static void scrubBackupOverflow(ScrubState *p, int pgno, u32 nByte){
          346  +  u8 *a, *aBuf;
          347  +
          348  +  aBuf = scrubBackupAllocPage(p);
          349  +  if( aBuf==0 ) return;
          350  +  while( nByte>0 && pgno!=0 ){
          351  +    a = scrubBackupRead(p, pgno, aBuf);
          352  +    if( a==0 ) break;
          353  +    if( nByte >= (p->szUsable)-4 ){
          354  +      nByte -= (p->szUsable) - 4;
          355  +    }else{
          356  +      u32 x = (p->szUsable - 4) - nByte;
          357  +      u32 i = p->szUsable - x;
          358  +      memset(&a[i], 0, x);
          359  +      nByte = 0;
          360  +    }
          361  +    scrubBackupWrite(p, pgno, a);
          362  +    pgno = scrubBackupInt32(a);
          363  +  }
          364  +  sqlite3_free(aBuf);      
          365  +}
          366  +   
          367  +
          368  +/*
          369  +** Copy B-Tree page pgno, and all of its children, from source to destination.
          370  +** Zero out deleted content during the copy.
          371  +*/
          372  +static void scrubBackupBtree(ScrubState *p, int pgno, int iDepth){
          373  +  u8 *a;
          374  +  u32 i, n, pc;
          375  +  u32 nCell;
          376  +  u32 nPrefix;
          377  +  u32 szHdr;
          378  +  u32 iChild;
          379  +  u8 *aTop;
          380  +  u8 *aCell;
          381  +  u32 x, y;
          382  +  int ln = 0;
          383  +
          384  +  
          385  +  if( p->rcErr ) return;
          386  +  if( iDepth>50 ){
          387  +    scrubBackupErr(p, "corrupt: b-tree too deep at page %d", pgno);
          388  +    return;
          389  +  }
          390  +  if( pgno==1 ){
          391  +    a = p->page1;
          392  +  }else{
          393  +    a = scrubBackupRead(p, pgno, 0);
          394  +    if( a==0 ) return;
          395  +  }
          396  +  nPrefix = pgno==1 ? 100 : 0;
          397  +  aTop = &a[nPrefix];
          398  +  szHdr = 8 + 4*(aTop[0]==0x02 || aTop[0]==0x05);
          399  +  aCell = aTop + szHdr;
          400  +  nCell = scrubBackupInt16(&aTop[3]);
          401  +
          402  +  /* Zero out the gap between the cell index and the start of the
          403  +  ** cell content area */
          404  +  x = scrubBackupInt16(&aTop[5]);  /* First byte of cell content area */
          405  +  if( x>p->szUsable ){ ln=__LINE__; goto btree_corrupt; }
          406  +  y = szHdr + nPrefix + nCell*2;
          407  +  if( y>x ){ ln=__LINE__; goto btree_corrupt; }
          408  +  if( y<x ) memset(a+y, 0, x-y);  /* Zero the gap */
          409  +
          410  +  /* Zero out all the free blocks */  
          411  +  pc = scrubBackupInt16(&aTop[1]);
          412  +  if( pc>0 && pc<x ){ ln=__LINE__; goto btree_corrupt; }
          413  +  while( pc ){
          414  +    if( pc>(p->szUsable)-4 ){ ln=__LINE__; goto btree_corrupt; }
          415  +    n = scrubBackupInt16(&a[pc+2]);
          416  +    if( pc+n>(p->szUsable) ){ ln=__LINE__; goto btree_corrupt; }
          417  +    if( n>4 ) memset(&a[pc+4], 0, n-4);
          418  +    x = scrubBackupInt16(&a[pc]);
          419  +    if( x<pc+4 && x>0 ){ ln=__LINE__; goto btree_corrupt; }
          420  +    pc = x;
          421  +  }
          422  +
          423  +  /* Write this one page */
          424  +  scrubBackupWrite(p, pgno, a);
          425  +
          426  +  /* Walk the tree and process child pages */
          427  +  for(i=0; i<nCell; i++){
          428  +    u32 X, M, K, nLocal;
          429  +    sqlite3_int64 P;
          430  +    pc = scrubBackupInt16(&aCell[i*2]);
          431  +    if( pc <= szHdr ){ ln=__LINE__; goto btree_corrupt; }
          432  +    if( pc > p->szUsable-3 ){ ln=__LINE__; goto btree_corrupt; }
          433  +    if( aTop[0]==0x05 || aTop[0]==0x02 ){
          434  +      if( pc+4 > p->szUsable ){ ln=__LINE__; goto btree_corrupt; }
          435  +      iChild = scrubBackupInt32(&a[pc]);
          436  +      pc += 4;
          437  +      scrubBackupBtree(p, iChild, iDepth+1);
          438  +      if( aTop[0]==0x05 ) continue;
          439  +    }
          440  +    pc += scrubBackupVarint(&a[pc], &P);
          441  +    if( pc >= p->szUsable ){ ln=__LINE__; goto btree_corrupt; }
          442  +    if( aTop[0]==0x0d ){
          443  +      X = p->szUsable - 35;
          444  +    }else{
          445  +      X = ((p->szUsable - 12)*64/255) - 23;
          446  +    }
          447  +    if( P<=X ){
          448  +      /* All content is local.  No overflow */
          449  +      continue;
          450  +    }
          451  +    M = ((p->szUsable - 12)*32/255)-23;
          452  +    K = M + ((P-M)%(p->szUsable-4));
          453  +    if( aTop[0]==0x0d ){
          454  +      pc += scrubBackupVarintSize(&a[pc]);
          455  +      if( pc > (p->szUsable-4) ){ ln=__LINE__; goto btree_corrupt; }
          456  +    }
          457  +    nLocal = K<=X ? K : M;
          458  +    if( pc+nLocal > p->szUsable-4 ){ ln=__LINE__; goto btree_corrupt; }
          459  +    iChild = scrubBackupInt32(&a[pc+nLocal]);
          460  +    scrubBackupOverflow(p, iChild, P-nLocal);
          461  +  }
          462  +
          463  +  /* Walk the right-most tree */
          464  +  if( aTop[0]==0x05 || aTop[0]==0x02 ){
          465  +    iChild = scrubBackupInt32(&aTop[8]);
          466  +    scrubBackupBtree(p, iChild, iDepth+1);
          467  +  }
          468  +
          469  +  /* All done */
          470  +  if( pgno>1 ) sqlite3_free(a);
          471  +  return;
          472  +
          473  +btree_corrupt:
          474  +  scrubBackupErr(p, "corruption on page %d of source database (errid=%d)",
          475  +                 pgno, ln);
          476  +  if( pgno>1 ) sqlite3_free(a);  
          477  +}
          478  +
          479  +/*
          480  +** Copy all ptrmap pages from source to destination.
          481  +** This routine is only called if the source database is in autovacuum
          482  +** or incremental vacuum mode.
          483  +*/
          484  +static void scrubBackupPtrmap(ScrubState *p){
          485  +  u32 pgno = 2;
          486  +  u32 J = p->szUsable/5;
          487  +  u32 iLock = (1073742335/p->szPage)+1;
          488  +  u8 *a, *pBuf;
          489  +  if( p->rcErr ) return;
          490  +  pBuf = scrubBackupAllocPage(p);
          491  +  if( pBuf==0 ) return;
          492  +  while( pgno<=p->nPage ){
          493  +    a = scrubBackupRead(p, pgno, pBuf);
          494  +    if( a==0 ) break;
          495  +    scrubBackupWrite(p, pgno, a);
          496  +    pgno += J+1;
          497  +    if( pgno==iLock ) pgno++;
          498  +  }
          499  +  sqlite3_free(pBuf);
          500  +}
          501  +
          502  +int sqlite3_scrub_backup(
          503  +  const char *zSrcFile,    /* Source file */
          504  +  const char *zDestFile,   /* Destination file */
          505  +  char **pzErr             /* Write error here if non-NULL */
          506  +){
          507  +  ScrubState s;
          508  +  u32 n, i;
          509  +  sqlite3_stmt *pStmt;
          510  +
          511  +  memset(&s, 0, sizeof(s));
          512  +  s.zSrcFile = zSrcFile;
          513  +  s.zDestFile = zDestFile;
          514  +
          515  +  /* Open both source and destination databases */
          516  +  scrubBackupOpenSrc(&s);
          517  +  scrubBackupOpenDest(&s);
          518  +
          519  +  /* Read in page 1 */
          520  +  s.page1 = scrubBackupRead(&s, 1, 0);
          521  +  if( s.page1==0 ) goto scrub_abort;
          522  +  s.szUsable = s.szPage - s.page1[20];
          523  +
          524  +  /* Copy the freelist */    
          525  +  n = scrubBackupInt32(&s.page1[36]);
          526  +  i = scrubBackupInt32(&s.page1[32]);
          527  +  if( n ) scrubBackupFreelist(&s, i, n);
          528  +
          529  +  /* Copy ptrmap pages */
          530  +  n = scrubBackupInt32(&s.page1[52]);
          531  +  if( n ) scrubBackupPtrmap(&s);
          532  +
          533  +  /* Copy all of the btrees */
          534  +  scrubBackupBtree(&s, 1, 0);
          535  +  pStmt = scrubBackupPrepare(&s, s.dbSrc,
          536  +       "SELECT rootpage FROM sqlite_master WHERE coalesce(rootpage,0)>0");
          537  +  if( pStmt==0 ) goto scrub_abort;
          538  +  while( sqlite3_step(pStmt)==SQLITE_ROW ){
          539  +    i = (u32)sqlite3_column_int(pStmt, 0);
          540  +    scrubBackupBtree(&s, i, 0);
          541  +  }
          542  +  sqlite3_finalize(pStmt);
          543  +
          544  +scrub_abort:    
          545  +  /* Close the destination database without closing the transaction. If we
          546  +  ** commit, page zero will be overwritten. */
          547  +  sqlite3_close(s.dbDest);
          548  +
          549  +  /* But do close out the read-transaction on the source database */
          550  +  sqlite3_exec(s.dbSrc, "COMMIT;", 0, 0, 0);
          551  +  sqlite3_close(s.dbSrc);
          552  +  sqlite3_free(s.page1);
          553  +  if( pzErr ){
          554  +    *pzErr = s.zErr;
          555  +  }else{
          556  +    sqlite3_free(s.zErr);
          557  +  }
          558  +  return s.rcErr;
          559  +}   
          560  +
          561  +#ifdef SCRUB_STANDALONE
          562  +/* Error and warning log */
          563  +static void errorLogCallback(void *pNotUsed, int iErr, const char *zMsg){
          564  +  const char *zType;
          565  +  switch( iErr&0xff ){
          566  +    case SQLITE_WARNING: zType = "WARNING";  break;
          567  +    case SQLITE_NOTICE:  zType = "NOTICE";   break;
          568  +    default:             zType = "ERROR";    break;
          569  +  }
          570  +  fprintf(stderr, "%s: %s\n", zType, zMsg);
          571  +}
          572  +
          573  +/* The main() routine when this utility is run as a stand-alone program */
          574  +int main(int argc, char **argv){
          575  +  char *zErr = 0;
          576  +  int rc;
          577  +  if( argc!=3 ){
          578  +    fprintf(stderr,"Usage: %s SOURCE DESTINATION\n", argv[0]);
          579  +    exit(1);
          580  +  }
          581  +  sqlite3_config(SQLITE_CONFIG_LOG, errorLogCallback, 0);
          582  +  rc = sqlite3_scrub_backup(argv[1], argv[2], &zErr);
          583  +  if( rc==SQLITE_NOMEM ){
          584  +    fprintf(stderr, "%s: out of memory\n", argv[0]);
          585  +    exit(1);
          586  +  }
          587  +  if( zErr ){
          588  +    fprintf(stderr, "%s: %s\n", argv[0], zErr);
          589  +    sqlite3_free(zErr);
          590  +    exit(1);
          591  +  }
          592  +  return 0;
          593  +}
          594  +#endif

Changes to main.mk.

   483    483   	$(TCCX) $(READLINE_FLAGS) -o sqlite3$(EXE) $(SHELL_OPT) \
   484    484   		$(TOP)/src/shell.c libsqlite3.a $(LIBREADLINE) $(TLIBS) $(THREADLIB)
   485    485   
   486    486   sqldiff$(EXE):	$(TOP)/tool/sqldiff.c sqlite3.c sqlite3.h
   487    487   	$(TCCX) -o sqldiff$(EXE) -DSQLITE_THREADSAFE=0 \
   488    488   		$(TOP)/tool/sqldiff.c sqlite3.c $(TLIBS) $(THREADLIB)
   489    489   
          490  +scrub$(EXE):	$(TOP)/ext/misc/scrub.c sqlite3.o
          491  +	$(TCC) -I. -DSCRUB_STANDALONE -o scrub$(EXE) $(TOP)/ext/misc/scrub.c sqlite3.o $(THREADLIB)
          492  +
   490    493   srcck1$(EXE):	$(TOP)/tool/srcck1.c
   491    494   	$(BCC) -o srcck1$(EXE) $(TOP)/tool/srcck1.c
   492    495   
   493    496   sourcetest:	srcck1$(EXE) sqlite3.c
   494    497   	./srcck1 sqlite3.c
   495    498   
   496    499   fuzzershell$(EXE):	$(TOP)/tool/fuzzershell.c sqlite3.c sqlite3.h