SQLite4
Check-in [76297939d3]
Not logged in

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

Overview
Comment:Merge compression-id branch with trunk.
Downloads: Tarball | ZIP archive | SQL archive
Timelines: family | ancestors | descendants | both | trunk
Files: files | file ages | folders
SHA1:76297939d385b893e2de7542da28465328a52b58
User & Date: dan 2013-02-08 11:30:48
Context
2013-02-08
14:39
Changes to lsm_unix.c to build on android: (a) do not use fdatasync() on android and (b) account for the fact that usleep() returns void on android. check-in: 302b222395 user: dan tags: trunk
11:30
Merge compression-id branch with trunk. check-in: 76297939d3 user: dan tags: trunk
2013-02-07
19:50
Add the INFO_COMPRESSION_ID request. And the factory method for providing compression/encryption functions. Leaf check-in: bb85de9cd3 user: dan tags: compression-id
2013-02-06
11:58
Fix bug to do with block redirection. check-in: 7cc153f523 user: dan tags: trunk
Changes
Hide Diffs Unified Diffs Ignore Whitespace Patch

Changes to src/lsm.h.

20
21
22
23
24
25
26

27
28
29
30
31
32
33
...
102
103
104
105
106
107
108


109
110
111
112
113
114
115
...
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
...
251
252
253
254
255
256
257




258
259
260
261
262
263
264
265
266
267
268
269
270
271

272
273
274
275
276
277
278
...
279
280
281
282
283
284
285

286
287








288
289
290
291
292
293
294
...
411
412
413
414
415
416
417





418
419
420
421
422
423
424
425
426
427
428
429
430
431

432
433
434
435
436
437
438
extern "C" {
#endif

/*
** Opaque handle types.
*/
typedef struct lsm_compress lsm_compress;   /* Compression library functions */

typedef struct lsm_cursor lsm_cursor;       /* Database cursor handle */
typedef struct lsm_db lsm_db;               /* Database connection handle */
typedef struct lsm_env lsm_env;             /* Runtime environment */
typedef struct lsm_file lsm_file;           /* OS file handle */
typedef struct lsm_mutex lsm_mutex;         /* Mutex handle */

/* 64-bit integer type used for file offsets. */
................................................................................
#define LSM_IOERR     10
#define LSM_CORRUPT   11
#define LSM_FULL      13
#define LSM_CANTOPEN  14
#define LSM_PROTOCOL  15
#define LSM_MISUSE    21



/* 
** CAPI: Creating and Destroying Database Connection Handles
**
** Open and close a database connection handle.
*/
int lsm_new(lsm_env*, lsm_db **ppDb);
int lsm_close(lsm_db *pDb);
................................................................................

/* 
** CAPI: Connecting to a Database
*/
int lsm_open(lsm_db *pDb, const char *zFilename);

/*
** CAPI: Obtaining pointers to databases environments
**
** Return a pointer to the environment used by the database connection 
** passed as the first argument. Assuming the argument is valid, this 
** function always returns a valid environment pointer - it cannot fail.
*/
lsm_env *lsm_get_env(lsm_db *pDb);

................................................................................
**
**   This option may only be used before lsm_open() is called. Invoking it
**   after lsm_open() has been called results in an LSM_MISUSE error.
**
** LSM_CONFIG_GET_COMPRESSION:
**   Query the compression methods used to compress and decompress database
**   content.




*/
#define LSM_CONFIG_AUTOFLUSH           1
#define LSM_CONFIG_PAGE_SIZE           2
#define LSM_CONFIG_SAFETY              3
#define LSM_CONFIG_BLOCK_SIZE          4
#define LSM_CONFIG_AUTOWORK            5
#define LSM_CONFIG_MMAP                7
#define LSM_CONFIG_USE_LOG             8
#define LSM_CONFIG_AUTOMERGE           9
#define LSM_CONFIG_MAX_FREELIST       10
#define LSM_CONFIG_MULTIPLE_PROCESSES 11
#define LSM_CONFIG_AUTOCHECKPOINT     12
#define LSM_CONFIG_SET_COMPRESSION    13
#define LSM_CONFIG_GET_COMPRESSION    14


#define LSM_SAFETY_OFF    0
#define LSM_SAFETY_NORMAL 1
#define LSM_SAFETY_FULL   2

/*
** CAPI: Compression and/or Encryption Hooks
................................................................................
*/
struct lsm_compress {
  void *pCtx;
  unsigned int iId;
  int (*xBound)(void *, int nSrc);
  int (*xCompress)(void *, char *, int *, const char *, int);
  int (*xUncompress)(void *, char *, int *, const char *, int);

};










/*
** CAPI: Allocating and Freeing Memory
**
** Invoke the memory allocation functions that belong to environment
** pEnv. Or the system defaults if no memory allocation functions have 
** been registered.
................................................................................
**   accumulate new data written to the database. The other tree structure -
**   the old tree - is a read-only tree holding older data and may be flushed 
**   to disk at any time.
** 
**   Assuming no error occurs, the location pointed to by the first of the two
**   (int *) arguments is set to the size of the old in-memory tree in KB.
**   The second is set to the size of the current, or live in-memory tree.





*/
#define LSM_INFO_NWRITE           1
#define LSM_INFO_NREAD            2
#define LSM_INFO_DB_STRUCTURE     3
#define LSM_INFO_LOG_STRUCTURE    4
#define LSM_INFO_ARRAY_STRUCTURE  5
#define LSM_INFO_PAGE_ASCII_DUMP  6
#define LSM_INFO_PAGE_HEX_DUMP    7
#define LSM_INFO_FREELIST         8
#define LSM_INFO_ARRAY_PAGES      9
#define LSM_INFO_CHECKPOINT_SIZE 10
#define LSM_INFO_TREE_SIZE       11

#define LSM_INFO_FREELIST_SIZE   12



/* 
** CAPI: Opening and Closing Write Transactions
**
** These functions are used to open and close transactions and nested 
** sub-transactions.







>







 







>
>







 







|







 







>
>
>
>

|
|
|
|
|
|
|
|
|
|
|
|
|
>







 







>


>
>
>
>
>
>
>
>







 







>
>
>
>
>












<

>







20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
...
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
...
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
...
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
...
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
...
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451

452
453
454
455
456
457
458
459
460
extern "C" {
#endif

/*
** Opaque handle types.
*/
typedef struct lsm_compress lsm_compress;   /* Compression library functions */
typedef struct lsm_compress_factory lsm_compress_factory;
typedef struct lsm_cursor lsm_cursor;       /* Database cursor handle */
typedef struct lsm_db lsm_db;               /* Database connection handle */
typedef struct lsm_env lsm_env;             /* Runtime environment */
typedef struct lsm_file lsm_file;           /* OS file handle */
typedef struct lsm_mutex lsm_mutex;         /* Mutex handle */

/* 64-bit integer type used for file offsets. */
................................................................................
#define LSM_IOERR     10
#define LSM_CORRUPT   11
#define LSM_FULL      13
#define LSM_CANTOPEN  14
#define LSM_PROTOCOL  15
#define LSM_MISUSE    21

#define LSM_MISMATCH  50

/* 
** CAPI: Creating and Destroying Database Connection Handles
**
** Open and close a database connection handle.
*/
int lsm_new(lsm_env*, lsm_db **ppDb);
int lsm_close(lsm_db *pDb);
................................................................................

/* 
** CAPI: Connecting to a Database
*/
int lsm_open(lsm_db *pDb, const char *zFilename);

/*
** CAPI: Obtaining pointers to database environments
**
** Return a pointer to the environment used by the database connection 
** passed as the first argument. Assuming the argument is valid, this 
** function always returns a valid environment pointer - it cannot fail.
*/
lsm_env *lsm_get_env(lsm_db *pDb);

................................................................................
**
**   This option may only be used before lsm_open() is called. Invoking it
**   after lsm_open() has been called results in an LSM_MISUSE error.
**
** LSM_CONFIG_GET_COMPRESSION:
**   Query the compression methods used to compress and decompress database
**   content.
**
** LSM_CONFIG_SET_COMPRESSION_FACTORY:
**   Configure a factory method to be invoked in case of an LSM_MISMATCH
**   error.
*/
#define LSM_CONFIG_AUTOFLUSH                1
#define LSM_CONFIG_PAGE_SIZE                2
#define LSM_CONFIG_SAFETY                   3
#define LSM_CONFIG_BLOCK_SIZE               4
#define LSM_CONFIG_AUTOWORK                 5
#define LSM_CONFIG_MMAP                     7
#define LSM_CONFIG_USE_LOG                  8
#define LSM_CONFIG_AUTOMERGE                9
#define LSM_CONFIG_MAX_FREELIST            10
#define LSM_CONFIG_MULTIPLE_PROCESSES      11
#define LSM_CONFIG_AUTOCHECKPOINT          12
#define LSM_CONFIG_SET_COMPRESSION         13
#define LSM_CONFIG_GET_COMPRESSION         14
#define LSM_CONFIG_SET_COMPRESSION_FACTORY 15

#define LSM_SAFETY_OFF    0
#define LSM_SAFETY_NORMAL 1
#define LSM_SAFETY_FULL   2

/*
** CAPI: Compression and/or Encryption Hooks
................................................................................
*/
struct lsm_compress {
  void *pCtx;
  unsigned int iId;
  int (*xBound)(void *, int nSrc);
  int (*xCompress)(void *, char *, int *, const char *, int);
  int (*xUncompress)(void *, char *, int *, const char *, int);
  void (*xFree)(void *pCtx);
};

struct lsm_compress_factory {
  void *pCtx;
  int (*xFactory)(void *, lsm_db *, unsigned int);
  void (*xFree)(void *pCtx);
};

#define LSM_COMPRESSION_EMPTY 0
#define LSM_COMPRESSION_NONE  1

/*
** CAPI: Allocating and Freeing Memory
**
** Invoke the memory allocation functions that belong to environment
** pEnv. Or the system defaults if no memory allocation functions have 
** been registered.
................................................................................
**   accumulate new data written to the database. The other tree structure -
**   the old tree - is a read-only tree holding older data and may be flushed 
**   to disk at any time.
** 
**   Assuming no error occurs, the location pointed to by the first of the two
**   (int *) arguments is set to the size of the old in-memory tree in KB.
**   The second is set to the size of the current, or live in-memory tree.
**
** LSM_INFO_COMPRESSION_ID:
**   This value should be followed by a single argument of type 
**   (unsigned int *). If successful, the location pointed to is populated 
**   with the database compression id before returning.
*/
#define LSM_INFO_NWRITE           1
#define LSM_INFO_NREAD            2
#define LSM_INFO_DB_STRUCTURE     3
#define LSM_INFO_LOG_STRUCTURE    4
#define LSM_INFO_ARRAY_STRUCTURE  5
#define LSM_INFO_PAGE_ASCII_DUMP  6
#define LSM_INFO_PAGE_HEX_DUMP    7
#define LSM_INFO_FREELIST         8
#define LSM_INFO_ARRAY_PAGES      9
#define LSM_INFO_CHECKPOINT_SIZE 10
#define LSM_INFO_TREE_SIZE       11

#define LSM_INFO_FREELIST_SIZE   12
#define LSM_INFO_COMPRESSION_ID  13


/* 
** CAPI: Opening and Closing Write Transactions
**
** These functions are used to open and close transactions and nested 
** sub-transactions.

Changes to src/lsmInt.h.

312
313
314
315
316
317
318

319
320
321
322
323
324
325
...
332
333
334
335
336
337
338


339
340
341
342
343
344
345
...
522
523
524
525
526
527
528

529
530
531
532
533
534
535
...
567
568
569
570
571
572
573


574
575
576
577
578
579
580
...
585
586
587
588
589
590
591

592
593
594
595
596
597
598
...
639
640
641
642
643
644
645


646
647
648
649
650
651
652
...
888
889
890
891
892
893
894


895
896
897
898
899
900
901
  int nDfltPgsz;                  /* Configured by LSM_CONFIG_PAGE_SIZE */
  int nDfltBlksz;                 /* Configured by LSM_CONFIG_BLOCK_SIZE */
  int nMaxFreelist;               /* Configured by LSM_CONFIG_MAX_FREELIST */
  int bMmap;                      /* Configured by LSM_CONFIG_MMAP */
  i64 nAutockpt;                  /* Configured by LSM_CONFIG_AUTOCHECKPOINT */
  int bMultiProc;                 /* Configured by L_C_MULTIPLE_PROCESSES */
  lsm_compress compress;          /* Compression callbacks */


  /* Sub-system handles */
  FileSystem *pFS;                /* On-disk portion of database */
  Database *pDatabase;            /* Database shared data */

  /* Client transaction context */
  Snapshot *pClient;              /* Client snapshot */
................................................................................
  IntArray rollback;              /* List of tree-nodes to roll back */

  /* Worker context */
  Snapshot *pWorker;              /* Worker snapshot (or NULL) */
  Freelist *pFreelist;            /* See sortedNewToplevel() */
  int bUseFreelist;               /* True to use pFreelist */
  int bIncrMerge;                 /* True if currently doing a merge */



  /* Debugging message callback */
  void (*xLog)(void *, int, const char *);
  void *pLogCtx;

  /* Work done notification callback */
  void (*xWork)(lsm_db *, void *);
................................................................................
/*
** A snapshot of a database. A snapshot contains all the information required
** to read or write a database file on disk. See the description of struct
** Database below for futher details.
*/
struct Snapshot {
  Database *pDatabase;            /* Database this snapshot belongs to */

  Level *pLevel;                  /* Pointer to level 0 of snapshot (or NULL) */
  i64 iId;                        /* Snapshot id */
  i64 iLogOff;                    /* Log file offset */
  Redirect redirect;              /* Block redirection array */

  /* Used by worker snapshots only */
  int nBlock;                     /* Number of blocks in database file */
................................................................................

int lsmCheckpointSaveWorker(lsm_db *pDb, int);
int lsmDatabaseFull(lsm_db *pDb);
int lsmCheckpointSynced(lsm_db *pDb, i64 *piId, i64 *piLog, u32 *pnWrite);

int lsmCheckpointSize(lsm_db *db, int *pnByte);



/* 
** Functions from file "lsm_tree.c".
*/
int lsmTreeNew(lsm_env *, int (*)(void *, int, void *, int), Tree **ppTree);
void lsmTreeRelease(lsm_env *, Tree *);
int lsmTreeInit(lsm_db *);
int lsmTreeRepair(lsm_db *);
................................................................................

int lsmTreeSize(lsm_db *);
int lsmTreeEndTransaction(lsm_db *pDb, int bCommit);
int lsmTreeLoadHeader(lsm_db *pDb, int *);
int lsmTreeLoadHeaderOk(lsm_db *, int);

int lsmTreeInsert(lsm_db *pDb, void *pKey, int nKey, void *pVal, int nVal);

void lsmTreeRollback(lsm_db *pDb, TreeMark *pMark);
void lsmTreeMark(lsm_db *pDb, TreeMark *pMark);

int lsmTreeCursorNew(lsm_db *pDb, int, TreeCursor **);
void lsmTreeCursorDestroy(TreeCursor *);

int lsmTreeCursorSeek(TreeCursor *pCsr, void *pKey, int nKey, int *pRes);
................................................................................
#endif

/**************************************************************************
** Start of functions from "lsm_file.c".
*/
int lsmFsOpen(lsm_db *, const char *);
void lsmFsClose(FileSystem *);



int lsmFsBlockSize(FileSystem *);
void lsmFsSetBlockSize(FileSystem *, int);

int lsmFsPageSize(FileSystem *);
void lsmFsSetPageSize(FileSystem *, int);

................................................................................
int lsmFreelistAppend(lsm_env *pEnv, Freelist *p, int iBlk, i64 iId);

int lsmDbMultiProc(lsm_db *);
void lsmDbDeferredClose(lsm_db *, lsm_file *, LsmFile *);
LsmFile *lsmDbRecycleFd(lsm_db *);

int lsmWalkFreelist(lsm_db *, int, int (*)(void *, int, i64), void *);




/**************************************************************************
** functions in lsm_str.c
*/
void lsmStringInit(LsmString*, lsm_env *pEnv);
int lsmStringExtend(LsmString*, int);







>







 







>
>







 







>







 







>
>







 







>







 







>
>







 







>
>







312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
...
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
...
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
...
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
...
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
...
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
...
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
  int nDfltPgsz;                  /* Configured by LSM_CONFIG_PAGE_SIZE */
  int nDfltBlksz;                 /* Configured by LSM_CONFIG_BLOCK_SIZE */
  int nMaxFreelist;               /* Configured by LSM_CONFIG_MAX_FREELIST */
  int bMmap;                      /* Configured by LSM_CONFIG_MMAP */
  i64 nAutockpt;                  /* Configured by LSM_CONFIG_AUTOCHECKPOINT */
  int bMultiProc;                 /* Configured by L_C_MULTIPLE_PROCESSES */
  lsm_compress compress;          /* Compression callbacks */
  lsm_compress_factory factory;   /* Compression callback factory */

  /* Sub-system handles */
  FileSystem *pFS;                /* On-disk portion of database */
  Database *pDatabase;            /* Database shared data */

  /* Client transaction context */
  Snapshot *pClient;              /* Client snapshot */
................................................................................
  IntArray rollback;              /* List of tree-nodes to roll back */

  /* Worker context */
  Snapshot *pWorker;              /* Worker snapshot (or NULL) */
  Freelist *pFreelist;            /* See sortedNewToplevel() */
  int bUseFreelist;               /* True to use pFreelist */
  int bIncrMerge;                 /* True if currently doing a merge */

  int bInFactory;                 /* True if within factory.xFactory() */

  /* Debugging message callback */
  void (*xLog)(void *, int, const char *);
  void *pLogCtx;

  /* Work done notification callback */
  void (*xWork)(lsm_db *, void *);
................................................................................
/*
** A snapshot of a database. A snapshot contains all the information required
** to read or write a database file on disk. See the description of struct
** Database below for futher details.
*/
struct Snapshot {
  Database *pDatabase;            /* Database this snapshot belongs to */
  u32 iCmpId;                     /* Id of compression scheme */
  Level *pLevel;                  /* Pointer to level 0 of snapshot (or NULL) */
  i64 iId;                        /* Snapshot id */
  i64 iLogOff;                    /* Log file offset */
  Redirect redirect;              /* Block redirection array */

  /* Used by worker snapshots only */
  int nBlock;                     /* Number of blocks in database file */
................................................................................

int lsmCheckpointSaveWorker(lsm_db *pDb, int);
int lsmDatabaseFull(lsm_db *pDb);
int lsmCheckpointSynced(lsm_db *pDb, i64 *piId, i64 *piLog, u32 *pnWrite);

int lsmCheckpointSize(lsm_db *db, int *pnByte);

int lsmInfoCompressionId(lsm_db *db, u32 *piCmpId);

/* 
** Functions from file "lsm_tree.c".
*/
int lsmTreeNew(lsm_env *, int (*)(void *, int, void *, int), Tree **ppTree);
void lsmTreeRelease(lsm_env *, Tree *);
int lsmTreeInit(lsm_db *);
int lsmTreeRepair(lsm_db *);
................................................................................

int lsmTreeSize(lsm_db *);
int lsmTreeEndTransaction(lsm_db *pDb, int bCommit);
int lsmTreeLoadHeader(lsm_db *pDb, int *);
int lsmTreeLoadHeaderOk(lsm_db *, int);

int lsmTreeInsert(lsm_db *pDb, void *pKey, int nKey, void *pVal, int nVal);
int lsmTreeDelete(lsm_db *db, void *pKey1, int nKey1, void *pKey2, int nKey2);
void lsmTreeRollback(lsm_db *pDb, TreeMark *pMark);
void lsmTreeMark(lsm_db *pDb, TreeMark *pMark);

int lsmTreeCursorNew(lsm_db *pDb, int, TreeCursor **);
void lsmTreeCursorDestroy(TreeCursor *);

int lsmTreeCursorSeek(TreeCursor *pCsr, void *pKey, int nKey, int *pRes);
................................................................................
#endif

/**************************************************************************
** Start of functions from "lsm_file.c".
*/
int lsmFsOpen(lsm_db *, const char *);
void lsmFsClose(FileSystem *);

int lsmFsConfigure(lsm_db *db);

int lsmFsBlockSize(FileSystem *);
void lsmFsSetBlockSize(FileSystem *, int);

int lsmFsPageSize(FileSystem *);
void lsmFsSetPageSize(FileSystem *, int);

................................................................................
int lsmFreelistAppend(lsm_env *pEnv, Freelist *p, int iBlk, i64 iId);

int lsmDbMultiProc(lsm_db *);
void lsmDbDeferredClose(lsm_db *, lsm_file *, LsmFile *);
LsmFile *lsmDbRecycleFd(lsm_db *);

int lsmWalkFreelist(lsm_db *, int, int (*)(void *, int, i64), void *);

int lsmCheckCompressionId(lsm_db *, u32);


/**************************************************************************
** functions in lsm_str.c
*/
void lsmStringInit(LsmString*, lsm_env *pEnv);
int lsmStringExtend(LsmString*, int);

Changes to src/lsm_ckpt.c.

28
29
30
31
32
33
34

35
36
37
38
39
40
41
42
43
44
45
46
...
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184

185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
...
445
446
447
448
449
450
451



452
453
454

455
456
457
458
459
460
461
...
757
758
759
760
761
762
763
764
765
766

767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
...
874
875
876
877
878
879
880












881
882
883
884
885
886
887
...
917
918
919
920
921
922
923




924
925
926
927
928
929
930
...
946
947
948
949
950
951
952

953
954
955
956
957
958
959
**
**   Checkpoint header (see the CKPT_HDR_XXX #defines):
**
**     1. The checkpoint id MSW.
**     2. The checkpoint id LSW.
**     3. The number of integer values in the entire checkpoint, including 
**        the two checksum values.

**     4. The total number of blocks in the database.
**     5. The block size.
**     6. The number of levels.
**     7. The nominal database page size.
**     8. The number of pages (in total) written to the database file.
**
**   Log pointer:
**
**     1. The log offset MSW.
**     2. The log offset LSW.
**     3. Log checksum 0.
**     4. Log checksum 1.
................................................................................
 + (((x)&0x00FF0000)>>8)  + (((x)&0xFF000000)>>24) \
)

static const int one = 1;
#define LSM_LITTLE_ENDIAN (*(u8 *)(&one))

/* Sizes, in integers, of various parts of the checkpoint. */
#define CKPT_HDR_SIZE         8
#define CKPT_LOGPTR_SIZE      4
#define CKPT_APPENDLIST_SIZE  (LSM_APPLIST_SZ * 2)

/* A #define to describe each integer in the checkpoint header. */
#define CKPT_HDR_ID_MSW   0
#define CKPT_HDR_ID_LSW   1
#define CKPT_HDR_NCKPT    2

#define CKPT_HDR_NBLOCK   3
#define CKPT_HDR_BLKSZ    4
#define CKPT_HDR_NLEVEL   5
#define CKPT_HDR_PGSZ     6
#define CKPT_HDR_NWRITE   7

#define CKPT_HDR_LO_MSW     8
#define CKPT_HDR_LO_LSW     9
#define CKPT_HDR_LO_CKSUM1 10
#define CKPT_HDR_LO_CKSUM2 11

typedef struct CkptBuffer CkptBuffer;

/*
** Dynamic buffer used to accumulate data for a checkpoint.
*/
struct CkptBuffer {
................................................................................
      ckptSetValue(&ckpt, iOut++, (p->iId >> 32) & 0xFFFFFFFF, &rc);
      ckptSetValue(&ckpt, iOut++, p->iId & 0xFFFFFFFF, &rc);
    }
  }

  /* Write the checkpoint header */
  assert( iId>=0 );



  ckptSetValue(&ckpt, CKPT_HDR_ID_MSW, (u32)(iId>>32), &rc);
  ckptSetValue(&ckpt, CKPT_HDR_ID_LSW, (u32)(iId&0xFFFFFFFF), &rc);
  ckptSetValue(&ckpt, CKPT_HDR_NCKPT, iOut+2, &rc);

  ckptSetValue(&ckpt, CKPT_HDR_NBLOCK, pSnap->nBlock, &rc);
  ckptSetValue(&ckpt, CKPT_HDR_BLKSZ, lsmFsBlockSize(pFS), &rc);
  ckptSetValue(&ckpt, CKPT_HDR_NLEVEL, nLevel, &rc);
  ckptSetValue(&ckpt, CKPT_HDR_PGSZ, lsmFsPageSize(pFS), &rc);
  ckptSetValue(&ckpt, CKPT_HDR_NWRITE, pSnap->nWrite, &rc);

  if( bCksum ){
................................................................................

/*
** Initialize the shared-memory header with an empty snapshot. This function
** is called when no valid snapshot can be found in the database header.
*/
static void ckptLoadEmpty(lsm_db *pDb){
  u32 aCkpt[] = {
    0,                  /* CKPT_HDR_ID_MSW */
    10,                 /* CKPT_HDR_ID_LSW */
    0,                  /* CKPT_HDR_NCKPT */

    0,                  /* CKPT_HDR_NBLOCK */
    0,                  /* CKPT_HDR_BLKSZ */
    0,                  /* CKPT_HDR_NLEVEL */
    0,                  /* CKPT_HDR_PGSZ */
    0,                  /* CKPT_HDR_OVFL */
    0,                  /* CKPT_HDR_NWRITE */
    0, 0, 1234, 5678,   /* The log pointer and initial checksum */
    0,0,0,0, 0,0,0,0,   /* The append list */
    0,                  /* The free block list */
    0, 0                /* Space for checksum values */
  };
  u32 nCkpt = array_size(aCkpt);
  ShmHeader *pShm = pDb->pShmhdr;

  aCkpt[CKPT_HDR_NCKPT] = nCkpt;
  aCkpt[CKPT_HDR_BLKSZ] = pDb->nDfltBlksz;
  aCkpt[CKPT_HDR_PGSZ] = pDb->nDfltPgsz;
................................................................................
      }
    }

    lsmShmBarrier(pDb);
  }
  return LSM_PROTOCOL;
}













int lsmCheckpointLoadOk(lsm_db *pDb, int iSnap){
  u32 *aShm;
  assert( iSnap==1 || iSnap==2 );
  aShm = (iSnap==1) ? pDb->pShmhdr->aSnap1 : pDb->pShmhdr->aSnap2;
  return (lsmCheckpointId(pDb->aSnapshot, 0)==lsmCheckpointId(aShm, 0) );
}
................................................................................
    }else{
      return LSM_PROTOCOL;
    }
  }

  rc = lsmCheckpointDeserialize(pDb, 1, pShm->aSnap1, &pDb->pWorker);
  if( pDb->pWorker ) pDb->pWorker->pDatabase = pDb->pDatabase;





#if 0
  assert( rc!=LSM_OK || lsmFsIntegrityCheck(pDb) );
#endif
  return rc;
}

................................................................................
    int iIn = CKPT_HDR_SIZE + CKPT_APPENDLIST_SIZE + CKPT_LOGPTR_SIZE;

    pNew->iId = lsmCheckpointId(aCkpt, 0);
    pNew->nBlock = aCkpt[CKPT_HDR_NBLOCK];
    pNew->nWrite = aCkpt[CKPT_HDR_NWRITE];
    rc = ckptLoadLevels(pDb, aCkpt, &iIn, nLevel, &pNew->pLevel);
    pNew->iLogOff = lsmCheckpointLogOffset(aCkpt);


    /* Make a copy of the append-list */
    for(i=0; i<LSM_APPLIST_SZ; i++){
      u32 *a = &aCkpt[CKPT_HDR_SIZE + CKPT_LOGPTR_SIZE + i*2];
      pNew->aiAppend[i] = ckptRead64(a);
    }








>
|
|
|
|
|







 







|







>
|
|
|
|
|

|
|
|
|







 







>
>
>



>







 







|
|
|
>
|
|
|
|
|
|
|
|
|
|







 







>
>
>
>
>
>
>
>
>
>
>
>







 







>
>
>
>







 







>







28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
...
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
...
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
...
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
...
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
...
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
...
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
**
**   Checkpoint header (see the CKPT_HDR_XXX #defines):
**
**     1. The checkpoint id MSW.
**     2. The checkpoint id LSW.
**     3. The number of integer values in the entire checkpoint, including 
**        the two checksum values.
**     4. The compression scheme id.
**     5. The total number of blocks in the database.
**     6. The block size.
**     7. The number of levels.
**     8. The nominal database page size.
**     9. The number of pages (in total) written to the database file.
**
**   Log pointer:
**
**     1. The log offset MSW.
**     2. The log offset LSW.
**     3. Log checksum 0.
**     4. Log checksum 1.
................................................................................
 + (((x)&0x00FF0000)>>8)  + (((x)&0xFF000000)>>24) \
)

static const int one = 1;
#define LSM_LITTLE_ENDIAN (*(u8 *)(&one))

/* Sizes, in integers, of various parts of the checkpoint. */
#define CKPT_HDR_SIZE         9
#define CKPT_LOGPTR_SIZE      4
#define CKPT_APPENDLIST_SIZE  (LSM_APPLIST_SZ * 2)

/* A #define to describe each integer in the checkpoint header. */
#define CKPT_HDR_ID_MSW   0
#define CKPT_HDR_ID_LSW   1
#define CKPT_HDR_NCKPT    2
#define CKPT_HDR_CMPID    3
#define CKPT_HDR_NBLOCK   4
#define CKPT_HDR_BLKSZ    5
#define CKPT_HDR_NLEVEL   6
#define CKPT_HDR_PGSZ     7
#define CKPT_HDR_NWRITE   8

#define CKPT_HDR_LO_MSW     9
#define CKPT_HDR_LO_LSW    10
#define CKPT_HDR_LO_CKSUM1 11
#define CKPT_HDR_LO_CKSUM2 12

typedef struct CkptBuffer CkptBuffer;

/*
** Dynamic buffer used to accumulate data for a checkpoint.
*/
struct CkptBuffer {
................................................................................
      ckptSetValue(&ckpt, iOut++, (p->iId >> 32) & 0xFFFFFFFF, &rc);
      ckptSetValue(&ckpt, iOut++, p->iId & 0xFFFFFFFF, &rc);
    }
  }

  /* Write the checkpoint header */
  assert( iId>=0 );
  assert( pSnap->iCmpId==pDb->compress.iId
       || pSnap->iCmpId==LSM_COMPRESSION_EMPTY 
  );
  ckptSetValue(&ckpt, CKPT_HDR_ID_MSW, (u32)(iId>>32), &rc);
  ckptSetValue(&ckpt, CKPT_HDR_ID_LSW, (u32)(iId&0xFFFFFFFF), &rc);
  ckptSetValue(&ckpt, CKPT_HDR_NCKPT, iOut+2, &rc);
  ckptSetValue(&ckpt, CKPT_HDR_CMPID, pDb->compress.iId, &rc);
  ckptSetValue(&ckpt, CKPT_HDR_NBLOCK, pSnap->nBlock, &rc);
  ckptSetValue(&ckpt, CKPT_HDR_BLKSZ, lsmFsBlockSize(pFS), &rc);
  ckptSetValue(&ckpt, CKPT_HDR_NLEVEL, nLevel, &rc);
  ckptSetValue(&ckpt, CKPT_HDR_PGSZ, lsmFsPageSize(pFS), &rc);
  ckptSetValue(&ckpt, CKPT_HDR_NWRITE, pSnap->nWrite, &rc);

  if( bCksum ){
................................................................................

/*
** Initialize the shared-memory header with an empty snapshot. This function
** is called when no valid snapshot can be found in the database header.
*/
static void ckptLoadEmpty(lsm_db *pDb){
  u32 aCkpt[] = {
    0,                       /* CKPT_HDR_ID_MSW */
    10,                      /* CKPT_HDR_ID_LSW */
    0,                       /* CKPT_HDR_NCKPT */
    LSM_COMPRESSION_EMPTY,   /* CKPT_HDR_CMPID */
    0,                       /* CKPT_HDR_NBLOCK */
    0,                       /* CKPT_HDR_BLKSZ */
    0,                       /* CKPT_HDR_NLEVEL */
    0,                       /* CKPT_HDR_PGSZ */
    0,                       /* CKPT_HDR_OVFL */
    0,                       /* CKPT_HDR_NWRITE */
    0, 0, 1234, 5678,        /* The log pointer and initial checksum */
    0,0,0,0, 0,0,0,0,        /* The append list */
    0,                       /* The free block list */
    0, 0                     /* Space for checksum values */
  };
  u32 nCkpt = array_size(aCkpt);
  ShmHeader *pShm = pDb->pShmhdr;

  aCkpt[CKPT_HDR_NCKPT] = nCkpt;
  aCkpt[CKPT_HDR_BLKSZ] = pDb->nDfltBlksz;
  aCkpt[CKPT_HDR_PGSZ] = pDb->nDfltPgsz;
................................................................................
      }
    }

    lsmShmBarrier(pDb);
  }
  return LSM_PROTOCOL;
}

int lsmInfoCompressionId(lsm_db *db, u32 *piCmpId){
  int rc;

  assert( db->pClient==0 && db->pWorker==0 );
  rc = lsmCheckpointLoad(db, 0);
  if( rc==LSM_OK ){
    *piCmpId = db->aSnapshot[CKPT_HDR_CMPID];
  }

  return rc;
}

int lsmCheckpointLoadOk(lsm_db *pDb, int iSnap){
  u32 *aShm;
  assert( iSnap==1 || iSnap==2 );
  aShm = (iSnap==1) ? pDb->pShmhdr->aSnap1 : pDb->pShmhdr->aSnap2;
  return (lsmCheckpointId(pDb->aSnapshot, 0)==lsmCheckpointId(aShm, 0) );
}
................................................................................
    }else{
      return LSM_PROTOCOL;
    }
  }

  rc = lsmCheckpointDeserialize(pDb, 1, pShm->aSnap1, &pDb->pWorker);
  if( pDb->pWorker ) pDb->pWorker->pDatabase = pDb->pDatabase;

  if( rc==LSM_OK ){
    rc = lsmCheckCompressionId(pDb, pDb->pWorker->iCmpId);
  }

#if 0
  assert( rc!=LSM_OK || lsmFsIntegrityCheck(pDb) );
#endif
  return rc;
}

................................................................................
    int iIn = CKPT_HDR_SIZE + CKPT_APPENDLIST_SIZE + CKPT_LOGPTR_SIZE;

    pNew->iId = lsmCheckpointId(aCkpt, 0);
    pNew->nBlock = aCkpt[CKPT_HDR_NBLOCK];
    pNew->nWrite = aCkpt[CKPT_HDR_NWRITE];
    rc = ckptLoadLevels(pDb, aCkpt, &iIn, nLevel, &pNew->pLevel);
    pNew->iLogOff = lsmCheckpointLogOffset(aCkpt);
    pNew->iCmpId = aCkpt[CKPT_HDR_CMPID];

    /* Make a copy of the append-list */
    for(i=0; i<LSM_APPLIST_SZ; i++){
      u32 *a = &aCkpt[CKPT_HDR_SIZE + CKPT_LOGPTR_SIZE + i*2];
      pNew->aiAppend[i] = ckptRead64(a);
    }

Changes to src/lsm_file.c.

506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
...
547
548
549
550
551
552
553


















































554
555
556
557
558
559
560
    pFS->zDb = (char *)&pFS[1];
    pFS->zLog = &pFS->zDb[nDb+1];
    pFS->nPagesize = LSM_DFLT_PAGE_SIZE;
    pFS->nBlocksize = LSM_DFLT_BLOCK_SIZE;
    pFS->nMetasize = 4 * 1024;
    pFS->pDb = pDb;
    pFS->pEnv = pDb->pEnv;
    if( pDb->compress.xCompress ){
      pFS->pCompress = &pDb->compress;
    }else{
      pFS->bUseMmap = pDb->bMmap;
    }

    /* Make a copy of the database and log file names. */
    memcpy(pFS->zDb, zDb, nDb+1);
    memcpy(pFS->zLog, zDb, nDb);
    memcpy(&pFS->zLog[nDb], "-log", 5);

    /* Allocate the hash-table here. At some point, it should be changed
................................................................................
      pFS->szSector = lsmEnvSectorSize(pFS->pEnv, pFS->fdDb);
    }
  }

  pDb->pFS = pFS;
  return rc;
}



















































/*
** Close and destroy a FileSystem object.
*/
void lsmFsClose(FileSystem *pFS){
  if( pFS ){
    Page *pPg;







<
<
<
<
<







 







>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>







506
507
508
509
510
511
512





513
514
515
516
517
518
519
...
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
    pFS->zDb = (char *)&pFS[1];
    pFS->zLog = &pFS->zDb[nDb+1];
    pFS->nPagesize = LSM_DFLT_PAGE_SIZE;
    pFS->nBlocksize = LSM_DFLT_BLOCK_SIZE;
    pFS->nMetasize = 4 * 1024;
    pFS->pDb = pDb;
    pFS->pEnv = pDb->pEnv;






    /* Make a copy of the database and log file names. */
    memcpy(pFS->zDb, zDb, nDb+1);
    memcpy(pFS->zLog, zDb, nDb);
    memcpy(&pFS->zLog[nDb], "-log", 5);

    /* Allocate the hash-table here. At some point, it should be changed
................................................................................
      pFS->szSector = lsmEnvSectorSize(pFS->pEnv, pFS->fdDb);
    }
  }

  pDb->pFS = pFS;
  return rc;
}

/*
** Configure the file-system object according to the current values of
** the LSM_CONFIG_MMAP and LSM_CONFIG_SET_COMPRESSION options.
*/
int lsmFsConfigure(lsm_db *db){
  FileSystem *pFS = db->pFS;
  lsm_env *pEnv = pFS->pEnv;
  Page *pPg;

  assert( pFS->nOut==0 );
  assert( pFS->pWaiting==0 );

  /* Reset any compression/decompression buffers already allocated */
  lsmFree(pEnv, pFS->aIBuffer);
  lsmFree(pEnv, pFS->aOBuffer);
  pFS->nBuffer = 0;

  /* Unmap the file, if it is currently mapped */
  if( pFS->pMap ){
    lsmEnvRemap(pEnv, pFS->fdDb, -1, &pFS->pMap, &pFS->nMap);
    pFS->bUseMmap = 0;
  }

  /* Free all allocate page structures */
  pPg = pFS->pLruFirst;
  while( pPg ){
    Page *pNext = pPg->pLruNext;
    if( pPg->flags & PAGE_FREE ) lsmFree(pEnv, pPg->aData);
    lsmFree(pEnv, pPg);
    pPg = pNext;
  }

  /* Zero pointers that point to deleted page objects */
  pFS->nCacheAlloc = 0;
  pFS->pLruFirst = 0;
  pFS->pLruLast = 0;
  pFS->pFree = 0;

  /* Configure the FileSystem object */
  if( db->compress.xCompress ){
    pFS->pCompress = &db->compress;
    pFS->bUseMmap = 0;
  }else{
    pFS->pCompress = 0;
    pFS->bUseMmap = db->bMmap;
  }

  return LSM_OK;
}

/*
** Close and destroy a FileSystem object.
*/
void lsmFsClose(FileSystem *pFS){
  if( pFS ){
    Page *pPg;

Changes to src/lsm_main.c.

92
93
94
95
96
97
98

99
100
101
102
103
104
105
...
190
191
192
193
194
195
196






197
198
199
200
201
202
203
...
333
334
335
336
337
338
339
340
341
342
343








344
345












346
347
348
349
350
351
352
...
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
...
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
...
564
565
566
567
568
569
570










571
572
573
574
575
576
577
...
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
  pDb->nMerge = LSM_DFLT_AUTOMERGE;
  pDb->nMaxFreelist = LSM_MAX_FREELIST_ENTRIES;
  pDb->bUseLog = LSM_DFLT_USE_LOG;
  pDb->iReader = -1;
  pDb->bMultiProc = LSM_DFLT_MULTIPLE_PROCESSES;
  pDb->bMmap = LSM_DFLT_MMAP;
  pDb->xLog = xLog;

  return LSM_OK;
}

lsm_env *lsm_get_env(lsm_db *pDb){
  assert( pDb->pEnv );
  return pDb->pEnv;
}
................................................................................
      rc = LSM_MISUSE_BKPT;
    }else{
      lsmFreeSnapshot(pDb->pEnv, pDb->pClient);
      pDb->pClient = 0;
      lsmDbDatabaseRelease(pDb);
      lsmLogClose(pDb);
      lsmFsClose(pDb->pFS);






      lsmFree(pDb->pEnv, pDb->rollback.aArray);
      lsmFree(pDb->pEnv, pDb->aTrans);
      lsmFree(pDb->pEnv, pDb->apShm);
      lsmFree(pDb->pEnv, pDb);
    }
  }
  return rc;
................................................................................
        pDb->bMultiProc = *piVal = (*piVal!=0);
      }
      break;
    }

    case LSM_CONFIG_SET_COMPRESSION: {
      lsm_compress *p = va_arg(ap, lsm_compress *);
      if( pDb->pDatabase ){
        /* If lsm_open() has been called, this call is against the rules. */
        rc = LSM_MISUSE_BKPT;
      }else{








        memcpy(&pDb->compress, p, sizeof(lsm_compress));
      }












      break;
    }

    case LSM_CONFIG_GET_COMPRESSION: {
      lsm_compress *p = va_arg(ap, lsm_compress *);
      memcpy(p, &pDb->compress, sizeof(lsm_compress));
      break;
................................................................................
  return 0;
}

int lsmInfoFreelist(lsm_db *pDb, char **pzOut){
  Snapshot *pWorker;              /* Worker snapshot */
  int bUnlock = 0;
  LsmString s;
  int i;
  int rc;

  /* Obtain the worker snapshot */
  rc = infoGetWorker(pDb, &pWorker, &bUnlock);
  if( rc!=LSM_OK ) return rc;

  lsmStringInit(&s, pDb->pEnv);
................................................................................
  }

  /* Release the snapshot and return */
  infoFreeWorker(pDb, bUnlock);
  return rc;
}

static int infoFreelistSize(lsm_db *pDb, int *pnFree, int *pnWaiting){
}

static int infoTreeSize(lsm_db *db, int *pnOldKB, int *pnNewKB){
  ShmHeader *pShm = db->pShmhdr;
  TreeHeader *p = &pShm->hdr1;

  /* The following code suffers from two race conditions, as it accesses and
  ** trusts the contents of shared memory without verifying checksums:
  **
................................................................................

    case LSM_INFO_TREE_SIZE: {
      int *pnOld = va_arg(ap, int *);
      int *pnNew = va_arg(ap, int *);
      rc = infoTreeSize(pDb, pnOld, pnNew);
      break;
    }











    default:
      rc = LSM_MISUSE;
      break;
  }

  va_end(ap);
................................................................................
  assert_db_state( pDb );

  /* A value less than zero means close the innermost nested transaction. */
  if( iLevel<0 ) iLevel = LSM_MAX(0, pDb->nTransOpen - 1);

  if( iLevel<pDb->nTransOpen ){
    if( iLevel==0 ){
      int bAutowork = 0;

      /* Commit the transaction to disk. */
      if( rc==LSM_OK ) rc = lsmLogCommit(pDb);
      if( rc==LSM_OK && pDb->eSafety==LSM_SAFETY_FULL ){
        rc = lsmFsSyncLog(pDb->pFS);
      }
      lsmFinishWriteTrans(pDb, (rc==LSM_OK));
    }







>







 







>
>
>
>
>
>







 







|
|


>
>
>
>
>
>
>
>
|
|
>
>
>
>
>
>
>
>
>
>
>
>







 







<







 







<
<
<







 







>
>
>
>
>
>
>
>
>
>







 







<
<







92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
...
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
...
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
...
455
456
457
458
459
460
461

462
463
464
465
466
467
468
...
474
475
476
477
478
479
480



481
482
483
484
485
486
487
...
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
...
868
869
870
871
872
873
874


875
876
877
878
879
880
881
  pDb->nMerge = LSM_DFLT_AUTOMERGE;
  pDb->nMaxFreelist = LSM_MAX_FREELIST_ENTRIES;
  pDb->bUseLog = LSM_DFLT_USE_LOG;
  pDb->iReader = -1;
  pDb->bMultiProc = LSM_DFLT_MULTIPLE_PROCESSES;
  pDb->bMmap = LSM_DFLT_MMAP;
  pDb->xLog = xLog;
  pDb->compress.iId = LSM_COMPRESSION_NONE;
  return LSM_OK;
}

lsm_env *lsm_get_env(lsm_db *pDb){
  assert( pDb->pEnv );
  return pDb->pEnv;
}
................................................................................
      rc = LSM_MISUSE_BKPT;
    }else{
      lsmFreeSnapshot(pDb->pEnv, pDb->pClient);
      pDb->pClient = 0;
      lsmDbDatabaseRelease(pDb);
      lsmLogClose(pDb);
      lsmFsClose(pDb->pFS);
      
      /* Invoke any destructors registered for the compression or 
      ** compression factory callbacks.  */
      if( pDb->factory.xFree ) pDb->factory.xFree(pDb->factory.pCtx);
      if( pDb->compress.xFree ) pDb->compress.xFree(pDb->compress.pCtx);

      lsmFree(pDb->pEnv, pDb->rollback.aArray);
      lsmFree(pDb->pEnv, pDb->aTrans);
      lsmFree(pDb->pEnv, pDb->apShm);
      lsmFree(pDb->pEnv, pDb);
    }
  }
  return rc;
................................................................................
        pDb->bMultiProc = *piVal = (*piVal!=0);
      }
      break;
    }

    case LSM_CONFIG_SET_COMPRESSION: {
      lsm_compress *p = va_arg(ap, lsm_compress *);
      if( pDb->iReader>=0 && pDb->bInFactory==0 ){
        /* May not change compression schemes with an open transaction */
        rc = LSM_MISUSE_BKPT;
      }else{
        if( pDb->compress.xFree ){
          /* Invoke any destructor belonging to the current compression. */
          pDb->compress.xFree(pDb->compress.pCtx);
        }
        if( p->xBound==0 ){
          memset(&pDb->compress, 0, sizeof(lsm_compress));
          pDb->compress.iId = LSM_COMPRESSION_NONE;
        }else{
          memcpy(&pDb->compress, p, sizeof(lsm_compress));
        }
        rc = lsmFsConfigure(pDb);
      }
      break;
    }

    case LSM_CONFIG_SET_COMPRESSION_FACTORY: {
      lsm_compress_factory *p = va_arg(ap, lsm_compress_factory *);
      if( pDb->factory.xFree ){
        /* Invoke any destructor belonging to the current factory. */
        pDb->factory.xFree(pDb->factory.pCtx);
      }
      memcpy(&pDb->factory, p, sizeof(lsm_compress_factory));
      break;
    }

    case LSM_CONFIG_GET_COMPRESSION: {
      lsm_compress *p = va_arg(ap, lsm_compress *);
      memcpy(p, &pDb->compress, sizeof(lsm_compress));
      break;
................................................................................
  return 0;
}

int lsmInfoFreelist(lsm_db *pDb, char **pzOut){
  Snapshot *pWorker;              /* Worker snapshot */
  int bUnlock = 0;
  LsmString s;

  int rc;

  /* Obtain the worker snapshot */
  rc = infoGetWorker(pDb, &pWorker, &bUnlock);
  if( rc!=LSM_OK ) return rc;

  lsmStringInit(&s, pDb->pEnv);
................................................................................
  }

  /* Release the snapshot and return */
  infoFreeWorker(pDb, bUnlock);
  return rc;
}




static int infoTreeSize(lsm_db *db, int *pnOldKB, int *pnNewKB){
  ShmHeader *pShm = db->pShmhdr;
  TreeHeader *p = &pShm->hdr1;

  /* The following code suffers from two race conditions, as it accesses and
  ** trusts the contents of shared memory without verifying checksums:
  **
................................................................................

    case LSM_INFO_TREE_SIZE: {
      int *pnOld = va_arg(ap, int *);
      int *pnNew = va_arg(ap, int *);
      rc = infoTreeSize(pDb, pnOld, pnNew);
      break;
    }

    case LSM_INFO_COMPRESSION_ID: {
      unsigned int *piOut = va_arg(ap, unsigned int *);
      if( pDb->pClient ){
        *piOut = pDb->pClient->iCmpId;
      }else{
        rc = lsmInfoCompressionId(pDb, piOut);
      }
      break;
    }

    default:
      rc = LSM_MISUSE;
      break;
  }

  va_end(ap);
................................................................................
  assert_db_state( pDb );

  /* A value less than zero means close the innermost nested transaction. */
  if( iLevel<0 ) iLevel = LSM_MAX(0, pDb->nTransOpen - 1);

  if( iLevel<pDb->nTransOpen ){
    if( iLevel==0 ){


      /* Commit the transaction to disk. */
      if( rc==LSM_OK ) rc = lsmLogCommit(pDb);
      if( rc==LSM_OK && pDb->eSafety==LSM_SAFETY_FULL ){
        rc = lsmFsSyncLog(pDb->pFS);
      }
      lsmFinishWriteTrans(pDb, (rc==LSM_OK));
    }

Changes to src/lsm_shared.c.

430
431
432
433
434
435
436



437
438
439
440
441
442
443
...
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923






























924
925
926
927
928
929
930
...
971
972
973
974
975
976
977






978
979
980
981

982
983
984
985
986
987
988
  if( rc==LSM_OK ){
    assert( p );
    rc = lsmFsOpen(pDb, zName);
  }
  if( rc==LSM_OK ){
    rc = doDbConnect(pDb);
  }




  return rc;
}

static void dbDeferClose(lsm_db *pDb){
  if( pDb->pFS ){
    LsmFile *pLsmFile = 0;
................................................................................
    lsmFreeSnapshot(pDb->pEnv, pDb->pWorker);
    pDb->pWorker = 0;
  }

  lsmShmLock(pDb, LSM_LOCK_WORKER, LSM_LOCK_UNLOCK, 0);
}


/*
** Called when recovery is finished.
*/
int lsmFinishRecovery(lsm_db *pDb){
  lsmTreeEndTransaction(pDb, 1);
  return LSM_OK;
}































/*
** Begin a read transaction. This function is a no-op if the connection
** passed as the only argument already has an open read transaction.
*/
int lsmBeginReadTrans(lsm_db *pDb){
  const int MAX_READLOCK_ATTEMPTS = 10;
................................................................................
          ** lsm_sorted.c is changed to work directly from the serialized
          ** version of the snapshot.  */
          if( pDb->pClient==0 ){
            rc = lsmCheckpointDeserialize(pDb, 0, pDb->aSnapshot,&pDb->pClient);
          }
          assert( (rc==LSM_OK)==(pDb->pClient!=0) );
          assert( pDb->iReader>=0 );






        }else{
          rc = lsmReleaseReadlock(pDb);
        }
      }

      if( rc==LSM_BUSY ){
        rc = LSM_OK;
      }
    }
#if 0
if( rc==LSM_OK && pDb->pClient ){
  fprintf(stderr, 







>
>
>







 







<







>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>







 







>
>
>
>
>
>




>







430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
...
912
913
914
915
916
917
918

919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
....
1003
1004
1005
1006
1007
1008
1009
1010
1011
1012
1013
1014
1015
1016
1017
1018
1019
1020
1021
1022
1023
1024
1025
1026
1027
  if( rc==LSM_OK ){
    assert( p );
    rc = lsmFsOpen(pDb, zName);
  }
  if( rc==LSM_OK ){
    rc = doDbConnect(pDb);
  }
  if( rc==LSM_OK ){
    rc = lsmFsConfigure(pDb);
  }

  return rc;
}

static void dbDeferClose(lsm_db *pDb){
  if( pDb->pFS ){
    LsmFile *pLsmFile = 0;
................................................................................
    lsmFreeSnapshot(pDb->pEnv, pDb->pWorker);
    pDb->pWorker = 0;
  }

  lsmShmLock(pDb, LSM_LOCK_WORKER, LSM_LOCK_UNLOCK, 0);
}


/*
** Called when recovery is finished.
*/
int lsmFinishRecovery(lsm_db *pDb){
  lsmTreeEndTransaction(pDb, 1);
  return LSM_OK;
}

/*
** Check if the currently configured compression functions
** (LSM_CONFIG_SET_COMPRESSION) are compatible with a database that has its
** compression id set to iReq. Compression routines are compatible if iReq
** is zero (indicating the database is empty), or if it is equal to the 
** compression id of the configured compression routines.
**
** If the check shows that the current compression are incompatible and there
** is a compression factory registered, give it a chance to install new
** compression routines.
**
** If, after any registered factory is invoked, the compression functions
** are still incompatible, return LSM_MISMATCH. Otherwise, LSM_OK.
*/
int lsmCheckCompressionId(lsm_db *pDb, u32 iReq){
  if( iReq!=LSM_COMPRESSION_EMPTY && pDb->compress.iId!=iReq ){
    if( pDb->factory.xFactory ){
      pDb->bInFactory = 1;
      pDb->factory.xFactory(pDb->factory.pCtx, pDb, iReq);
      pDb->bInFactory = 0;
    }
    if( pDb->compress.iId!=iReq ){
      /* Incompatible */
      return LSM_MISMATCH;
    }
  }
  /* Compatible */
  return LSM_OK;
}

/*
** Begin a read transaction. This function is a no-op if the connection
** passed as the only argument already has an open read transaction.
*/
int lsmBeginReadTrans(lsm_db *pDb){
  const int MAX_READLOCK_ATTEMPTS = 10;
................................................................................
          ** lsm_sorted.c is changed to work directly from the serialized
          ** version of the snapshot.  */
          if( pDb->pClient==0 ){
            rc = lsmCheckpointDeserialize(pDb, 0, pDb->aSnapshot,&pDb->pClient);
          }
          assert( (rc==LSM_OK)==(pDb->pClient!=0) );
          assert( pDb->iReader>=0 );

          /* Check that the client has the right compression hooks loaded.
          ** If not, set rc to LSM_MISMATCH.  */
          if( rc==LSM_OK ){
            rc = lsmCheckCompressionId(pDb, pDb->pClient->iCmpId);
          }
        }else{
          rc = lsmReleaseReadlock(pDb);
        }
      }

      if( rc==LSM_BUSY ){
        rc = LSM_OK;
      }
    }
#if 0
if( rc==LSM_OK && pDb->pClient ){
  fprintf(stderr, 

Changes to src/lsm_unix.c.

190
191
192
193
194
195
196

197
198
199
200
201
202
203
204
205
206
207
208
209

210
211
212
213
214
215
216

  if( p->pMap ){
    munmap(p->pMap, p->nMap);
    *ppOut = p->pMap = 0;
    *pnOut = p->nMap = 0;
  }


  memset(&buf, 0, sizeof(buf));
  prc = fstat(p->fd, &buf);
  if( prc!=0 ) return LSM_IOERR_BKPT;
  iSz = buf.st_size;
  if( iSz<iMin ){
    iSz = ((iMin + (2<<20) - 1) / (2<<20)) * (2<<20);
    prc = ftruncate(p->fd, iSz);
    if( prc!=0 ) return LSM_IOERR_BKPT;
  }

  p->pMap = mmap(0, iSz, PROT_READ|PROT_WRITE, MAP_SHARED, p->fd, 0);
  p->nMap = iSz;


  *ppOut = p->pMap;
  *pnOut = p->nMap;
  return LSM_OK;
}

static int lsmPosixOsFullpath(
  lsm_env *pEnv,







>
|
|
|
|
|
|
|
|
|

|
|
|
>







190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218

  if( p->pMap ){
    munmap(p->pMap, p->nMap);
    *ppOut = p->pMap = 0;
    *pnOut = p->nMap = 0;
  }

  if( iMin>=0 ){
    memset(&buf, 0, sizeof(buf));
    prc = fstat(p->fd, &buf);
    if( prc!=0 ) return LSM_IOERR_BKPT;
    iSz = buf.st_size;
    if( iSz<iMin ){
      iSz = ((iMin + (2<<20) - 1) / (2<<20)) * (2<<20);
      prc = ftruncate(p->fd, iSz);
      if( prc!=0 ) return LSM_IOERR_BKPT;
    }

    p->pMap = mmap(0, iSz, PROT_READ|PROT_WRITE, MAP_SHARED, p->fd, 0);
    p->nMap = iSz;
  }

  *ppOut = p->pMap;
  *pnOut = p->nMap;
  return LSM_OK;
}

static int lsmPosixOsFullpath(
  lsm_env *pEnv,

Added test/lsm4.test.













































































































































































































































>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
# 2013 February 06
#
# The author disclaims copyright to this source code.  In place of
# a legal notice, here is a blessing:
#
#    May you do good and not evil.
#    May you find forgiveness for yourself and forgive others.
#    May you share freely, never taking more than you give.
#
#***********************************************************************
#

set testdir [file dirname $argv0]
source $testdir/tester.tcl
set testprefix lsm4
db close

# Compression scheme ids (defined in test_lsm.c):
#
set compression_id(encrypt) 43
set compression_id(rle)     44
set compression_id(noop)    45

proc db_fetch {db key} {
  db csr_open csr
  csr seek $key eq
  set ret [csr value]
  csr close
  set ret
}

do_test 1.1 {
  lsm_open db test.db 
  db config {set_compression noop}
  db write 1 abc
  db write 2 def
  db close
} {}

do_test 1.2 {
  lsm_open db test.db 
  db config {set_compression noop}
  list [db_fetch db 1] [db_fetch db 2]
} {abc def}

do_test 1.3 {
  db close
  lsm_open db test.db 
  db config {set_compression rle}
  list [catch {db_fetch db 1} msg] $msg
} {1 {error in lsm_csr_open() - 50}}

do_test 1.4 {
  db close
  lsm_open db test.db 
  list [catch {db_fetch db 1} msg] $msg
} {1 {error in lsm_csr_open() - 50}}

do_test 1.5 {
  db config {set_compression_factory true}
  list [db_fetch db 1] [db_fetch db 2]
} {abc def}

do_test 1.6 { db info compression_id } $compression_id(noop)
db close

#-------------------------------------------------------------------------
#
forcedelete test.db

do_test 2.1 {
  lsm_open db test.db
  db info compression_id
} {0}

do_test 2.2 {
  db write 1 abc
  db write 2 abc
  db info compression_id
} {0}

do_test 2.3 {
  lsm_open db2 test.db
  db2 info compression_id
} {0}

do_test 2.4 {
  db close
  db2 info compression_id
} {0}

do_test 2.5 {
  db2 close
  lsm_open db test.db
  db info compression_id
} {1}

db close
forcedelete test.db

do_test 2.6 {
  lsm_open db test.db
  db config {set_compression rle}
  db write 3 three
  db write 4 four
  db close

  lsm_open db test.db
  db info compression_id
} $compression_id(rle)

do_test 2.7 {
  db config {set_compression rle}
  list [db_fetch db 3] [db_fetch db 4]
} {three four}

finish_test

Changes to test/test_lsm.c.

5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22































































































































23
24
25
26
27
28
29
..
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
...
282
283
284
285
286
287
288
289




































































290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306


307
308
309
310
311
312
313
...
321
322
323
324
325
326
327







328
329
330
331


332
333
334
335
336
337
338

339
340
341
342
343
344
345
...
371
372
373
374
375
376
377
































378
379
380
381
382
383
384
...
517
518
519
520
521
522
523

524
525
526
527
528
529
530
...
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
...
652
653
654
655
656
657
658




659
660
661
662
663
664
665
** a legal notice, here is a blessing:
**
**    May you do good and not evil.
**    May you find forgiveness for yourself and forgive others.
**    May you share freely, never taking more than you give.
**
*************************************************************************
**
*/

#include <tcl.h>
#include "lsm.h"
#include "sqlite4.h"
#include <assert.h>
#include <string.h>

extern int getDbPointer(Tcl_Interp *interp, const char *zA, sqlite4 **ppDb);
extern const char *sqlite4TestErrorName(int);
































































































































/*
** TCLCMD:    sqlite4_lsm_config DB DBNAME PARAM ...
*/
static int test_sqlite4_lsm_config(
  void * clientData,
  Tcl_Interp *interp,
................................................................................
  int objc,
  Tcl_Obj *CONST objv[]
){
  struct Switch {
    const char *zSwitch;
    int iVal;
  } aParam[] = {
    { "log-size",       LSM_CONFIG_LOG_SIZE }, 
    { "safety",         LSM_CONFIG_SAFETY }, 
    { "autoflush",      LSM_CONFIG_AUTOFLUSH }, 
    { "mmap",           LSM_CONFIG_MMAP }, 
    { "page-size",      LSM_CONFIG_PAGE_SIZE }, 
    { "autowork",       LSM_CONFIG_AUTOWORK }, 
    { 0, 0 }
  };
................................................................................
    Tcl_SetResult(interp, (char *)sqlite4TestErrorName(rc), TCL_STATIC);
    return TCL_ERROR;
  }

  Tcl_ResetResult(interp);
  return TCL_OK;
}





































































static int testConfigureLsm(Tcl_Interp *interp, lsm_db *db, Tcl_Obj *pObj){
  struct Lsmconfig {
    const char *zOpt;
    int eOpt;
  } aConfig[] = {
    { "autoflush",        LSM_CONFIG_AUTOFLUSH },
    { "page_size",        LSM_CONFIG_PAGE_SIZE },
    { "block_size",       LSM_CONFIG_BLOCK_SIZE },
    { "safety",           LSM_CONFIG_SAFETY },
    { "autowork",         LSM_CONFIG_AUTOWORK },
    { "autocheckpoint",   LSM_CONFIG_AUTOCHECKPOINT },
    { "log_size",         LSM_CONFIG_LOG_SIZE },
    { "mmap",             LSM_CONFIG_MMAP },
    { "use_log",          LSM_CONFIG_USE_LOG },
    { "automerge",        LSM_CONFIG_AUTOMERGE },
    { "max_freelist",     LSM_CONFIG_MAX_FREELIST },
    { "multi_proc",       LSM_CONFIG_MULTIPLE_PROCESSES },


    { 0, 0 }
  };
  int nElem;
  int i;
  Tcl_Obj **apElem;
  int rc;

................................................................................
      if( i==(nElem-1) ){
        Tcl_ResetResult(interp);
        Tcl_AppendResult(interp, "option \"", Tcl_GetString(apElem[i]), 
            "\" requires an argument", 0
            );
        rc = TCL_ERROR;
      }else{







        int iVal;
        rc = Tcl_GetIntFromObj(interp, apElem[i+1], &iVal);
        if( rc==TCL_OK ){
          lsm_config(db, aConfig[iOpt].eOpt, &iVal);


        }
      }
    }
  }

  return rc;
}


typedef struct TclLsmCursor TclLsmCursor;
typedef struct TclLsm TclLsm;

struct TclLsm {
  lsm_db *db;
};
................................................................................
static void test_lsm_del(void *ctx){
  TclLsm *p = (TclLsm *)ctx;
  if( p ){
    lsm_close(p->db);
    ckfree((char *)p);
  }
}

































/*
** Usage: CSR sub-command ...
*/
static int test_lsm_cursor_cmd(
  void * clientData,
  Tcl_Interp *interp,
................................................................................
    /*  5 */ {"commit",       1, "LEVEL"},
    /*  6 */ {"rollback",     1, "LEVEL"},
    /*  7 */ {"csr_open",     1, "CSR"},
    /*  8 */ {"work",        -1, "?NMERGE? NPAGE"},
    /*  9 */ {"flush",        0, ""},
    /* 10 */ {"config",       1, "LIST"},
    /* 11 */ {"checkpoint",   0, ""},

    {0, 0, 0}
  };
  int iCmd;
  int rc;
  TclLsm *p = (TclLsm *)clientData;

  if( objc<2 ){
................................................................................
      return TCL_OK;
    }

    case 8: assert( 0==strcmp(aCmd[8].zCmd, "work") ); {
      int nWork = 0;
      int nMerge = 1;
      int nWrite = 0;
      int i;

      if( objc==3 ){
        rc = Tcl_GetIntFromObj(interp, objv[2], &nWork);
      }else if( objc==4 ){
        rc = Tcl_GetIntFromObj(interp, objv[2], &nMerge);
        if( rc!=TCL_OK ) return rc;
        rc = Tcl_GetIntFromObj(interp, objv[3], &nWork);
................................................................................
      return testConfigureLsm(interp, p->db, objv[2]);
    }

    case 11: assert( 0==strcmp(aCmd[11].zCmd, "checkpoint") ); {
      rc = lsm_checkpoint(p->db, 0);
      return test_lsm_error(interp, "lsm_checkpoint", rc);
    }





    default:
      assert( 0 );
  }

  Tcl_AppendResult(interp, "internal error", 0);
  return TCL_ERROR;







<










>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>







 







<







 








>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>





|
|
|
|
|
|
<
|
|
|
|
|
>
>







 







>
>
>
>
>
>
>
|
|
|
|
>
>







>







 







>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>







 







>







 







<







 







>
>
>
>







5
6
7
8
9
10
11

12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
...
156
157
158
159
160
161
162

163
164
165
166
167
168
169
...
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493

494
495
496
497
498
499
500
501
502
503
504
505
506
507
...
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
...
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
...
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
...
856
857
858
859
860
861
862

863
864
865
866
867
868
869
...
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
** a legal notice, here is a blessing:
**
**    May you do good and not evil.
**    May you find forgiveness for yourself and forgive others.
**    May you share freely, never taking more than you give.
**
*************************************************************************

*/

#include <tcl.h>
#include "lsm.h"
#include "sqlite4.h"
#include <assert.h>
#include <string.h>

extern int getDbPointer(Tcl_Interp *interp, const char *zA, sqlite4 **ppDb);
extern const char *sqlite4TestErrorName(int);

/*************************************************************************
*/
#define ENCRYPTION_XOR_MASK 0x23b2bbb6
static int testCompressEncBound(void *pCtx, int nSrc){
  return nSrc;
}
static int testCompressEncCompress(
  void *pCtx, 
  char *pOut, int *pnOut, 
  const char *pIn, int nIn
){
  int i;
  unsigned int *aIn = (unsigned int *)pOut;
  unsigned int *aOut = (unsigned int *)pIn;

  assert( (nIn%4)==0 );
  for(i=0; i<(nIn/4); i++){
    aOut[i] = (aIn[i] ^ ENCRYPTION_XOR_MASK);
  }
  *pnOut = nIn;

  return LSM_OK;
}
static int testCompressEncUncompress(
  void *pCtx, 
  char *pOut, int *pnOut, 
  const char *pIn, int nIn
){
  return testCompressEncCompress(pCtx, pOut, pnOut, pIn, nIn);
}
static void testCompressEncFree(void *pCtx){
  /* no-op */
}
/* 
** End of compression routines "encrypt".
*************************************************************************/

/*************************************************************************
*/
static int testCompressRleBound(void *pCtx, int nSrc){
  return nSrc*2;
}
static int testCompressRleCompress(
  void *pCtx, 
  char *pOut, int *pnOut, 
  const char *pIn, int nIn
){
  int iOut = 0;
  int i;
  char c;
  int n;

  c = pIn[0];
  n = 1;
  for(i=1; i<nIn; i++){
    if( pIn[i]==c && n<127 ){
      n++;
    }else{
      pOut[iOut++] = c;
      pOut[iOut++] = (char)n;
      c = pIn[i];
      n = 1;
    }
  }

  pOut[iOut++] = c;
  pOut[iOut++] = (char)n;
  *pnOut = iOut;

  return LSM_OK;
}
static int testCompressRleUncompress(
  void *pCtx, 
  char *pOut, int *pnOut, 
  const char *pIn, int nIn
){
  int i;
  int iOut = 0;

  for(i=0; i<nIn; i+=2){
    int iRep;
    char c = pIn[i];
    int n = (int)(pIn[i+1]);

    for(iRep=0; iRep<n; iRep++){
      pOut[iOut++] = c;
    }
  }

  *pnOut = iOut;
  return LSM_OK;
}
static void testCompressRleFree(void *pCtx){
}
/* 
** End of compression routines "rle".
*************************************************************************/

/*************************************************************************
*/
static int testCompressNoopBound(void *pCtx, int nSrc){
  return nSrc;
}
static int testCompressNoopCompress(
  void *pCtx, 
  char *pOut, int *pnOut, 
  const char *pIn, int nIn
){
  *pnOut = nIn;
  memcpy(pOut, pIn, nIn);
  return LSM_OK;
}
static int testCompressNoopUncompress(
  void *pCtx, 
  char *pOut, int *pnOut, 
  const char *pIn, int nIn
){
  *pnOut = nIn;
  memcpy(pOut, pIn, nIn);
  return LSM_OK;
}
static void testCompressNoopFree(void *pCtx){
}
/* 
** End of compression routines "noop".
*************************************************************************/

/*
** TCLCMD:    sqlite4_lsm_config DB DBNAME PARAM ...
*/
static int test_sqlite4_lsm_config(
  void * clientData,
  Tcl_Interp *interp,
................................................................................
  int objc,
  Tcl_Obj *CONST objv[]
){
  struct Switch {
    const char *zSwitch;
    int iVal;
  } aParam[] = {

    { "safety",         LSM_CONFIG_SAFETY }, 
    { "autoflush",      LSM_CONFIG_AUTOFLUSH }, 
    { "mmap",           LSM_CONFIG_MMAP }, 
    { "page-size",      LSM_CONFIG_PAGE_SIZE }, 
    { "autowork",       LSM_CONFIG_AUTOWORK }, 
    { 0, 0 }
  };
................................................................................
    Tcl_SetResult(interp, (char *)sqlite4TestErrorName(rc), TCL_STATIC);
    return TCL_ERROR;
  }

  Tcl_ResetResult(interp);
  return TCL_OK;
}

static int testConfigureSetCompression(
  Tcl_Interp *interp, 
  lsm_db *db, 
  Tcl_Obj *pCmp,
  unsigned int iId
){
  struct CompressionScheme {
    const char *zName;
    lsm_compress cmp;
  } aCmp[] = {
    { "encrypt", { 0, 43, 
        testCompressEncBound, testCompressEncCompress,
        testCompressEncUncompress, testCompressEncFree
    } },
    { "rle", { 0, 44, 
        testCompressRleBound, testCompressRleCompress,
        testCompressRleUncompress, testCompressRleFree
    } },
    { "noop", { 0, 45, 
        testCompressNoopBound, testCompressNoopCompress,
        testCompressNoopUncompress, testCompressNoopFree
    } },
    { 0, {0, 0, 0, 0, 0, 0} }
  };
  int iOpt;
  int rc;

  if( interp ){
    rc = Tcl_GetIndexFromObjStruct(
        interp, pCmp, aCmp, sizeof(aCmp[0]), "scheme", 0, &iOpt
        );
    if( rc!=TCL_OK ) return rc;
  }else{
    int nOpt = sizeof(aCmp)/sizeof(aCmp[0]);
    for(iOpt=0; iOpt<nOpt; iOpt++){
      if( iId==aCmp[iOpt].cmp.iId ) break;
    }
    if( iOpt==nOpt ) return 0;
  }

  rc = lsm_config(db, LSM_CONFIG_SET_COMPRESSION, &aCmp[iOpt].cmp);
  return rc;
}

static int testCompressFactory(void *pCtx, lsm_db *db, unsigned int iId){
  return testConfigureSetCompression(0, db, 0, iId);
}

static int testConfigureSetFactory(
  Tcl_Interp *interp, 
  lsm_db *db, 
  Tcl_Obj *pArg
){
  lsm_compress_factory aFactory[2] = {
    { 0, 0, 0 },
    { 0, testCompressFactory, 0 },
  };
  int bArg = 0;
  int rc;

  rc = Tcl_GetBooleanFromObj(interp, pArg, &bArg);
  if( rc!=TCL_OK ) return rc;
  assert( bArg==1 || bArg==0 );

  rc = lsm_config(db, LSM_CONFIG_SET_COMPRESSION_FACTORY, &aFactory[bArg]);
  return rc;
}

static int testConfigureLsm(Tcl_Interp *interp, lsm_db *db, Tcl_Obj *pObj){
  struct Lsmconfig {
    const char *zOpt;
    int eOpt;
  } aConfig[] = {
    { "autoflush",               LSM_CONFIG_AUTOFLUSH },
    { "page_size",               LSM_CONFIG_PAGE_SIZE },
    { "block_size",              LSM_CONFIG_BLOCK_SIZE },
    { "safety",                  LSM_CONFIG_SAFETY },
    { "autowork",                LSM_CONFIG_AUTOWORK },
    { "autocheckpoint",          LSM_CONFIG_AUTOCHECKPOINT },

    { "mmap",                    LSM_CONFIG_MMAP },
    { "use_log",                 LSM_CONFIG_USE_LOG },
    { "automerge",               LSM_CONFIG_AUTOMERGE },
    { "max_freelist",            LSM_CONFIG_MAX_FREELIST },
    { "multi_proc",              LSM_CONFIG_MULTIPLE_PROCESSES },
    { "set_compression",         LSM_CONFIG_SET_COMPRESSION },
    { "set_compression_factory", LSM_CONFIG_SET_COMPRESSION_FACTORY },
    { 0, 0 }
  };
  int nElem;
  int i;
  Tcl_Obj **apElem;
  int rc;

................................................................................
      if( i==(nElem-1) ){
        Tcl_ResetResult(interp);
        Tcl_AppendResult(interp, "option \"", Tcl_GetString(apElem[i]), 
            "\" requires an argument", 0
            );
        rc = TCL_ERROR;
      }else{
        if( aConfig[iOpt].eOpt==LSM_CONFIG_SET_COMPRESSION ){
          rc = testConfigureSetCompression(interp, db, apElem[i+1], 0);
        }
        else if( aConfig[iOpt].eOpt==LSM_CONFIG_SET_COMPRESSION_FACTORY ){
          rc = testConfigureSetFactory(interp, db, apElem[i+1]);
        }
        else {
          int iVal;
          rc = Tcl_GetIntFromObj(interp, apElem[i+1], &iVal);
          if( rc==TCL_OK ){
            lsm_config(db, aConfig[iOpt].eOpt, &iVal);
          }
          Tcl_SetObjResult(interp, Tcl_NewIntObj(iVal));
        }
      }
    }
  }

  return rc;
}


typedef struct TclLsmCursor TclLsmCursor;
typedef struct TclLsm TclLsm;

struct TclLsm {
  lsm_db *db;
};
................................................................................
static void test_lsm_del(void *ctx){
  TclLsm *p = (TclLsm *)ctx;
  if( p ){
    lsm_close(p->db);
    ckfree((char *)p);
  }
}

static int testInfoLsm(Tcl_Interp *interp, lsm_db *db, Tcl_Obj *pObj){
  struct Lsminfo {
    const char *zOpt;
    int eOpt;
  } aInfo[] = {
    { "compression_id",          LSM_INFO_COMPRESSION_ID },
    { 0, 0 }
  };
  int rc;
  int iOpt;

  rc = Tcl_GetIndexFromObjStruct(
      interp, pObj, aInfo, sizeof(aInfo[0]), "option", 0, &iOpt
  );
  if( rc==LSM_OK ){
    switch( aInfo[iOpt].eOpt ){
      case LSM_INFO_COMPRESSION_ID: {
        unsigned int iCmpId = 0;
        rc = lsm_info(db, LSM_INFO_COMPRESSION_ID, &iCmpId);
        if( rc==LSM_OK ){
          Tcl_SetObjResult(interp, Tcl_NewWideIntObj((Tcl_WideInt)iCmpId));
        }else{
          test_lsm_error(interp, "lsm_info", rc);
        }
        break;
      }
    }
  }

  return rc;
}

/*
** Usage: CSR sub-command ...
*/
static int test_lsm_cursor_cmd(
  void * clientData,
  Tcl_Interp *interp,
................................................................................
    /*  5 */ {"commit",       1, "LEVEL"},
    /*  6 */ {"rollback",     1, "LEVEL"},
    /*  7 */ {"csr_open",     1, "CSR"},
    /*  8 */ {"work",        -1, "?NMERGE? NPAGE"},
    /*  9 */ {"flush",        0, ""},
    /* 10 */ {"config",       1, "LIST"},
    /* 11 */ {"checkpoint",   0, ""},
    /* 12 */ {"info",         1, "OPTION"},
    {0, 0, 0}
  };
  int iCmd;
  int rc;
  TclLsm *p = (TclLsm *)clientData;

  if( objc<2 ){
................................................................................
      return TCL_OK;
    }

    case 8: assert( 0==strcmp(aCmd[8].zCmd, "work") ); {
      int nWork = 0;
      int nMerge = 1;
      int nWrite = 0;


      if( objc==3 ){
        rc = Tcl_GetIntFromObj(interp, objv[2], &nWork);
      }else if( objc==4 ){
        rc = Tcl_GetIntFromObj(interp, objv[2], &nMerge);
        if( rc!=TCL_OK ) return rc;
        rc = Tcl_GetIntFromObj(interp, objv[3], &nWork);
................................................................................
      return testConfigureLsm(interp, p->db, objv[2]);
    }

    case 11: assert( 0==strcmp(aCmd[11].zCmd, "checkpoint") ); {
      rc = lsm_checkpoint(p->db, 0);
      return test_lsm_error(interp, "lsm_checkpoint", rc);
    }

    case 12: assert( 0==strcmp(aCmd[12].zCmd, "info") ); {
      return testInfoLsm(interp, p->db, objv[2]);
    }

    default:
      assert( 0 );
  }

  Tcl_AppendResult(interp, "internal error", 0);
  return TCL_ERROR;

Changes to www/lsmapi.wiki.

17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
..
63
64
65
66
67
68
69

70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87

88
89
90
91
92
93
94
95
96

97
98
99
100
101
102
103
...
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209

210
211
212
213
214
215
216
217
218








219
220
221
222
223







224
225
226
227
228
229
230
231
232
233
234
235
236











237
238
239
240
241
242
243
...
255
256
257
258
259
260
261


262
263
264
265
266
267
268






269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291

292
293
294
295
296
297
298
...
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
...
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429

<h1>LSM API Topics</h1>
<ol>
<li><a href="#database" style=text-decoration:none>Database Runtime Environment</a>
<li><a href="#lsm" style=text-decoration:none>LSM Error Codes</a>
<li><a href="#creating" style=text-decoration:none>Creating and Destroying Database Connection Handles</a>
<li><a href="#connecting" style=text-decoration:none>Connecting to a Database</a>
<li><a href="#obtaining" style=text-decoration:none>Obtaining pointers to databases environments</a>
<li><a href="#configuring" style=text-decoration:none>Configuring a database connection.</a>
<li><a href="#compression" style=text-decoration:none>Compression and/or Encryption Hooks</a>
<li><a href="#allocating" style=text-decoration:none>Allocating and Freeing Memory</a>
<li><a href="#querying" style=text-decoration:none>Querying a Connection For Operational Data</a>
<li><a href="#opening" style=text-decoration:none>Opening and Closing Write Transactions</a>
<li><a href="#writing" style=text-decoration:none>Writing to a Database</a>
<li><a href="#explicit" style=text-decoration:none>Explicit Database Work and Checkpointing</a>
................................................................................
<span style=display:block;float:left;width:35ex><a href=#lsm_new>lsm_new</a></span>
<span style=display:block;float:left;width:35ex><a href=#lsm_open>lsm_open</a></span>
<span style=display:block;float:left;width:35ex><a href=#lsm_rollback>lsm_rollback</a></span>
<span style=display:block;float:left;width:35ex><a href=#lsm_work>lsm_work</a></span>
<br style=clear:both>
<h1 style=clear:both>All LSM API Types</h1>
<span style=display:block;float:left;width:35ex><a href=#lsm_compress>lsm_compress</a></span>

<span style=display:block;float:left;width:35ex><a href=#lsm_env>lsm_env</a></span>
<br style=clear:both>
<h1>All LSM API Constants</h1>
<span style=display:block;float:left;width:35ex><a href=#LSM_BUSY>LSM_BUSY</a></span>
<span style=display:block;float:left;width:35ex><a href=#LSM_CANTOPEN>LSM_CANTOPEN</a></span>
<span style=display:block;float:left;width:35ex><a href=#LSM_CONFIG_AUTOCHECKPOINT>LSM_CONFIG_AUTOCHECKPOINT</a></span>
<span style=display:block;float:left;width:35ex><a href=#LSM_CONFIG_AUTOFLUSH>LSM_CONFIG_AUTOFLUSH</a></span>
<span style=display:block;float:left;width:35ex><a href=#LSM_CONFIG_AUTOMERGE>LSM_CONFIG_AUTOMERGE</a></span>
<span style=display:block;float:left;width:35ex><a href=#LSM_CONFIG_AUTOWORK>LSM_CONFIG_AUTOWORK</a></span>
<span style=display:block;float:left;width:35ex><a href=#LSM_CONFIG_BLOCK_SIZE>LSM_CONFIG_BLOCK_SIZE</a></span>
<span style=display:block;float:left;width:35ex><a href=#LSM_CONFIG_GET_COMPRESSION>LSM_CONFIG_GET_COMPRESSION</a></span>
<span style=display:block;float:left;width:35ex><a href=#LSM_CONFIG_LOG_SIZE>LSM_CONFIG_LOG_SIZE</a></span>
<span style=display:block;float:left;width:35ex><a href=#LSM_CONFIG_MAX_FREELIST>LSM_CONFIG_MAX_FREELIST</a></span>
<span style=display:block;float:left;width:35ex><a href=#LSM_CONFIG_MMAP>LSM_CONFIG_MMAP</a></span>
<span style=display:block;float:left;width:35ex><a href=#LSM_CONFIG_MULTIPLE_PROCESSES>LSM_CONFIG_MULTIPLE_PROCESSES</a></span>
<span style=display:block;float:left;width:35ex><a href=#LSM_CONFIG_PAGE_SIZE>LSM_CONFIG_PAGE_SIZE</a></span>
<span style=display:block;float:left;width:35ex><a href=#LSM_CONFIG_SAFETY>LSM_CONFIG_SAFETY</a></span>
<span style=display:block;float:left;width:35ex><a href=#LSM_CONFIG_SET_COMPRESSION>LSM_CONFIG_SET_COMPRESSION</a></span>

<span style=display:block;float:left;width:35ex><a href=#LSM_CONFIG_USE_LOG>LSM_CONFIG_USE_LOG</a></span>
<span style=display:block;float:left;width:35ex><a href=#LSM_CORRUPT>LSM_CORRUPT</a></span>
<span style=display:block;float:left;width:35ex><a href=#LSM_ERROR>LSM_ERROR</a></span>
<span style=display:block;float:left;width:35ex><a href=#LSM_FULL>LSM_FULL</a></span>
<span style=display:block;float:left;width:35ex><a href=#LSM_INFO_ARRAY_PAGES>LSM_INFO_ARRAY_PAGES</a></span>
<span style=display:block;float:left;width:35ex><a href=#LSM_INFO_ARRAY_STRUCTURE>LSM_INFO_ARRAY_STRUCTURE</a></span>
<span style=display:block;float:left;width:35ex><a href=#LSM_INFO_CHECKPOINT_SIZE>LSM_INFO_CHECKPOINT_SIZE</a></span>
<span style=display:block;float:left;width:35ex><a href=#LSM_INFO_DB_STRUCTURE>LSM_INFO_DB_STRUCTURE</a></span>
<span style=display:block;float:left;width:35ex><a href=#LSM_INFO_FREELIST>LSM_INFO_FREELIST</a></span>

<span style=display:block;float:left;width:35ex><a href=#LSM_INFO_LOG_STRUCTURE>LSM_INFO_LOG_STRUCTURE</a></span>
<span style=display:block;float:left;width:35ex><a href=#LSM_INFO_NREAD>LSM_INFO_NREAD</a></span>
<span style=display:block;float:left;width:35ex><a href=#LSM_INFO_NWRITE>LSM_INFO_NWRITE</a></span>
<span style=display:block;float:left;width:35ex><a href=#LSM_INFO_PAGE_ASCII_DUMP>LSM_INFO_PAGE_ASCII_DUMP</a></span>
<span style=display:block;float:left;width:35ex><a href=#LSM_INFO_PAGE_HEX_DUMP>LSM_INFO_PAGE_HEX_DUMP</a></span>
<span style=display:block;float:left;width:35ex><a href=#LSM_INFO_TREE_SIZE>LSM_INFO_TREE_SIZE</a></span>
<span style=display:block;float:left;width:35ex><a href=#LSM_IOERR>LSM_IOERR</a></span>
................................................................................
<verbatim>int lsm_new(lsm_env*, lsm_db **ppDb);
int lsm_close(lsm_db *pDb);
</verbatim>
<p>Open and close a database connection handle.
<h2 id=connecting>Connecting to a Database<a id=lsm_open></a></h2>
<verbatim>int lsm_open(lsm_db *pDb, const char *zFilename);
</verbatim>
<h2 id=obtaining>Obtaining pointers to databases environments</h2>
<verbatim>lsm_env *lsm_get_env(lsm_db *pDb);
lsm_env *lsm_default_env(void);
</verbatim>
<p>Return a pointer to the environment used by the database connection 
passed as the first argument. Assuming the argument is valid, this 
function always returns a valid environment pointer - it cannot fail.
The lsm_default_env() function returns a pointer to the default LSM
environment for the current platform.
<h2 id=configuring>Configuring a database connection.<a id=lsm_config></a><a id=LSM_CONFIG_AUTOFLUSH></a><a id=LSM_CONFIG_PAGE_SIZE></a><a id=LSM_CONFIG_SAFETY></a><a id=LSM_CONFIG_BLOCK_SIZE></a><a id=LSM_CONFIG_AUTOWORK></a><a id=LSM_CONFIG_LOG_SIZE></a><a id=LSM_CONFIG_MMAP></a><a id=LSM_CONFIG_USE_LOG></a><a id=LSM_CONFIG_AUTOMERGE></a><a id=LSM_CONFIG_MAX_FREELIST></a><a id=LSM_CONFIG_MULTIPLE_PROCESSES></a><a id=LSM_CONFIG_AUTOCHECKPOINT></a><a id=LSM_CONFIG_SET_COMPRESSION></a><a id=LSM_CONFIG_GET_COMPRESSION></a><a id=LSM_SAFETY_OFF></a><a id=LSM_SAFETY_NORMAL></a><a id=LSM_SAFETY_FULL></a></h2>
<verbatim>int lsm_config(lsm_db *, int, ...);
#define LSM_CONFIG_AUTOFLUSH           1
#define LSM_CONFIG_PAGE_SIZE           2
#define LSM_CONFIG_SAFETY              3
#define LSM_CONFIG_BLOCK_SIZE          4
#define LSM_CONFIG_AUTOWORK            5
#define LSM_CONFIG_LOG_SIZE            6
#define LSM_CONFIG_MMAP                7
#define LSM_CONFIG_USE_LOG             8
#define LSM_CONFIG_AUTOMERGE           9
#define LSM_CONFIG_MAX_FREELIST       10
#define LSM_CONFIG_MULTIPLE_PROCESSES 11
#define LSM_CONFIG_AUTOCHECKPOINT     12
#define LSM_CONFIG_SET_COMPRESSION    13
#define LSM_CONFIG_GET_COMPRESSION    14

#define LSM_SAFETY_OFF    0
#define LSM_SAFETY_NORMAL 1
#define LSM_SAFETY_FULL   2
</verbatim>
<p>The lsm_config() function is used to configure a database connection.
The following values may be passed as the second argument to lsm_config().
<p><dl><dt>LSM_CONFIG_AUTOFLUSH<dd>A read/write integer parameter. This value determines the maximum amount
of space (in bytes) used to accumulate writes in main-memory before 
they are flushed to a level 0 segment.








<p><dt>LSM_CONFIG_PAGE_SIZE<dd>A read/write integer parameter. This parameter may only be set before
lsm_open() has been called.
<p><dt>LSM_CONFIG_BLOCK_SIZE<dd>A read/write integer parameter. This parameter may only be set before
lsm_open() has been called.
<p><dt>LSM_CONFIG_LOG_SIZE<dd>A read/write integer parameter.







<p><dt>LSM_CONFIG_SAFETY<dd>A read/write integer parameter. Valid values are 0, 1 (the default) 
and 2. This parameter determines how robust the database is in the
face of a system crash (e.g. a power failure or operating system 
crash). As follows:
<p>0 (off):    No robustness. A system crash may corrupt the database.
<p>1 (normal): Some robustness. A system crash may not corrupt the
database file, but recently committed transactions may
be lost following recovery.
<p>2 (full):   Full robustness. A system crash may not corrupt the
database file. Following recovery the database file
contains all successfully committed transactions.
<p><dt>LSM_CONFIG_AUTOWORK<dd>A read/write integer parameter.
<p><dt>LSM_CONFIG_AUTOCHECKPOINT<dd>A read/write integer parameter.











<p><dt>LSM_CONFIG_MMAP<dd>A read/write integer parameter. True to use mmap() to access the 
database file. False otherwise.
<p><dt>LSM_CONFIG_USE_LOG<dd>A read/write boolean parameter. True (the default) to use the log
file normally. False otherwise.
<p><dt>LSM_CONFIG_AUTOMERGE<dd>A read/write integer parameter. The minimum number of segments to
merge together at a time. Default value 4.
<p><dt>LSM_CONFIG_MAX_FREELIST<dd>A read/write integer parameter. The maximum number of free-list 
................................................................................
content. The argument to this option should be a pointer to a structure
of type lsm_compress. The lsm_config() method takes a copy of the 
structures contents.
<p>This option may only be used before lsm_open() is called. Invoking it
after lsm_open() has been called results in an LSM_MISUSE error.
<p><dt>LSM_CONFIG_GET_COMPRESSION<dd>Query the compression methods used to compress and decompress database
content.


</dl><h2 id=compression>Compression and/or Encryption Hooks<a id=lsm_compress></a></h2>
<verbatim>struct lsm_compress {
  void *pCtx;
  unsigned int iId;
  int (*xBound)(void *, int nSrc);
  int (*xCompress)(void *, char *, int *, const char *, int);
  int (*xUncompress)(void *, char *, int *, const char *, int);






};
</verbatim>
<h2 id=allocating>Allocating and Freeing Memory<a id=lsm_free></a></h2>
<verbatim>void *lsm_malloc(lsm_env*, size_t);
void *lsm_realloc(lsm_env*, void *, size_t);
void lsm_free(lsm_env*, void *);
</verbatim>
<p>Invoke the memory allocation functions that belong to environment
pEnv. Or the system defaults if no memory allocation functions have 
been registered.
<h2 id=querying>Querying a Connection For Operational Data<a id=lsm_info></a><a id=LSM_INFO_NWRITE></a><a id=LSM_INFO_NREAD></a><a id=LSM_INFO_DB_STRUCTURE></a><a id=LSM_INFO_LOG_STRUCTURE></a><a id=LSM_INFO_ARRAY_STRUCTURE></a><a id=LSM_INFO_PAGE_ASCII_DUMP></a><a id=LSM_INFO_PAGE_HEX_DUMP></a><a id=LSM_INFO_FREELIST></a><a id=LSM_INFO_ARRAY_PAGES></a><a id=LSM_INFO_CHECKPOINT_SIZE></a><a id=LSM_INFO_TREE_SIZE></a></h2>
<verbatim>int lsm_info(lsm_db *, int, ...);
#define LSM_INFO_NWRITE           1
#define LSM_INFO_NREAD            2
#define LSM_INFO_DB_STRUCTURE     3
#define LSM_INFO_LOG_STRUCTURE    4
#define LSM_INFO_ARRAY_STRUCTURE  5
#define LSM_INFO_PAGE_ASCII_DUMP  6
#define LSM_INFO_PAGE_HEX_DUMP    7
#define LSM_INFO_FREELIST         8
#define LSM_INFO_ARRAY_PAGES      9
#define LSM_INFO_CHECKPOINT_SIZE 10
#define LSM_INFO_TREE_SIZE       11

</verbatim>
<p>Query a database connection for operational statistics or data.
The following values may be passed as the second argument to lsm_info().
<p><dl><dt>LSM_INFO_NWRITE<dd>The third parameter should be of type (int *). The location pointed
to by the third parameter is set to the number of 4KB pages written to
the database file during the lifetime of this connection. 
<p><dt>LSM_INFO_NREAD<dd>The third parameter should be of type (int *). The location pointed
................................................................................
to is populated with a pointer to a nul-terminated string containing
the string representation of a Tcl data-structure. The returned 
string should be eventually freed by the caller using lsm_free().
<p>The Tcl structure returned is a list containing one element for each
free block in the database. The element itself consists of two 
integers - the block number and the id of the snapshot that freed it.
<p><dt>LSM_INFO_CHECKPOINT_SIZE<dd>The third argument should be of type (int *). The location pointed to
by this argument is populated with the number of bytes written to the
database file since the most recent checkpoint.
<p><dt>LSM_INFO_TREE_SIZE<dd>If this value is passed as the second argument to an lsm_info() call, it
should be followed by two arguments of type (int *) (for a total of four
arguments).
<p>At any time, there are either one or two tree structures held in shared
memory that new database clients will access (there may also be additional
tree structures being used by older clients - this API does not provide
information on them). One tree structure - the current tree - is used to
accumulate new data written to the database. The other tree structure -
the old tree - is a read-only tree holding older data and may be flushed 
to disk at any time.
<p>Assuming no error occurs, the location pointed to by the first of the two
(int *) arguments is set to the size of the old in-memory tree in bytes.
The second is set to the size of the current, or live in-memory tree.
</dl><h2 id=opening>Opening and Closing Write Transactions<a id=lsm_begin></a><a id=lsm_commit></a><a id=lsm_rollback></a></h2>
<verbatim>int lsm_begin(lsm_db *pDb, int iLevel);
int lsm_commit(lsm_db *pDb, int iLevel);
int lsm_rollback(lsm_db *pDb, int iLevel);
</verbatim>
<p>These functions are used to open and close transactions and nested 
................................................................................
Delete a value from the database. No error is returned if the specified
key value does not exist in the database.
Delete all database entries with keys that are greater than (pKey1/nKey1) 
and smaller than (pKey2/nKey2). Note that keys (pKey1/nKey1) and
(pKey2/nKey2) themselves, if they exist in the database, are not deleted.
<p>Return LSM_OK if successful, or an LSM error code otherwise.
<h2 id=explicit>Explicit Database Work and Checkpointing<a id=lsm_work></a><a id=lsm_flush></a><a id=lsm_checkpoint></a></h2>
<verbatim>int lsm_work(lsm_db *pDb, int nMerge, int nPage, int *pnWrite);
int lsm_flush(lsm_db *pDb);
int lsm_checkpoint(lsm_db *pDb, int *pnByte);
</verbatim>
<p>This function is called by a thread to work on the database structure.
Attempt to checkpoint the current database snapshot. Return an LSM
error code if an error occurs or LSM_OK otherwise.
<p>If the current snapshot has already been checkpointed, calling this 
function is a no-op. In this case if pnByte is not NULL, *pnByte is
set to 0. Or, if the current snapshot is successfully checkpointed
by this function and pbCkpt is not NULL, *pnByte is set to the number
of bytes written to the database file since the previous checkpoint
(the same measure as returned by the LSM_INFO_CHECKPOINT_SIZE query).
<h2 id=opening>Opening and Closing Database Cursors<a id=lsm_csr_open></a><a id=lsm_csr_close></a></h2>
<verbatim>int lsm_csr_open(lsm_db *pDb, lsm_cursor **ppCsr);
int lsm_csr_close(lsm_cursor *pCsr);
</verbatim>
<p>Open and close a database cursor.







|







 







>











<






>









>







 







|








|

|
|
|
|
|
<
|
|
|
|
|
|
|
|
>






|
|
|
>
>
>
>
>
>
>
>


|
|
<
>
>
>
>
>
>
>













>
>
>
>
>
>
>
>
>
>
>







 







>
>
|






>
>
>
>
>
>










|












>







 







|












|







 







|

|





|

|







17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
..
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81

82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
...
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202

203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232

233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
...
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
...
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
...
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465

<h1>LSM API Topics</h1>
<ol>
<li><a href="#database" style=text-decoration:none>Database Runtime Environment</a>
<li><a href="#lsm" style=text-decoration:none>LSM Error Codes</a>
<li><a href="#creating" style=text-decoration:none>Creating and Destroying Database Connection Handles</a>
<li><a href="#connecting" style=text-decoration:none>Connecting to a Database</a>
<li><a href="#obtaining" style=text-decoration:none>Obtaining pointers to database environments</a>
<li><a href="#configuring" style=text-decoration:none>Configuring a database connection.</a>
<li><a href="#compression" style=text-decoration:none>Compression and/or Encryption Hooks</a>
<li><a href="#allocating" style=text-decoration:none>Allocating and Freeing Memory</a>
<li><a href="#querying" style=text-decoration:none>Querying a Connection For Operational Data</a>
<li><a href="#opening" style=text-decoration:none>Opening and Closing Write Transactions</a>
<li><a href="#writing" style=text-decoration:none>Writing to a Database</a>
<li><a href="#explicit" style=text-decoration:none>Explicit Database Work and Checkpointing</a>
................................................................................
<span style=display:block;float:left;width:35ex><a href=#lsm_new>lsm_new</a></span>
<span style=display:block;float:left;width:35ex><a href=#lsm_open>lsm_open</a></span>
<span style=display:block;float:left;width:35ex><a href=#lsm_rollback>lsm_rollback</a></span>
<span style=display:block;float:left;width:35ex><a href=#lsm_work>lsm_work</a></span>
<br style=clear:both>
<h1 style=clear:both>All LSM API Types</h1>
<span style=display:block;float:left;width:35ex><a href=#lsm_compress>lsm_compress</a></span>
<span style=display:block;float:left;width:35ex><a href=#lsm_compress>lsm_compress</a></span>
<span style=display:block;float:left;width:35ex><a href=#lsm_env>lsm_env</a></span>
<br style=clear:both>
<h1>All LSM API Constants</h1>
<span style=display:block;float:left;width:35ex><a href=#LSM_BUSY>LSM_BUSY</a></span>
<span style=display:block;float:left;width:35ex><a href=#LSM_CANTOPEN>LSM_CANTOPEN</a></span>
<span style=display:block;float:left;width:35ex><a href=#LSM_CONFIG_AUTOCHECKPOINT>LSM_CONFIG_AUTOCHECKPOINT</a></span>
<span style=display:block;float:left;width:35ex><a href=#LSM_CONFIG_AUTOFLUSH>LSM_CONFIG_AUTOFLUSH</a></span>
<span style=display:block;float:left;width:35ex><a href=#LSM_CONFIG_AUTOMERGE>LSM_CONFIG_AUTOMERGE</a></span>
<span style=display:block;float:left;width:35ex><a href=#LSM_CONFIG_AUTOWORK>LSM_CONFIG_AUTOWORK</a></span>
<span style=display:block;float:left;width:35ex><a href=#LSM_CONFIG_BLOCK_SIZE>LSM_CONFIG_BLOCK_SIZE</a></span>
<span style=display:block;float:left;width:35ex><a href=#LSM_CONFIG_GET_COMPRESSION>LSM_CONFIG_GET_COMPRESSION</a></span>

<span style=display:block;float:left;width:35ex><a href=#LSM_CONFIG_MAX_FREELIST>LSM_CONFIG_MAX_FREELIST</a></span>
<span style=display:block;float:left;width:35ex><a href=#LSM_CONFIG_MMAP>LSM_CONFIG_MMAP</a></span>
<span style=display:block;float:left;width:35ex><a href=#LSM_CONFIG_MULTIPLE_PROCESSES>LSM_CONFIG_MULTIPLE_PROCESSES</a></span>
<span style=display:block;float:left;width:35ex><a href=#LSM_CONFIG_PAGE_SIZE>LSM_CONFIG_PAGE_SIZE</a></span>
<span style=display:block;float:left;width:35ex><a href=#LSM_CONFIG_SAFETY>LSM_CONFIG_SAFETY</a></span>
<span style=display:block;float:left;width:35ex><a href=#LSM_CONFIG_SET_COMPRESSION>LSM_CONFIG_SET_COMPRESSION</a></span>
<span style=display:block;float:left;width:35ex><a href=#LSM_CONFIG_SET_COMPRESSION_FACTORY>LSM_CONFIG_SET_COMPRESSION_FACTORY</a></span>
<span style=display:block;float:left;width:35ex><a href=#LSM_CONFIG_USE_LOG>LSM_CONFIG_USE_LOG</a></span>
<span style=display:block;float:left;width:35ex><a href=#LSM_CORRUPT>LSM_CORRUPT</a></span>
<span style=display:block;float:left;width:35ex><a href=#LSM_ERROR>LSM_ERROR</a></span>
<span style=display:block;float:left;width:35ex><a href=#LSM_FULL>LSM_FULL</a></span>
<span style=display:block;float:left;width:35ex><a href=#LSM_INFO_ARRAY_PAGES>LSM_INFO_ARRAY_PAGES</a></span>
<span style=display:block;float:left;width:35ex><a href=#LSM_INFO_ARRAY_STRUCTURE>LSM_INFO_ARRAY_STRUCTURE</a></span>
<span style=display:block;float:left;width:35ex><a href=#LSM_INFO_CHECKPOINT_SIZE>LSM_INFO_CHECKPOINT_SIZE</a></span>
<span style=display:block;float:left;width:35ex><a href=#LSM_INFO_DB_STRUCTURE>LSM_INFO_DB_STRUCTURE</a></span>
<span style=display:block;float:left;width:35ex><a href=#LSM_INFO_FREELIST>LSM_INFO_FREELIST</a></span>
<span style=display:block;float:left;width:35ex><a href=#LSM_INFO_FREELIST_SIZE>LSM_INFO_FREELIST_SIZE</a></span>
<span style=display:block;float:left;width:35ex><a href=#LSM_INFO_LOG_STRUCTURE>LSM_INFO_LOG_STRUCTURE</a></span>
<span style=display:block;float:left;width:35ex><a href=#LSM_INFO_NREAD>LSM_INFO_NREAD</a></span>
<span style=display:block;float:left;width:35ex><a href=#LSM_INFO_NWRITE>LSM_INFO_NWRITE</a></span>
<span style=display:block;float:left;width:35ex><a href=#LSM_INFO_PAGE_ASCII_DUMP>LSM_INFO_PAGE_ASCII_DUMP</a></span>
<span style=display:block;float:left;width:35ex><a href=#LSM_INFO_PAGE_HEX_DUMP>LSM_INFO_PAGE_HEX_DUMP</a></span>
<span style=display:block;float:left;width:35ex><a href=#LSM_INFO_TREE_SIZE>LSM_INFO_TREE_SIZE</a></span>
<span style=display:block;float:left;width:35ex><a href=#LSM_IOERR>LSM_IOERR</a></span>
................................................................................
<verbatim>int lsm_new(lsm_env*, lsm_db **ppDb);
int lsm_close(lsm_db *pDb);
</verbatim>
<p>Open and close a database connection handle.
<h2 id=connecting>Connecting to a Database<a id=lsm_open></a></h2>
<verbatim>int lsm_open(lsm_db *pDb, const char *zFilename);
</verbatim>
<h2 id=obtaining>Obtaining pointers to database environments</h2>
<verbatim>lsm_env *lsm_get_env(lsm_db *pDb);
lsm_env *lsm_default_env(void);
</verbatim>
<p>Return a pointer to the environment used by the database connection 
passed as the first argument. Assuming the argument is valid, this 
function always returns a valid environment pointer - it cannot fail.
The lsm_default_env() function returns a pointer to the default LSM
environment for the current platform.
<h2 id=configuring>Configuring a database connection.<a id=lsm_config></a><a id=LSM_CONFIG_AUTOFLUSH></a><a id=LSM_CONFIG_PAGE_SIZE></a><a id=LSM_CONFIG_SAFETY></a><a id=LSM_CONFIG_BLOCK_SIZE></a><a id=LSM_CONFIG_AUTOWORK></a><a id=LSM_CONFIG_MMAP></a><a id=LSM_CONFIG_USE_LOG></a><a id=LSM_CONFIG_AUTOMERGE></a><a id=LSM_CONFIG_MAX_FREELIST></a><a id=LSM_CONFIG_MULTIPLE_PROCESSES></a><a id=LSM_CONFIG_AUTOCHECKPOINT></a><a id=LSM_CONFIG_SET_COMPRESSION></a><a id=LSM_CONFIG_GET_COMPRESSION></a><a id=LSM_CONFIG_SET_COMPRESSION_FACTORY></a><a id=LSM_SAFETY_OFF></a><a id=LSM_SAFETY_NORMAL></a><a id=LSM_SAFETY_FULL></a></h2>
<verbatim>int lsm_config(lsm_db *, int, ...);
#define LSM_CONFIG_AUTOFLUSH                1
#define LSM_CONFIG_PAGE_SIZE                2
#define LSM_CONFIG_SAFETY                   3
#define LSM_CONFIG_BLOCK_SIZE               4
#define LSM_CONFIG_AUTOWORK                 5

#define LSM_CONFIG_MMAP                     7
#define LSM_CONFIG_USE_LOG                  8
#define LSM_CONFIG_AUTOMERGE                9
#define LSM_CONFIG_MAX_FREELIST            10
#define LSM_CONFIG_MULTIPLE_PROCESSES      11
#define LSM_CONFIG_AUTOCHECKPOINT          12
#define LSM_CONFIG_SET_COMPRESSION         13
#define LSM_CONFIG_GET_COMPRESSION         14
#define LSM_CONFIG_SET_COMPRESSION_FACTORY 15
#define LSM_SAFETY_OFF    0
#define LSM_SAFETY_NORMAL 1
#define LSM_SAFETY_FULL   2
</verbatim>
<p>The lsm_config() function is used to configure a database connection.
The following values may be passed as the second argument to lsm_config().
<p><dl><dt>LSM_CONFIG_AUTOFLUSH<dd>A read/write integer parameter. 
<p>This value determines the amount of data allowed to accumulate in a
live in-memory tree before it is marked as old. After committing a
transaction, a connection checks if the size of the live in-memory tree,
including data structure overhead, is greater than the value of this
option in KB. If it is, and there is not already an old in-memory tree,
the live in-memory tree is marked as old.
<p>The maximum allowable value is 1048576 (1GB). There is no minimum 
value. If this parameter is set to zero, then an attempt is made to
mark the live in-memory tree as old after each transaction is committed.
<p>The default value is 1024 (1MB).
<p><dt>LSM_CONFIG_PAGE_SIZE<dd>A read/write integer parameter. This parameter may only be set before
lsm_open() has been called.
<p><dt>LSM_CONFIG_BLOCK_SIZE<dd>A read/write integer parameter. 
<p>This parameter may only be set before lsm_open() has been called. It

must be set to a power of two between 64 and 65536, inclusive (block 
sizes between 64KB and 64MB).
<p>If the connection creates a new database, the block size of the new
database is set to the value of this option in KB. After lsm_open()
has been called, querying this parameter returns the actual block
size of the opened database.
<p>The default value is 1024 (1MB blocks).
<p><dt>LSM_CONFIG_SAFETY<dd>A read/write integer parameter. Valid values are 0, 1 (the default) 
and 2. This parameter determines how robust the database is in the
face of a system crash (e.g. a power failure or operating system 
crash). As follows:
<p>0 (off):    No robustness. A system crash may corrupt the database.
<p>1 (normal): Some robustness. A system crash may not corrupt the
database file, but recently committed transactions may
be lost following recovery.
<p>2 (full):   Full robustness. A system crash may not corrupt the
database file. Following recovery the database file
contains all successfully committed transactions.
<p><dt>LSM_CONFIG_AUTOWORK<dd>A read/write integer parameter.
<p><dt>LSM_CONFIG_AUTOCHECKPOINT<dd>A read/write integer parameter.
<p>If this option is set to non-zero value N, then a checkpoint is
automatically attempted after each N KB of data have been written to 
the database file.
<p>The amount of uncheckpointed data already written to the database file
is a global parameter. After performing database work (writing to the
database file), the process checks if the total amount of uncheckpointed 
data exceeds the value of this paramter. If so, a checkpoint is performed.
This means that this option may cause the connection to perform a 
checkpoint even if the current connection has itself written very little
data into the database file.
<p>The default value is 2048 (checkpoint every 2MB).
<p><dt>LSM_CONFIG_MMAP<dd>A read/write integer parameter. True to use mmap() to access the 
database file. False otherwise.
<p><dt>LSM_CONFIG_USE_LOG<dd>A read/write boolean parameter. True (the default) to use the log
file normally. False otherwise.
<p><dt>LSM_CONFIG_AUTOMERGE<dd>A read/write integer parameter. The minimum number of segments to
merge together at a time. Default value 4.
<p><dt>LSM_CONFIG_MAX_FREELIST<dd>A read/write integer parameter. The maximum number of free-list 
................................................................................
content. The argument to this option should be a pointer to a structure
of type lsm_compress. The lsm_config() method takes a copy of the 
structures contents.
<p>This option may only be used before lsm_open() is called. Invoking it
after lsm_open() has been called results in an LSM_MISUSE error.
<p><dt>LSM_CONFIG_GET_COMPRESSION<dd>Query the compression methods used to compress and decompress database
content.
<p><dt>LSM_CONFIG_SET_COMPRESSION_FACTORY<dd>Configure a factory method to be invoked in case of an LSM_MISMATCH
error.
</dl><h2 id=compression>Compression and/or Encryption Hooks<a id=lsm_compress></a><a id=lsm_compress></a></h2>
<verbatim>struct lsm_compress {
  void *pCtx;
  unsigned int iId;
  int (*xBound)(void *, int nSrc);
  int (*xCompress)(void *, char *, int *, const char *, int);
  int (*xUncompress)(void *, char *, int *, const char *, int);
  void (*xFree)(void *pCtx);
};
struct lsm_compress_factory {
  void *pCtx;
  int (*xFactory)(void *, lsm_db *, u32);
  void (*xFree)(void *pCtx);
};
</verbatim>
<h2 id=allocating>Allocating and Freeing Memory<a id=lsm_free></a></h2>
<verbatim>void *lsm_malloc(lsm_env*, size_t);
void *lsm_realloc(lsm_env*, void *, size_t);
void lsm_free(lsm_env*, void *);
</verbatim>
<p>Invoke the memory allocation functions that belong to environment
pEnv. Or the system defaults if no memory allocation functions have 
been registered.
<h2 id=querying>Querying a Connection For Operational Data<a id=lsm_info></a><a id=LSM_INFO_NWRITE></a><a id=LSM_INFO_NREAD></a><a id=LSM_INFO_DB_STRUCTURE></a><a id=LSM_INFO_LOG_STRUCTURE></a><a id=LSM_INFO_ARRAY_STRUCTURE></a><a id=LSM_INFO_PAGE_ASCII_DUMP></a><a id=LSM_INFO_PAGE_HEX_DUMP></a><a id=LSM_INFO_FREELIST></a><a id=LSM_INFO_ARRAY_PAGES></a><a id=LSM_INFO_CHECKPOINT_SIZE></a><a id=LSM_INFO_TREE_SIZE></a><a id=LSM_INFO_FREELIST_SIZE></a></h2>
<verbatim>int lsm_info(lsm_db *, int, ...);
#define LSM_INFO_NWRITE           1
#define LSM_INFO_NREAD            2
#define LSM_INFO_DB_STRUCTURE     3
#define LSM_INFO_LOG_STRUCTURE    4
#define LSM_INFO_ARRAY_STRUCTURE  5
#define LSM_INFO_PAGE_ASCII_DUMP  6
#define LSM_INFO_PAGE_HEX_DUMP    7
#define LSM_INFO_FREELIST         8
#define LSM_INFO_ARRAY_PAGES      9
#define LSM_INFO_CHECKPOINT_SIZE 10
#define LSM_INFO_TREE_SIZE       11
#define LSM_INFO_FREELIST_SIZE   12
</verbatim>
<p>Query a database connection for operational statistics or data.
The following values may be passed as the second argument to lsm_info().
<p><dl><dt>LSM_INFO_NWRITE<dd>The third parameter should be of type (int *). The location pointed
to by the third parameter is set to the number of 4KB pages written to
the database file during the lifetime of this connection. 
<p><dt>LSM_INFO_NREAD<dd>The third parameter should be of type (int *). The location pointed
................................................................................
to is populated with a pointer to a nul-terminated string containing
the string representation of a Tcl data-structure. The returned 
string should be eventually freed by the caller using lsm_free().
<p>The Tcl structure returned is a list containing one element for each
free block in the database. The element itself consists of two 
integers - the block number and the id of the snapshot that freed it.
<p><dt>LSM_INFO_CHECKPOINT_SIZE<dd>The third argument should be of type (int *). The location pointed to
by this argument is populated with the number of KB written to the
database file since the most recent checkpoint.
<p><dt>LSM_INFO_TREE_SIZE<dd>If this value is passed as the second argument to an lsm_info() call, it
should be followed by two arguments of type (int *) (for a total of four
arguments).
<p>At any time, there are either one or two tree structures held in shared
memory that new database clients will access (there may also be additional
tree structures being used by older clients - this API does not provide
information on them). One tree structure - the current tree - is used to
accumulate new data written to the database. The other tree structure -
the old tree - is a read-only tree holding older data and may be flushed 
to disk at any time.
<p>Assuming no error occurs, the location pointed to by the first of the two
(int *) arguments is set to the size of the old in-memory tree in KB.
The second is set to the size of the current, or live in-memory tree.
</dl><h2 id=opening>Opening and Closing Write Transactions<a id=lsm_begin></a><a id=lsm_commit></a><a id=lsm_rollback></a></h2>
<verbatim>int lsm_begin(lsm_db *pDb, int iLevel);
int lsm_commit(lsm_db *pDb, int iLevel);
int lsm_rollback(lsm_db *pDb, int iLevel);
</verbatim>
<p>These functions are used to open and close transactions and nested 
................................................................................
Delete a value from the database. No error is returned if the specified
key value does not exist in the database.
Delete all database entries with keys that are greater than (pKey1/nKey1) 
and smaller than (pKey2/nKey2). Note that keys (pKey1/nKey1) and
(pKey2/nKey2) themselves, if they exist in the database, are not deleted.
<p>Return LSM_OK if successful, or an LSM error code otherwise.
<h2 id=explicit>Explicit Database Work and Checkpointing<a id=lsm_work></a><a id=lsm_flush></a><a id=lsm_checkpoint></a></h2>
<verbatim>int lsm_work(lsm_db *pDb, int nMerge, int nKB, int *pnWrite);
int lsm_flush(lsm_db *pDb);
int lsm_checkpoint(lsm_db *pDb, int *pnKB);
</verbatim>
<p>This function is called by a thread to work on the database structure.
Attempt to checkpoint the current database snapshot. Return an LSM
error code if an error occurs or LSM_OK otherwise.
<p>If the current snapshot has already been checkpointed, calling this 
function is a no-op. In this case if pnKB is not NULL, *pnKB is
set to 0. Or, if the current snapshot is successfully checkpointed
by this function and pbKB is not NULL, *pnKB is set to the number
of bytes written to the database file since the previous checkpoint
(the same measure as returned by the LSM_INFO_CHECKPOINT_SIZE query).
<h2 id=opening>Opening and Closing Database Cursors<a id=lsm_csr_open></a><a id=lsm_csr_close></a></h2>
<verbatim>int lsm_csr_open(lsm_db *pDb, lsm_cursor **ppCsr);
int lsm_csr_close(lsm_cursor *pCsr);
</verbatim>
<p>Open and close a database cursor.

Changes to www/lsmusr.wiki.

370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
...
723
724
725
726
727
728
729
730

731
732
733
734
735
736
737
738
739
740
741
742










743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763

764
765
766
767
768
769







770
















771
772
773
774
775
776
777
....
1146
1147
1148
1149
1150
1151
1152
1153
1154
1155
1156
1157
1158
1159
1160
....
1164
1165
1166
1167
1168
1169
1170
1171
1172
1173
1174
1175
1176
1177
1178
the next entry. After lsm_csr_next() is called to advance past the final
entry in the database, the cursor is left pointing to no entry at all,
lsm_csr_valid() returns 0, and the loop is finished. API function 
<a href=lsmapi.wiki#lsm_csr_key>lsm_csr_key()</a> is used to retrieve the
key associated with each database entry visited.

<verbatim>
  for(rc = lsm_csr_first(csr); lsm_csr_valid(csr); rc = lsm_csr_next(csr)){
    const void *pKey; int nKey;
    const void *pVal; int nVal;

    rc = lsm_csr_key(csr, &pKey, &nKey);
    if( rc==LSM_OK ) rc = lsm_csr_value(csr, &pVal, &nVal);
    if( rc!=LSM_OK ) break;

    /* At this point pKey points to the current key (size nKey bytes) and
    ** pVal points to the corresponding value (size nVal bytes).  */
  }
</verbatim>

<p> The example code above could be modified to iterate backwards through
................................................................................
    int (*xBound)(void *pCtx, int nIn);
    int (*xCompress)(void *pCtx, void *pOut, int *pnOut, const void *pIn, int nIn);
    int (*xUncompress)(void *pCtx, void *pOut, int *pnOut, const void *pIn, int nIn);
    void (*xFree)(void *pCtx);
  };
</verbatim>

<p><i> Explain how the hooks work here (same as zipvfs) </i>


<p><i> Example code? Using zlib? Or something simple like an RLE
implementation?</i>

<p>The database file header of any LSM database contains a 32-bit unsigned
"compression id" field. If the database is not a compressed database, this
field is set to 1. Otherwise, it is set to an application supplied value
identifying the compression and/or encryption scheme in use. Application
compression scheme ids must be greater than or equal to 10000. Values smaller
than 10000 are reserved for internal use.

<p>The lsm_compression_id() API may be used to read the compression id from










a database connection. Because the compression id is stored in the database
header, it may be read before any required compression or encryption hooks
are configured.

<verbatim>
  #define LSM_COMPRESSION_EMPTY    0
  #define LSM_COMPRESSION_NONE     1
  int lsm_compression_id(lsm_db *db, u32 *piId);
</verbatim>

<p>When a database is opened for the first time, before it is first written,
the compression id field is set to LSM_COMPRESSION_EMPTY (0). The first time
a transaction is committed, the database compression id is set to a copy of 
the lsm_compress.iId field of the compression hooks for the database handle
committing the transaction, or to LSM_COMPRESSION_NONE (1) if no compression
hooks are configured.

<p>Once the compression id is set to something other than 
LSM_COMPRESSION_EMPTY, when a database handle opens a read or write 
transaction on the database, the compression id is compared against the 
lsm_compress.iId field of the configured compression hooks, or against LSM_COMPRESSION_NONE if no compression hooks are configured. If the compression id

does not match, then an LSM_MISMATCH error is returned and the operation 
fails (no transaction or database cursor is opened).

<p><i>Maybe there should be a way to register a mismatch-handler callback.
Otherwise, applications have to handle LSM_MISMATCH everywhere...
</i>

























<h1 id=performance_tuning>6. Performance Tuning</h1>

<p> This section describes the various measures that can be taken in order to
fine-tune LSM in order to improve performance in specific circumstances.
Sub-section 6.1 contains a high-level overview of the 
<a href=#overview_of_lsm_architecture>system architecture</a>
................................................................................
<p>The example code below might be executed in a background thread or process
in order to perform database work and checkpointing. In this case all other
clients should set the LSM_CONFIG_AUTOWORK parameter to zero.

<verbatim>
  int rc;
  lsm_db *db;
  int nCkpt = 4*1024*1024;

  /* Open a database connection to database "test.db". 
  **
  ** Configure the connection to automatically checkpoint the database after
  ** writing each 4MB of data to it (instead of the default 2MB). As well
  ** as to auto-work, the LSM_CONFIG_AUTOCHECKPOINT parameter applies to data
  ** written by explicit calls to lsm_work().
................................................................................
  lsm_open(db, "test.db");

  while( 1 ){
    int nWrite;

    /* Attempt up to 512KB of work. Set nWrite to the number of bytes
    ** actually written to disk.  */
    rc = lsm_work(db, 2, 512*1024, &nWrite);
    if( rc!=LSM_OK && rc!=LSM_BUSY ){
      /* Anything other than LSM_OK or LSM_BUSY is a problem. LSM_BUSY
      ** indicates that some other client has taken the WORKER lock. Any
      ** other error indicates something has gone quite wrong.  */
      lsm_close(db);
      return rc;
    }







|





|







 







|
>

|
|








|
>
>
>
>
>
>
>
>
>
>
|






<



|
|
|
|
|


|
|
|
>
|
|

<
|
|
>
>
>
>
>
>
>

>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>







 







|







 







|







370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
...
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760

761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777

778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
....
1179
1180
1181
1182
1183
1184
1185
1186
1187
1188
1189
1190
1191
1192
1193
....
1197
1198
1199
1200
1201
1202
1203
1204
1205
1206
1207
1208
1209
1210
1211
the next entry. After lsm_csr_next() is called to advance past the final
entry in the database, the cursor is left pointing to no entry at all,
lsm_csr_valid() returns 0, and the loop is finished. API function 
<a href=lsmapi.wiki#lsm_csr_key>lsm_csr_key()</a> is used to retrieve the
key associated with each database entry visited.

<verbatim>
  for(rc=lsm_csr_first(csr); rc==LSM_OK && lsm_csr_valid(csr); rc=lsm_csr_next(csr)){
    const void *pKey; int nKey;
    const void *pVal; int nVal;

    rc = lsm_csr_key(csr, &pKey, &nKey);
    if( rc==LSM_OK ) rc = lsm_csr_value(csr, &pVal, &nVal);
    if( rc==LSM_OK ) break;

    /* At this point pKey points to the current key (size nKey bytes) and
    ** pVal points to the corresponding value (size nVal bytes).  */
  }
</verbatim>

<p> The example code above could be modified to iterate backwards through
................................................................................
    int (*xBound)(void *pCtx, int nIn);
    int (*xCompress)(void *pCtx, void *pOut, int *pnOut, const void *pIn, int nIn);
    int (*xUncompress)(void *pCtx, void *pOut, int *pnOut, const void *pIn, int nIn);
    void (*xFree)(void *pCtx);
  };
</verbatim>

<p><span style=color:red> Explain how the hooks work here (same as zipvfs)
</span>

<p><span style=color:red> Example code? Using zlib? Or something simple like an
RLE implementation?</span>

<p>The database file header of any LSM database contains a 32-bit unsigned
"compression id" field. If the database is not a compressed database, this
field is set to 1. Otherwise, it is set to an application supplied value
identifying the compression and/or encryption scheme in use. Application
compression scheme ids must be greater than or equal to 10000. Values smaller
than 10000 are reserved for internal use.

<p>The lsm_info() API may be used to read the compression id from a database 
connection as follows: 

<verbatim>
  unsigned int iCompressionId;
  rc = lsm_info(db, LSM_INFO_COMPRESSION_ID, &iCompressionId);
  if( rc==LSM_OK ){
    /* Variable iCompressionId now contains the db compression id */
  }
</verbatim>

Because the compression id is stored in the database
header, it may be read before any required compression or encryption hooks
are configured.

<verbatim>
  #define LSM_COMPRESSION_EMPTY    0
  #define LSM_COMPRESSION_NONE     1

</verbatim>

<p>When a database is opened for the first time, before it is first written,
the compression id field is set to LSM_COMPRESSION_EMPTY (0). After data is
written into the database file, the database compression id is set to a copy 
of the lsm_compress.iId field of the compression hooks for the database handle
doing the writing, or to LSM_COMPRESSION_NONE (1) if no compression hooks 
are configured.

<p>Once the compression id is set to something other than 
LSM_COMPRESSION_EMPTY, when a database handle attempts to read or write the
database file, the compression id is compared against the lsm_compress.iId 
field of the configured compression hooks, or against LSM_COMPRESSION_NONE if
no compression hooks are configured. If the compression id does not match, then
an LSM_MISMATCH error is returned and the operation fails (no transaction or
database cursor is opened).


<p>It is also possible to register a compression factory callback with a 
database handle. If one is registered, the compression factory callback is
invoked instead of returning LSM_MISMATCH if the configured compression hooks
do not match the compression id of a database. If the callback registers
compatible compression hooks with the database handle (using the normal
lsm_config() interface), then the database read or write operation resumes
after it returns. Otherwise, if the compression factory callback does not
register new, compatible, compression hooks with the database handle,
LSM_MISMATCH is returned to the user.

<p>A compression factory callback is registered with a database handle
by calling lsm_config() with the second argument set to
LSM_CONFIG_SET_COMPRESSION_FACTORY, and the third argument set to point to
an instance of structure lsm_compress_factory. The lsm_config() copies the
contents of the structure - it does not retain a pointer to it.

<verbatim>
  typedef struct lsm_compress_factory lsm_compress_factory;
  struct lsm_compress_factory {
    void *pCtx;
    int (*xFactory)(void *pCtx, lsm_db *db, unsigned int iCompressionId);
    void (*xFree)(void *pCtx);
  };
</verbatim>

<p><span style=color:red> Explain how the xFactory hook works here. </span>

<h1 id=performance_tuning>6. Performance Tuning</h1>

<p> This section describes the various measures that can be taken in order to
fine-tune LSM in order to improve performance in specific circumstances.
Sub-section 6.1 contains a high-level overview of the 
<a href=#overview_of_lsm_architecture>system architecture</a>
................................................................................
<p>The example code below might be executed in a background thread or process
in order to perform database work and checkpointing. In this case all other
clients should set the LSM_CONFIG_AUTOWORK parameter to zero.

<verbatim>
  int rc;
  lsm_db *db;
  int nCkpt = 4*1024;             /* 4096KB == 4MB */

  /* Open a database connection to database "test.db". 
  **
  ** Configure the connection to automatically checkpoint the database after
  ** writing each 4MB of data to it (instead of the default 2MB). As well
  ** as to auto-work, the LSM_CONFIG_AUTOCHECKPOINT parameter applies to data
  ** written by explicit calls to lsm_work().
................................................................................
  lsm_open(db, "test.db");

  while( 1 ){
    int nWrite;

    /* Attempt up to 512KB of work. Set nWrite to the number of bytes
    ** actually written to disk.  */
    rc = lsm_work(db, 2, 512, &nWrite);
    if( rc!=LSM_OK && rc!=LSM_BUSY ){
      /* Anything other than LSM_OK or LSM_BUSY is a problem. LSM_BUSY
      ** indicates that some other client has taken the WORKER lock. Any
      ** other error indicates something has gone quite wrong.  */
      lsm_close(db);
      return rc;
    }