Low-Level Memory Allocation
Code to implement the low-level memory allocation is in one of the following files:
- mem1.c
- mem2.c
- mem3.c
New memN.c files may added in the future. All of these files have appropriate #ifdef...#endif constructs so that only the code from one of the files is used for an given build. Mem1.c gets memory from the system malloc() and is intended as the default implementation. Mem2.c gets memory from system malloc() but also does lots of debugging and memory-leak detection. Mem2.c is used when -DSQLITE_MEMDEBUG is defined. Mem3.c ues a static array of size SQLITE_MEMORY_SIZE to satisfy all memory allocation requests.
In version 3.5.0 and 3.5.1, one could compile with -DSQLITE_OMIT_MEMORY_ALLOCATION and none of mem[123].c would be used. The application program was expected to supply its own memory allocator. This capability is no longer supported.
All of the mem*.c modules have the same interface:
void *sqlite3_malloc(int); void *sqlite3_realloc(void*, int); void sqlite3_free(void*); sqlite3_int64 sqlite3_memory_used(void); sqlite3_int64 sqlite3_memory_highwater(int); int sqlite3_memory_alarm(...) int sqlite3MemsysSize(void*); int sqlite3MemsysFree(void*); int sqlite3MemsysIsValid(void*);
The first six routines have been seen before and are well-documented. The last two are new.
An allocation is considered "valid" if the void* pointer that immediately preceded the allocation is not a valid pointer. A valid pointer always has zero lowest three bits. The allocations returned sqlite3_malloc() and sqlite3_realloc() are guaranteed to be valid. In particular, we guarantee that for any pointer pMem returned by the low-level allocators, the following is true:
assert( ((long long int)(((void**)pMem)[-1])&1)==1 );
The sqlite3MemsysIsValid() interface tests the condition above. It returns true if the pointer might be a valid pointer returned from sqlite3_malloc() or sqlite3_realloc() and it returns false if its input is clearly not a pointer from either of these interfaces. sqlite3MemsysFree() is a wrapper around sqlite3_free() as follows:
int sqlite3MemsysFree(void *pOld){
if( pOld==0 ){
return 1;
}else if( !sqlite3MemsysIsValid(pOld) ){
return 0;
}else{
sqlite3_free(pOld);
return 1;
}
}
The sqlite3MemsysFree() interface is equivalent to the above but might be implemented more efficiently since it is on the critical path.
The sqlite3MemsysSize() interface returns the size in bytes of an outstanding allocation. The size includes an extra bytes resulting from the allocation being rounded up to the next allocatable size, but omits memory associated with the header or trailer on the allocation.
Memory Testing Subsystem
A new source file named memtest.c contains code that implements a memory allocation failure test harness. The interface to this subsystem is as follows:
void sqlite3MemsysSetTitle(const char*); void sqlite3MemsysDump(const char*); int sqlite3MemsysSetPending(int,int); int sqlite3MemsysFailureCount(int); int sqlite3MemsysPending(int); void sqlite3MemsysBenignOnce(int); void sqlite3MemsysBenignBegin(int); void sqlite3MemsysBenignEnd(void); void sqlite3MemsysDisable(int);
This interface is is only activated if debugging is enabled. For a non-debugging build, these routines are macros that evaluate to no-ops.
The sqlite3MemsysSetTitle() interface records the name of the current test. The test name is used by mem2.c to record the test in which an allocation occurred. This is useful in tracking down memory leaks. The sqlite3MemsysSetTitle() is currently a no-op for mem1.c and mem3.c.
The sqlite3MemsysDump() command writes status information about the allocator to the file named in its argument. If the argument is NULL then the information is written to standard output.
The sqlite3MemsetSetTitle() and sqlite3MemsetDump() interfaces are implemented by mem*.c, not memtest.c, since they both require access to internal data structures within mem*.c. Their interface definition is included with memtest.c because they logically belong to that module.
The sqlite3MemsysSetPending() interface sets up a simulated memory allocation failure. The first argument is the number of successful memory allocations that will occur before the failure. The second argument is one if the failure is permanent and 0 if the failure is a one-time occurrance. Each call to sqlite3MemsysSetPending() overwrites previous settings.
The sqlite3MemsysPending() interface returns the number of successful memory allocations that will occur before the next error. If a non-repeating error has occurred or if no simulated errors are pending then -1 is returned. If the next memory allocation will fail, either as the first error or as a repeat, then 0 is returned. If the argument is zero, then the counter is unchanged by this call. But if the argument is true, the counter is decremented. Routines that actually do memory allocation, such as sqlite3_malloc(), can invoke sqlite3MemsysPending() with a non-zero argument in order to see if they should simulate a failure and to notify the failure simulator that another allocation has occurred. Note that calls to sqlite3MemsysPending() with a true argument are used by both the low-level memory allocators and also the high-level linear and stack memory pools in cases where the memory pool is merely distributing part of a prior allocation.
The sqlite3MemsysFailureCount() interface returns the number of failures that have occurred since the last sqlite3MemsysSetPending(). In other words, it returns the number of times that sqlite3MemsysPending() has returned zero when it was given a non-zero argument. If the argument to sqlite3MemsysFailureCount() then a count of all failures is returned. If the argument is non-zero then a count of "benign" failures is returned. A benign failure is a memory allocation that occurs in a context where SQLite is able to continue operating correctly without the requested memory.
The sqlite3MemsysBenignOnce(), sqlite3MemsysBenignBegin(), and sqlite3MemsysBenignEnd() interfaces are used to specify which memory allocations are benign. The sqlite3MemsysBenignOnce() routine means that the next memory allocation is benign. All memory allocations that occur in between sqlite3MemsysBenignBegin() and sqlite3MemsysBenignEnd() are considered benign.
The sqlite3MemsysDisable() interface is used to disable or enable memory allocation. If the argument is 1 the memory allocation subsystem is disabled and if 0 it is reenabled. The subsystem is enabled by default. Any call to sqlite3MemsysPending() with a nonzero argument while the memory subsystem is disabled results in an assertion fault.
Memory Pools
As seen in the next section, most of the interfaces to the memory allocator require a pointer to an MPool object as a parameter. The MPool object is a "memory pool". Memory pools are created as follows:
MPool *sqlite3MPoolDb(MPool*, sqlite3*); MPool *sqlite3MPoolLinear(MPool*); MPool *sqlite3MPoolStack(MPool*); void sqlite3MPoolDestroy(MPool*);
A memory pool is an area from which memory is allocated. The MPool* argument can always be NULL in which case memory is taken directly from the low-level routines. So, for example, the following two statements have exactly the same effect:
p = sqlite3MallocRaw(0, 1234); p = sqlite3_malloc(1234);
Even though the result is the same, all core SQLite code should use the first form in preference to the second.
Memory allocators are created using the constructors defined above. The sqlite3MPoolDB() constructor builds a memory pool that works just like the NULL memory pool in that it goes directly to the low-level interfaces to acquire its memory. The difference is that when a memory allocation fails, the sqlite3.mallocFailed flag on the specified database connection is set automatically. The first argument to sqlite3MPoolDb() is the memory pool from which the new MPool object itself is allocated. This is often NULL.
The sqlite3MPoolLinear() constructor creates a memory pool that allocates by advancing a pointer through a large buffer. The sqlite3Free() operation is a no-op for such allocations. The memory is not released until the memory pool is destroyed by calling sqlite3MPoolDestroy(). The large buffer is obtained from a separate "parent" memory allocator which might be the NULL allocator or (more commonly) an sqlite3MPoolDb() allocator. A linear memory pool is intended for such tasks as constructing a new sqlite3_stmt - tasks for which there is a long sequence of small allocations that are deallocated all at once when the object is destroyed.
The sqlite3MPoolStack() allocator is intended to be a last-in first-out (LIFO) allocator. This is currently just place-holder. We do not yet know exactly how the stack pool will be useful.
The sqlite3MPoolDestroy() interface is the destructor for memory pools. For a linear or stack memory pool, the destructor causes all memory allocated under that pool to be deallocated. For the Db pool, the destructor causes the underlying MPoolDb object to be destroyed but does not deallocate any memory. For the NULL pull, the destructor is a no-op.
The Main Memory Allocation Interface
All subsystems within the SQLite core allocate memory using the following interface:
void *sqlite3MallocRaw(MPool*, int); void *sqlite3MallocZero(MPool*, int); void sqlite3Free(void*); void *sqlite3Realloc(MPool*, void*, int); void *sqlite3ReallocOrFree(MPool*, void*, int); char *sqlite3StrDup(Mpool*, const char*); char *sqlite3StrNDup(Mpool*, const char*, int); char *sqlite3MPrintf(Mpool*, const char*, ...); char *sqlite3VMPrintf(Mpool*, const char*, va_list);
The SQLite core never calls the low-level allocators directly. All memory allocation is handled through these higher-level interface. (But extension such as virtual tables, application-defined functions and collating sequences do not have access to these internal interfaces and must use the low-level allocators.)
The sqlite3MallocRaw() and sqlite3MallocZero() interfaces allocate new memory. The difference is that sqlite3MallocZero() takes the extra step of zeroing the newly allocated memory before returning it. The content of the memory returned by sqlite3MallocRaw() is undefined.
The sqlite3Free() interface releases an allocation. Memory obtained from low-level routines sqlite3_malloc() and sqlite3_realloc() can be released by calling sqlite3Free(). However, memory obtained from linear and stack memory pools cannot be released by calling sqlite3_free(). If memory from a linear or stack memory pool is passed to sqlite3_free() the behavior is undefined and probably undesirable. In other words, sqlite3Free() is a universal deallocator whereas sqlite3_free() only works for allocations from NULL or Db memory pools.
The sqlite3Realloc() and sqlite3ReallocOrFree() interface are used to change the size or the memory pool of an existing allocation. The allocation might or might not be copied to a new location. The return pointer always points to the new allocation regardless of whether or not it has been copied. The return pointer is NULL if the reallocation failed. The sqlite3Realloc() does not free the old allocation if the reallocation fails, but sqlite3ReallocOrFree() does. The memory pool need not be the same memory pool that was used for the original allocation. If a different memory pool is specified, then the allocation is transferred to the new memory pool.
The sqlite3StrDup() and sqlite3StrNDup() interfaces are string duplicators. The sqlite3MPrintf() and sqlite3VMPrintf() are the string formatter interfaces.