Abstract

​ I discovered an integer overflow bug in the sqlite3.c file at line 182342 using code auditing techniques. This line is in the setupLookaside function, with the following code: nSm = (szAlloc - sz*nBig)/LOOKASIDE_SMALL;. The result of sz*nBig can potentially cause an integer overflow.

Function Description

​ The function's purpose is to divide the incoming lookaside buffer block into large and small memory chunks, with specific memory size allocation based on the provided sz and cnt parameters. Here, sz is the size of the lookaside buffer block defined by the user, and cnt is the number of such blocks. The user defines sz and cnt when calling the SQL function sqlite3_db_config(db, SQLITE_DBCONFIG_LOOKASIDE, NULL , sz , cnt, NULL);. After sz and cnt are passed to the sqlite3 program, the program proceeds to execute the code near line 182342:

#define LOOKASIDE_SMALL 128

...
sqlite3_int64 szAlloc = sz*(sqlite3_int64)cnt;
...
if( sz>=LOOKASIDE_SMALL*3 ){
    nBig = szAlloc/(3*LOOKASIDE_SMALL+sz);
    nSm = (szAlloc - sz*nBig)/LOOKASIDE_SMALL;
}
...

​ This portion of the code calculates the number of large memory blocks nBig first using szAlloc/(3*LOOKASIDE_SMALL+sz), where szAlloc is the total memory to be allocated, i.e., sz*cnt, LOOKASIDE_SMALL is a defined constant of 128, and sz is the passed value. After calculating the large memory blocks, the remaining memory is divided into small chunks of size LOOKASIDE_SMALL, determining the number of small blocks. The calculation is done by subtracting the allocated large memory size (sz * nBig) from the total memory (szAlloc) and then dividing by LOOKASIDE_SMALL: (szAlloc - sz*nBig)/LOOKASIDE_SMALL. This gives the number of small blocks to allocate.

Integer Overflow Bug

​ When calculating sz*nBig, the result was not cast to sqlite3_int64, leading to an integer overflow vulnerability. In normal cases, with sz = 1200 and cnt = 100, the value of szAlloc would be sz*cnt = 1200*100 = 120,000. Since sz = 1200 is greater than LOOKASIDE_SMALL*3 = 128*3 = 384, the program calculates nBig as 120000/(3*128 + 1200) = 75.75, which is rounded down to 75. Next, nSm is calculated as (szAlloc - sz*nBig)/LOOKASIDE_SMALL = (120000 - 1200*75)/128 = 234.

​ However, when sz = 1200 and cnt = 2500000, an integer overflow occurs. First, calculating szAlloc results in sz*cnt = 1200*2500000 = 3000000000. Similarly, since sz = 1200 is still greater than LOOKASIDE_SMALL*3, nBig is computed as 3000000000/(3*128 + 1200), which gives nBig = 1893939. Next, the calculation for nSm theoretically should be (3000000000 - 1200*1893939)/128 = 5681821. However, due to the overflow when calculating 1200*1893939, the actual result in gdb debugging is 39236253. Here is the gdb output:

...
(gdb) next
182340    if( sz>=LOOKASIDE_SMALL*3 ){
(gdb) step
182341      nBig = szAlloc/(3*LOOKASIDE_SMALL+sz);
(gdb) step
182345      nSm = (szAlloc - sz*nBig)/LOOKASIDE_SMALL; //LOOKASIDE_SMALL is 128
(gdb) step
182354    db->lookaside.pStart = pStart;
(gdb) print nBig
$1 = 1893939
(gdb) print nSm
$2 = 39236253
...

Potential Impact and Suggested Fix

​ Your code accounted for the possibility that the total allocated space might exceed the size of an int, using (sqlite3_int64) to cast the result of sz*cnt to sqlite3_int64: sqlite3_int64 szAlloc = sz*(sqlite3_int64)cnt;. However, when allocating the memory blocks, this casting was not applied, leading to potential issues where the lookaside buffer might fail to divide correctly, and the allocated memory may exceed the intended size. This could cause crashes or even allow attackers to exploit the bug for arbitrary memory writes, potentially leading to remote code execution vulnerabilities.

​ I recommend adding the same type casting to sz*nBig during the calculation of nSm, as follows:

nSm = (szAlloc - sz*(sqlite3_int64)nBig)/LOOKASIDE_SMALL; 

​ This will resolve the overflow bug.