sqlite3session_changeset fails with SQLITE_NOMEM when sqlite3_session >= 1.0 GB
(1.1) By R4N (mmoore) on 2021-05-24 19:00:05 edited from 1.0 [source]
When attempting to generate a changeset of a sqlite3_session attached to table(s) which have tracked >= 1.0 GB of data, sqlite3session_changeset fails with SQLITE_NOMEM.
Similarly when attempting to generate a changeset of a sqlite3_session from a sqlite3session_diff which is >= 1.0 GB of data, sqlite3session_changeset fails with SQLITE_NOMEM.
When looking at the documentation for the session extension: https://sqlite.org/sessionintro.html and sqlite3session_changeset: https://sqlite.org/session/sqlite3session_changeset.html I don't see any mention of this limit/restriction.
The culprit appears to be sessionBufferGrow(SessionBuffer, size_t, int):
Specifically this block of code:
i64 nNew = p->nAlloc ? p->nAlloc : 128;
do {
nNew = nNew*2;
}while( (size_t)(nNew-p->nBuf)<nByte );
If p->nAlloc >= 1073741696 which is then set to nNew it doubles once and will be >= 2147483392 which is over the max threshold in sqlite3Realloc (see below).
Eventually this nNew gets passed into sqlite3_realloc64(void *, sqlite3_uint64) which calls through the sqlite3Realloc(void *, u64) where we hit our limit at this block of code:
if( nBytes>=0x7fffff00 ){
/* The 0x7ffff00 limit term is explained in comments on sqlite3Malloc() */
return 0;
}
The comment in sqlite3Malloc() is as follows:
/* A memory allocation of a number of bytes which is near the maximum
** signed integer value might cause an integer overflow inside of the
** xMalloc(). Hence we limit the maximum size to 0x7fffff00, giving
** 255 bytes of overhead. SQLite itself will never use anything near
** this amount. The only way to reach the limit is with sqlite3_malloc() */
My suggested patch to resolve this allows sqlite3session_changeset to successfully generate changesets from sessions tracking >= 1.0 GB and properly fails with SQLITE_NOMEM when attempting to generate changesets from sessions tracking >= 0x7FFFFEFE (2 bytes below the max threshold in sqlite3Malloc()):
static int sessionBufferGrow(SessionBuffer *p, size_t nByte, int *pRc){
if( *pRc==SQLITE_OK && (size_t)(p->nAlloc-p->nBuf)<nByte ){
u8 *aNew;
i64 nNew = p->nAlloc ? p->nAlloc : 128;
do {
/* limit growth to 2 bytes below our max threshold as defined
in sqlite3Malloc(), otherwise we would hit OOM condition when
requesting to grow the buffer >= 1/2 max threshold */
nNew = MIN(nNew*2, 0x7FFFFEFE);
/* if we've hit 2 bytes below our max threshold and still haven't
reached the requested size, break out with OOM */
if ( nNew == 0x7FFFFEFE && (size_t)(nNew-p->nBuf)<nByte ){
*pRc = SQLITE_NOMEM;
break;
}
}while( (size_t)(nNew-p->nBuf)<nByte );
aNew = (u8 *)sqlite3_realloc64(p->aBuf, nNew);
if( 0==aNew ){
*pRc = SQLITE_NOMEM;
}else{
p->aBuf = aNew;
p->nAlloc = nNew;
}
}
return (*pRc!=SQLITE_OK);
}
I've linked a reproduction case displaying the issue when attempting to generate a chanegset using sqlite3session_changeset from a session tracking tables and an additional example when the session was from a sqlite3session_diff: https://gist.github.com/R4N/f12407697e06bd22f9111cb9f4eb4a16
(2) By R4N (mmoore) on 2021-05-24 19:32:37 in reply to 1.1 [link] [source]
Wanted to follow up and see if anyone had any thoughts about this. To summarize: attempting to generate chagnesets >= 1.0 GB and < 2.0 GB fails with OOM. I believe this is the incorrect behavior caused by an issue with sessionBufferGrow. Patch included here: https://gist.github.com/R4N/fe4f0696bbb835cd5796e46f99328647
(3) By Dan Kennedy (dan) on 2021-05-25 16:15:40 in reply to 2 [link] [source]
(4) By R4N (mmoore) on 2021-05-28 19:13:12 in reply to 3 [link] [source]
Great, Thanks!