SQLite Forum

sqlite3session_changeset fails with SQLITE_NOMEM when sqlite3_session >= 1.0 GB
Login

sqlite3session_changeset fails with SQLITE_NOMEM when sqlite3_session >= 1.0 GB

(1.1) By R4N (mmoore) on 2021-05-24 19:00:05 edited from 1.0 [link] [source]

When attempting to generate a changeset of a sqlite3_session attached to table(s) which have tracked >= 1.0 GB of data, sqlite3session_changeset fails with SQLITE_NOMEM.

Similarly when attempting to generate a changeset of a sqlite3_session from a sqlite3session_diff which is >= 1.0 GB of data, sqlite3session_changeset fails with SQLITE_NOMEM.

When looking at the documentation for the session extension: https://sqlite.org/sessionintro.html and sqlite3session_changeset: https://sqlite.org/session/sqlite3session_changeset.html I don't see any mention of this limit/restriction.

The culprit appears to be sessionBufferGrow(SessionBuffer, size_t, int):

Specifically this block of code:

    i64 nNew = p->nAlloc ? p->nAlloc : 128;
    do {
      nNew = nNew*2;
    }while( (size_t)(nNew-p->nBuf)<nByte );

If p->nAlloc >= 1073741696 which is then set to nNew it doubles once and will be >= 2147483392 which is over the max threshold in sqlite3Realloc (see below).

Eventually this nNew gets passed into sqlite3_realloc64(void *, sqlite3_uint64) which calls through the sqlite3Realloc(void *, u64) where we hit our limit at this block of code:

  if( nBytes>=0x7fffff00 ){
    /* The 0x7ffff00 limit term is explained in comments on sqlite3Malloc() */
    return 0;
  }

The comment in sqlite3Malloc() is as follows:

    /* A memory allocation of a number of bytes which is near the maximum
    ** signed integer value might cause an integer overflow inside of the
    ** xMalloc().  Hence we limit the maximum size to 0x7fffff00, giving
    ** 255 bytes of overhead.  SQLite itself will never use anything near
    ** this amount.  The only way to reach the limit is with sqlite3_malloc() */

My suggested patch to resolve this allows sqlite3session_changeset to successfully generate changesets from sessions tracking >= 1.0 GB and properly fails with SQLITE_NOMEM when attempting to generate changesets from sessions tracking >= 0x7FFFFEFE (2 bytes below the max threshold in sqlite3Malloc()):

static int sessionBufferGrow(SessionBuffer *p, size_t nByte, int *pRc){
  if( *pRc==SQLITE_OK && (size_t)(p->nAlloc-p->nBuf)<nByte ){
    u8 *aNew;
    i64 nNew = p->nAlloc ? p->nAlloc : 128;
    do {
      /* limit growth to 2 bytes below our max threshold as defined
       in sqlite3Malloc(), otherwise we would hit OOM condition when
       requesting to grow the buffer >= 1/2 max threshold */
      nNew = MIN(nNew*2, 0x7FFFFEFE);
      /* if we've hit 2 bytes below our max threshold and still haven't
       reached the requested size, break out with OOM */
      if ( nNew == 0x7FFFFEFE && (size_t)(nNew-p->nBuf)<nByte ){
        *pRc = SQLITE_NOMEM;
        break;
      }
    }while( (size_t)(nNew-p->nBuf)<nByte );

    aNew = (u8 *)sqlite3_realloc64(p->aBuf, nNew);
    if( 0==aNew ){
      *pRc = SQLITE_NOMEM;
    }else{
      p->aBuf = aNew;
      p->nAlloc = nNew;
    }
  }
  return (*pRc!=SQLITE_OK);
}

I've linked a reproduction case displaying the issue when attempting to generate a chanegset using sqlite3session_changeset from a session tracking tables and an additional example when the session was from a sqlite3session_diff: https://gist.github.com/R4N/f12407697e06bd22f9111cb9f4eb4a16

(2) By R4N (mmoore) on 2021-05-24 19:32:37 in reply to 1.1 [link] [source]

Wanted to follow up and see if anyone had any thoughts about this. To summarize: attempting to generate chagnesets >= 1.0 GB and < 2.0 GB fails with OOM. I believe this is the incorrect behavior caused by an issue with sessionBufferGrow. Patch included here: https://gist.github.com/R4N/fe4f0696bbb835cd5796e46f99328647

(3) By Dan Kennedy (dan) on 2021-05-25 16:15:40 in reply to 2 [source]

(4) By R4N (mmoore) on 2021-05-28 19:13:12 in reply to 3 [link] [source]

Great, Thanks!