SQLite Forum

Timeline
Login

4 forum posts by user mmoore

2021-05-28
19:13 Reply: sqlite3session_changeset fails with SQLITE_NOMEM when sqlite3_session >= 1.0 GB (artifact: d5196bc5a8 user: mmoore)

Great, Thanks!

2021-05-24
19:32 Reply: sqlite3session_changeset fails with SQLITE_NOMEM when sqlite3_session >= 1.0 GB (artifact: 786fa39efe user: mmoore)

Wanted to follow up and see if anyone had any thoughts about this. To summarize: attempting to generate chagnesets >= 1.0 GB and < 2.0 GB fails with OOM. I believe this is the incorrect behavior caused by an issue with sessionBufferGrow. Patch included here: https://gist.github.com/R4N/fe4f0696bbb835cd5796e46f99328647

19:00 Edit: sqlite3session_changeset fails with SQLITE_NOMEM when sqlite3_session >= 1.0 GB (artifact: cc0e50b6fa user: mmoore)

When attempting to generate a changeset of a sqlite3_session attached to table(s) which have tracked >= 1.0 GB of data, sqlite3session_changeset fails with SQLITE_NOMEM.

Similarly when attempting to generate a changeset of a sqlite3_session from a sqlite3session_diff which is >= 1.0 GB of data, sqlite3session_changeset fails with SQLITE_NOMEM.

When looking at the documentation for the session extension: https://sqlite.org/sessionintro.html and sqlite3session_changeset: https://sqlite.org/session/sqlite3session_changeset.html I don't see any mention of this limit/restriction.

The culprit appears to be sessionBufferGrow(SessionBuffer, size_t, int):

Specifically this block of code:

    i64 nNew = p->nAlloc ? p->nAlloc : 128;
    do {
      nNew = nNew*2;
    }while( (size_t)(nNew-p->nBuf)<nByte );

If p->nAlloc >= 1073741696 which is then set to nNew it doubles once and will be >= 2147483392 which is over the max threshold in sqlite3Realloc (see below).

Eventually this nNew gets passed into sqlite3_realloc64(void *, sqlite3_uint64) which calls through the sqlite3Realloc(void *, u64) where we hit our limit at this block of code:

  if( nBytes>=0x7fffff00 ){
    /* The 0x7ffff00 limit term is explained in comments on sqlite3Malloc() */
    return 0;
  }

The comment in sqlite3Malloc() is as follows:

    /* A memory allocation of a number of bytes which is near the maximum
    ** signed integer value might cause an integer overflow inside of the
    ** xMalloc().  Hence we limit the maximum size to 0x7fffff00, giving
    ** 255 bytes of overhead.  SQLite itself will never use anything near
    ** this amount.  The only way to reach the limit is with sqlite3_malloc() */

My suggested patch to resolve this allows sqlite3session_changeset to successfully generate changesets from sessions tracking >= 1.0 GB and properly fails with SQLITE_NOMEM when attempting to generate changesets from sessions tracking >= 0x7FFFFEFE (2 bytes below the max threshold in sqlite3Malloc()):

static int sessionBufferGrow(SessionBuffer *p, size_t nByte, int *pRc){
  if( *pRc==SQLITE_OK && (size_t)(p->nAlloc-p->nBuf)<nByte ){
    u8 *aNew;
    i64 nNew = p->nAlloc ? p->nAlloc : 128;
    do {
      /* limit growth to 2 bytes below our max threshold as defined
       in sqlite3Malloc(), otherwise we would hit OOM condition when
       requesting to grow the buffer >= 1/2 max threshold */
      nNew = MIN(nNew*2, 0x7FFFFEFE);
      /* if we've hit 2 bytes below our max threshold and still haven't
       reached the requested size, break out with OOM */
      if ( nNew == 0x7FFFFEFE && (size_t)(nNew-p->nBuf)<nByte ){
        *pRc = SQLITE_NOMEM;
        break;
      }
    }while( (size_t)(nNew-p->nBuf)<nByte );

    aNew = (u8 *)sqlite3_realloc64(p->aBuf, nNew);
    if( 0==aNew ){
      *pRc = SQLITE_NOMEM;
    }else{
      p->aBuf = aNew;
      p->nAlloc = nNew;
    }
  }
  return (*pRc!=SQLITE_OK);
}

I've linked a reproduction case displaying the issue when attempting to generate a chanegset using sqlite3session_changeset from a session tracking tables and an additional example when the session was from a sqlite3session_diff: https://gist.github.com/R4N/f12407697e06bd22f9111cb9f4eb4a16

2021-05-14
21:34 Post: sqlite3session_changeset fails with SQLITE_NOMEM when sqlite3_session >= 1.0 GB (artifact: ed46fdad70 user: mmoore)

When attempting to generate a changeset of a sqlite3_session attached to table(s) which have tracked >= 1.0 GB of data, sqlite3session_changeset fails with SQLITE_NOMEM.

Similarly when attempting to generate a changeset of a sqlite3_session from a sqlite3session_diff which is >= 1.0 GB of data, sqlite3session_changeset fails with SQLITE_NOMEM.

When looking at the documentation for the session extension: https://sqlite.org/sessionintro.html and sqlite3session_changeset: https://sqlite.org/session/sqlite3session_changeset.html I don't see any mention of this limit/restriction.

The culprit appears to be sessionBufferGrow(SessionBuffer, size_t, int):

Specifically this block of code:

    i64 nNew = p->nAlloc ? p->nAlloc : 128;
    do {
      nNew = nNew*2;
    }while( (size_t)(nNew-p->nBuf)<nByte );

If p->nAlloc >= 1073741696 which is then set to nNew it doubles once and will be >= 2147483392 which is over the max threshold in sqlite3Realloc (see below).

Eventually this nNew gets passed into sqlite3_realloc64(void *, sqlite3_uint64) which calls through the sqlite3Realloc(void *, u64) where we hit our limit at this block of code:

  if( nBytes>=0x7fffff00 ){
    /* The 0x7ffff00 limit term is explained in comments on sqlite3Malloc() */
    return 0;
  }

The comment in sqlite3Malloc() is as follows:

    /* A memory allocation of a number of bytes which is near the maximum
    ** signed integer value might cause an integer overflow inside of the
    ** xMalloc().  Hence we limit the maximum size to 0x7fffff00, giving
    ** 255 bytes of overhead.  SQLite itself will never use anything near
    ** this amount.  The only way to reach the limit is with sqlite3_malloc() */

My suggested patch to resolve this allows sqlite3session_changeset to successfully generate changesets from sessions tracking >= 1.0 GB and properly fails with SQLITE_NOMEM when attempting to generate changesets from sessions tracking >= 0x7FFFFEFE (2 bytes below the max threshold in sqlite3Malloc()):

static int sessionBufferGrow(SessionBuffer *p, size_t nByte, int *pRc){
  if( *pRc==SQLITE_OK ){
    if( (size_t)(p->nAlloc-p->nBuf)<nByte ){
      u8 *aNew;
      i64 nNew = p->nAlloc ? p->nAlloc : 128;
      do {
        /* limit growth to 2 bytes below our max threshold as defined
         in sqlite3Malloc(), otherwise we would hit OOM condition when
         requesting to grow the buffer >= 1/2 max threshold */
        nNew = MIN(nNew*2, 0x7FFFFEFE);
        /* if we've hit 2 bytes below our max threshold and still haven't
         reached the requested size, break out with OOM */
        if ( nNew == 0x7FFFFEFE && (size_t)(nNew-p->nBuf)<nByte ){
          *pRc = SQLITE_NOMEM;
          break;
        }
      }while( (size_t)(nNew-p->nBuf)<nByte );
      aNew = (u8 *)sqlite3_realloc64(p->aBuf, nNew);
      if( 0==aNew ){
        *pRc = SQLITE_NOMEM;
      }else{
        p->aBuf = aNew;
        p->nAlloc = nNew;
      }
    }
  }
  return (*pRc!=SQLITE_OK);
}

I've linked a reproduction case displaying the issue when attempting to generate a chanegset using sqlite3session_changeset from a session tracking tables and an additional example when the session was from a sqlite3session_diff: https://gist.github.com/R4N/f12407697e06bd22f9111cb9f4eb4a16