SQLite Forum

sqlite3_serialize / deserialize 2GB limit on 64 bit
Login

sqlite3_serialize / deserialize 2GB limit on 64 bit

(1) By Roger Binns (rogerbinns) on 2021-10-08 21:34:10 [source]

The underlying code calls sqlite3_malloc64 which calls sqlite3Malloc. That method fails with allocations above 2GB even on 64 bit platforms. Is the intention to have this undocumented limit to serialize/deserialize?

The code for sqlite3Malloc has this comment:

if( n==0 || n>=0x7fffff00 ){
    /* A memory allocation of a number of bytes which is near the maximum
    ** signed integer value might cause an integer overflow inside of the
    ** xMalloc().  Hence we limit the maximum size to 0x7fffff00, giving
    ** 255 bytes of overhead.  SQLite itself will never use anything near
   ** this amount.  The only way to reach the limit is with sqlite3_malloc() */

SQLite needs this amount and more for serialization and for me to call deserialize and have SQLite free the memory.

Roger

(2) By VĂ­ctor (wzrlpy) on 2021-10-09 08:38:30 in reply to 1 [link] [source]

That 2GB limit is documented as "SQLite will refuse to allocate more than about 2GB of memory at one go" at https://www.sqlite.org/malloc.html

(3) By Roger Binns (rogerbinns) on 2021-10-12 13:49:38 in reply to 2 [link] [source]

That doesn't address the issue. The documentation for sqlite3_serialize, sqlite3_deserialize, and sqlite3_malloc64 do not mention any 2GB limit. In this specific case SQLite is doing a bulk export or import of the database so even the comment in the page you linked isn't obviously pertinent.

In any event can the SQLite team please either document an intended limit of 2GB for serialization, or make it work (eq use a new sqlite3_bulk_alloc with no 2GB limit).

Roger

(4) By Richard Hipp (drh) on 2021-10-12 14:03:26 in reply to 3 [link] [source]

We have discussed making it work during the next release cycle. No promises, though.