SQLite Forum

Feature request: Bring your own buffer to sqlite3_serialize
Login
Currently, there is no way to pass an existing buffer to sqlite3_serialize.
I want such a feature to implement a wrapper around the serialize/deserialize interface in rusqlite that works with Rust vectors. Performance wise, it is not great to have to copy the in-memory database twice to get a `Vec<u8>` in Rust. One workaround is to use the backup API and a temporary connection, but this adds much overhead (see <https://github.com/rusqlite/rusqlite/pull/786>).

My idea to implement this is to add a new flag for `sqlite3_serialize(D,S,P,F)` to bring your own buffer: `#define SQLITE_SERIALIZE_BYOB 2`.

If the F argument contains the SQLITE_SERIALIZE_BYOB bit, P must point
to an int64 followed by a pointer, representing the buffer size and start
address. The sqlite3_serialize() function copies the serialization of the
database into that buffer if it fits and returns a pointer to it,
otherwise it will just allocate like normal.

This is a diff for `src/memfile.c` (manifest uuid fca8dc8b578f215a969cd899336378966156154710873e68b3d9ac5881b0ff3f).

```
***************
*** 474,475 ****
--- 474,476 ----
    int iDb;
+   sqlite3_int64 szByob = -1;
    Btree *pBt;
***************
*** 492,493 ****
--- 493,499 ----
    iDb = sqlite3FindDbName(db, zSchema);
+   if( mFlags & SQLITE_SERIALIZE_BYOB ){
+     if( !piSize ) return 0;
+     szByob = *piSize;
+     pOut = *(unsigned char **)(piSize + 1);
+   }
    if( piSize ) *piSize = -1;
***************
*** 499,501 ****
      }else{
!       pOut = sqlite3_malloc64( p->sz );
        if( pOut ) memcpy(pOut, p->aData, p->sz);
--- 505,507 ----
      }else{
!       if( szByob < p->sz) pOut = sqlite3_malloc64( p->sz );
        if( pOut ) memcpy(pOut, p->aData, p->sz);
***************
*** 520,522 ****
      }else{
!       pOut = sqlite3_malloc64( sz );
        if( pOut ){
--- 526,528 ----
      }else{
!       if( szByob < sz ) pOut = sqlite3_malloc64( sz );
        if( pOut ){
```

This was tested by a quick patch in the amalgamation as seen here: <https://github.com/TjeuKayim/rusqlite/compare/8ee55b3e5a4c098ef772c8e91ef43a75959fd7f7..serialize-byob>.

What do you think of this feature and the suggested API?