Currently, there is no way to pass an existing buffer to sqlite3_serialize. I want such a feature to implement a wrapper around the serialize/deserialize interface in rusqlite that works with Rust vectors. Performance wise, it is not great to have to copy the in-memory database twice to get a Vec<u8> in Rust. One workaround is to use the backup API and a temporary connection, but this adds much overhead (see https://github.com/rusqlite/rusqlite/pull/786).

My idea to implement this is to add a new flag for sqlite3_serialize(D,S,P,F) to bring your own buffer: #define SQLITE_SERIALIZE_BYOB 2.

If the F argument contains the SQLITE_SERIALIZE_BYOB bit, P must point to an int64 followed by a pointer, representing the buffer size and start address. The sqlite3_serialize() function copies the serialization of the database into that buffer if it fits and returns a pointer to it, otherwise it will just allocate like normal.

This is a diff for src/memfile.c (manifest uuid fca8dc8b578f215a969cd899336378966156154710873e68b3d9ac5881b0ff3f).

***************
*** 474,475 ****
--- 474,476 ----
    int iDb;
+   sqlite3_int64 szByob = -1;
    Btree *pBt;
***************
*** 492,493 ****
--- 493,499 ----
    iDb = sqlite3FindDbName(db, zSchema);
+   if( mFlags & SQLITE_SERIALIZE_BYOB ){
+     if( !piSize ) return 0;
+     szByob = *piSize;
+     pOut = *(unsigned char **)(piSize + 1);
+   }
    if( piSize ) *piSize = -1;
***************
*** 499,501 ****
      }else{
!       pOut = sqlite3_malloc64( p->sz );
        if( pOut ) memcpy(pOut, p->aData, p->sz);
--- 505,507 ----
      }else{
!       if( szByob < p->sz) pOut = sqlite3_malloc64( p->sz );
        if( pOut ) memcpy(pOut, p->aData, p->sz);
***************
*** 520,522 ****
      }else{
!       pOut = sqlite3_malloc64( sz );
        if( pOut ){
--- 526,528 ----
      }else{
!       if( szByob < sz ) pOut = sqlite3_malloc64( sz );
        if( pOut ){

This was tested by a quick patch in the amalgamation as seen here: https://github.com/TjeuKayim/rusqlite/compare/8ee55b3e5a4c098ef772c8e91ef43a75959fd7f7..serialize-byob.

What do you think of this feature and the suggested API?