Feature request: Bring your own buffer to sqlite3_serialize
(1) By Tjeu (15987676) on 2020-09-06 12:16:06 updated by 1.1 [source]
Currently, there is no way to pass an existing buffer to sqlite3_serialize. I want such a feature to implement a wrapper around the serialize/deserialize interface in rusqlite that works with Rust vectors. Performance wise, it is not great to have to copy the in-memory database twice to get a `Vec<u8>` in Rust. One workaround is to use the backup API and a temporary connection, but this adds much overhead (see <https://github.com/rusqlite/rusqlite/pull/786>). My idea to implement this is to add a new flag for `sqlite3_serialize(D,S,P,F)` to bring your own buffer: `#define SQLITE_SERIALIZE_BYOB 0x010`. If the F argument contains the `SQLITE_SERIALIZE_BYOB` bit, the `sqlite3_serialize()` function copies the serialization of the database into a buffer provided by the caller. P points to the size of this buffer as `sqlite3_int64`, followed by a pointer to the start of the allocation. This is a diff for `src/memfile.c` (manifest uuid fca8dc8b578f215a969cd899336378966156154710873e68b3d9ac5881b0ff3f). ``` *************** *** 474,475 **** --- 474,476 ---- int iDb; + sqlite3_int64 szByob = -1; Btree *pBt; *************** *** 492,493 **** --- 493,499 ---- iDb = sqlite3FindDbName(db, zSchema); + if( mFlags & SQLITE_SERIALIZE_BYOB ){ + if( !piSize ) return 0; + szByob = *piSize; + pOut = *(unsigned char **)(piSize + 1); + } if( piSize ) *piSize = -1; *************** *** 499,501 **** }else{ ! pOut = sqlite3_malloc64( p->sz ); if( pOut ) memcpy(pOut, p->aData, p->sz); --- 505,507 ---- }else{ ! if( szByob != p->sz) pOut = sqlite3_malloc64( p->sz ); if( pOut ) memcpy(pOut, p->aData, p->sz); *************** *** 520,522 **** }else{ ! pOut = sqlite3_malloc64( sz ); if( pOut ){ --- 526,528 ---- }else{ ! if( szByob != sz ) pOut = sqlite3_malloc64( sz ); if( pOut ){ ``` This was tested by a quick patch in the amalgamation as seen here: <https://github.com/rusqlite/rusqlite/commit/73f5db6c0c78069b352c14ffd11b3d2fd4a46c72>. What do you think of this feature and the suggested API?
Feature request: Bring your own buffer to sqlite3_serialize
(1.1) By Tjeu (15987676) on 2020-09-09 18:48:17 edited from 1.0 [link] [source]
Currently, there is no way to pass an existing buffer to sqlite3_serialize.
I want such a feature to implement a wrapper around the serialize/deserialize interface in rusqlite that works with Rust vectors. Performance wise, it is not great to have to copy the in-memory database twice to get a Vec<u8>
in Rust. One workaround is to use the backup API and a temporary connection, but this adds much overhead (see https://github.com/rusqlite/rusqlite/pull/786).
My idea to implement this is to add a new flag for sqlite3_serialize(D,S,P,F)
to bring your own buffer: #define SQLITE_SERIALIZE_BYOB 2
.
If the F argument contains the SQLITE_SERIALIZE_BYOB bit, P must point to an int64 followed by a pointer, representing the buffer size and start address. The sqlite3_serialize() function copies the serialization of the database into that buffer if it fits and returns a pointer to it, otherwise it will just allocate like normal.
This is a diff for src/memfile.c
(manifest uuid fca8dc8b578f215a969cd899336378966156154710873e68b3d9ac5881b0ff3f).
***************
*** 474,475 ****
--- 474,476 ----
int iDb;
+ sqlite3_int64 szByob = -1;
Btree *pBt;
***************
*** 492,493 ****
--- 493,499 ----
iDb = sqlite3FindDbName(db, zSchema);
+ if( mFlags & SQLITE_SERIALIZE_BYOB ){
+ if( !piSize ) return 0;
+ szByob = *piSize;
+ pOut = *(unsigned char **)(piSize + 1);
+ }
if( piSize ) *piSize = -1;
***************
*** 499,501 ****
}else{
! pOut = sqlite3_malloc64( p->sz );
if( pOut ) memcpy(pOut, p->aData, p->sz);
--- 505,507 ----
}else{
! if( szByob < p->sz) pOut = sqlite3_malloc64( p->sz );
if( pOut ) memcpy(pOut, p->aData, p->sz);
***************
*** 520,522 ****
}else{
! pOut = sqlite3_malloc64( sz );
if( pOut ){
--- 526,528 ----
}else{
! if( szByob < sz ) pOut = sqlite3_malloc64( sz );
if( pOut ){
This was tested by a quick patch in the amalgamation as seen here: https://github.com/TjeuKayim/rusqlite/compare/8ee55b3e5a4c098ef772c8e91ef43a75959fd7f7..serialize-byob.
What do you think of this feature and the suggested API?