SQLite User Forum

Speedup BLOB
Login

Speedup BLOB

(1) By anonymous on 2023-07-25 15:07:22 [link] [source]

Hello,

I want to speedup writing to BLOB as much as possible.
The table has three INTEGER and the latest column, something like:
CREATE TABLE IF NOT EXISTS "Value" ("I" INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT, "J" INTEGER NOT NULL, "K" INTEGER NOT NULL, "B" BLOB);

Here are my database wide options:
journal_mode = wal
temp_store = memory
synchronous = normal or even off
locking_mode = exclusive

And here is how I use the API:
Using transactions.
Use static while binding the blob, and setting the pointer to a safe and already prepared memory.

I tried changing page_size to 65536 and it sped up near 30%.
I tried big (1GB) to small (4KB) BLOB and still the max speed I can get is near 1.5GB/S. In a SSD with speed of 7GB/S of sequential write and 2.5GB speed of a simple copy paste. 
And I am using latest version of SQLite with Latest Windows 10.
My code is not in C and it makes sharing it harder, but I use direct headers. I can get to speed of insert of near 100 millions of record per second if I don't bind the blob column and insert NULL, so API is not the problem.
My sole problem seems to the speed of copying the data to SQLite BLOB.

Any suggestion to get more speed or I did what I could?

(2) By Stephan Beal (stephan) on 2023-07-25 15:24:39 in reply to 1 [link] [source]

My code is not in C ... Any suggestion to get more speed...

If it's in a language other than C or C++ then you will almost certainly have extra copy overhead of your massive blobs. i'm not personally aware of any non-C/C++ binding which would permit you to pass blobs from the host language to sqlite without first copying it from that host language to C.

(3) By anonymous on 2023-07-25 15:30:10 in reply to 2 [link] [source]

There is one:
https://github.com/synopse/mORMot2
It even changed the internal memory manager of SQLite so both codes use the same memory management. And as I said, I send the pointer to Bind, and set the flag to static, so hopefully SQLite does not copy the memory and uses the provided memory.

Also, is there any benchmark on speed of BLOB? I know the one saying it is 35% faster than the file system, but the benchmark is old and talking about small blobs.

Hopefully there is benchmark checking just raw insert speed of blob, not the table insert speed.

(6.1) By mlaw (tantaman) on 2023-07-26 20:30:13 edited from 6.0 in reply to 2 [source]

for future searchers, I maintain some Rust bindings that do 0 copy transfer of strings and blobs between SQLite and Rust -- https://github.com/vlcn-io/sqlite-rs-embedded

(4) By David Raymond (dvdraymond) on 2023-07-25 15:34:46 in reply to 1 [link] [source]

If you're doing an initial load or just expanding an existing database and not updating existing records, then use a rollback journal mode and not WAL.

In WAL mode everything gets written to the wal file, then checkpointed over to the main database file. So 2 writes every time.

With the rollback journal if you're expanding the database file then not many pages need to be copied to the rollback journal, and the bulk of the data will only be written once.

(5) By anonymous on 2023-07-25 15:38:47 in reply to 4 [link] [source]

You are right. I tried with wal, and off. The reported speed of off is for 1.5GB/S (I missed noting that the max speed with all security features of is this).
With WAL, it is near 350MB/S. I consider this the slowness of check pointing. 
My concerns was speed of BLOB and not WAL.