SQLite Forum

Increasing insert performance of an in-memory db
Login
# Some context
We have an in-memory database with a simple table(3 integers and a string, the first 3 numbers form the primary key, no other indexes at this point ). We need to create the DB from scratch on launch and populate 100 million rows in the table. The DB grows to ~9GB after it's completely populated. Currently it takes around 90 minutes which means the insertion rate is ~18000 rows/second. 

Our code is written in C++. I have tried everything in [this thread](https://stackoverflow.com/questions/1711631/improve-insert-per-second-performance-of-sqlite). My machine is on Azure with 12 cores and has 128 GB of RAM. We compile sqlite with `SQLITE_WIN32_MALLOC` and `SQLITE_WIN32_HEAP_CREATE`.


# On profiling:
I see that around 75% of the time is spent in `memcpy_repmovs()`. 97% of the time is spent in `sqlite3BtreeMovetoUnpacked()`. 

# Questions
1) Why does `memcpy()` hog so much of the CPU when we're only storing around ~1.5MB/sec? Is this normal?
2) For storing around 100 million rows of around 90 byes each, what do you expect the best case insertion rate to be? I'm trying to set my own expectations of what we can achieve with our current approach.

Thanks,

Bhargava