SQLite Forum

Regarding memory mapped i/o
Login

Regarding memory mapped i/o

(1) By Vijay (Vijay_123) on 2022-01-19 12:54:29 [link] [source]

Hi team,

I am new to memory mapped I/o concept. I am currently trying to implement memory mapped i/o so that my i/o intensive operations are fast.

Can anyone let me know how to implement this? Any suitable example link or steps will be helpful.

Thanks in advance.

(2) By Simon Slavin (slavin) on 2022-01-19 13:47:38 in reply to 1 [source]

You should read this page

https://www.sqlite.org/mmap.html

Given your question, note especially

Performance does not always increase with memory-mapped I/O. In fact, it is possible to construct test cases where performance is reduced by the use of memory-mapped I/O.

If you have specific questions, please ask, giving us enough information about your setup to be able to make good guesses.

(3) By Gunter Hick (gunter_hick) on 2022-01-20 07:29:45 in reply to 1 [link] [source]

I would suggest investing into database and query design first, which can lead to performance improvements by several orders of magnitude, before focusing on improving I/O speed - which is likely to yield only percentage improvements.

(4) By MBL (UserMBL) on 2022-01-20 08:09:40 in reply to 1 [link] [source]

I/O intensive operations ...

What is your use case? Is your process the only one or is concurrency an issue for your use case? If you can provide some more info about your application(s) you probably will get some good hints in this forum of what can be done better other than creating an own I/O concept. (I guess you are not the first and only person having thought about such work-to-do.)

Where I do not require persistence but speed in my use cases I try to make use of an in-memory database.

For my own write intensive use case in a single process I can say that SQLite3 is really fast in doing that job. I simply avoid any indexing and even the definition of a primary key - the only key I really rely on -when querying later on- is the ROWID to get the records out in original and correct sequence.

In others of my applications I try to do multiple inserts/updates within a single transaction, which has high impact on speed in my environment. Where avoidance of duplicates is required (because of key frame overlap) I use a primary key with the on conflict ignore clause.

(5) By Keith Medcalf (kmedcalf) on 2022-01-20 23:16:02 in reply to 1 [link] [source]

Memory Mapped I/O does not increase I/O speed. The only wy to "speed up" I/O is to not do it at all. This has been the case for more than a century and is unlikely to change in the future unless the underlying limiting factor (called Physics) changes. Unfortunately, these underlying limiting factors have been stable and unchanging for BILLIONS of years and are unlikely to change in your lifetime.

Memory-mapped I/O merely eliminates a buffer copy operation and might also eliminate a transition between user and supervisor mode.