SQLite Forum

Feedback on writing a SQLite VFS for a distributed object store
Login
> it could do priming reads for a range of pages it will need in the future and let the VFS asynchronously fetch/cache those reads for a future blocking read. 

This is kind of what I was getting at with my suggestion that we look at prefetching interior b-tree pages, as I think having them upfront should greatly reduce the need for serial chains of network roundtrips to navigate the database file.

The problem (AFAIK; I'd be delighted to be corrected) is that those pages are scattered arbitrarily throughout the database file, without any tendency to contiguity, even after vacuum. So, short of changing the b-tree and pager implementation to promote that, we'd have to maintain somewhere a consistent list of the desired page numbers, and preferably even a full copy of their content, to be fetched upfront.

> All I do know is that trying to implement some kind of readahead in the VFS would be unlikely to yield the benefits we would like.

My experience in sqlite_web_vfs and sqlite_zstd_vfs is that readahead does work great for cases it should obviously help, namely, sequential scans of large tables in vacuumed database files. But, the "time to first result row" can be pretty bad due to the aforementioned need to serially read numerous scattered pages, in order to navigate to the part of the database file that'll be read sequentially.

* https://github.com/mlin/sqlite_web_vfs
* https://github.com/mlin/sqlite_zstd_vfs