SQLite Forum

Network filesystems for sqlite DBs
Login
> That will perform much better on an SSD than your test predicts

Yep. The test wasn't really about the SSD performance though. 

> I can tell you from experience that you want to batch that up and save many state changes all at once

So there's another wrinkle. The current system runs with a REDIS database that manages all the flags for the app, and as I said, the app will open multiple connections to the service at once, which means it gets different IP addresses on the load-balancer and so different connections to the REDIS database. If the app round-robins the commands it sends, and command N sets state to X, then command N+1 depends on that state, we have a problem if we're batching things.

Performance isn't something I'm overly concerned with, at least compared to reliability and correctness. Batching is a good idea in general, don't get me wrong, but specifically here I think it could be a problem. I don't get to change the way the client application works apart from under *really* difficult circumstances.

> Only if you run it from multiple nodes

Understood, and I intend to rectify that. It will be interesting to see if the results over the network are even vaguely similar.

> Informed guess: you aren't testing against NVMe on the server side of the NFS connection

Possibly. The NFS server in question is an Asustor AS6510T, with 10 x 16TB drives, 32 GB of RAM and 2TB of NVME SSDs as cache to the array. The SSD's on the device are specifically to cache the RAID (they don't contribute to the storage volume) but it's entirely possible it is using the hierarchy of RAM <-> SSD <-> HDD, given that it's a special-purpose device. And yes, because I'm paranoid, it does have its own UPS*.

> There are several organizations that will happily rent you a Gluster or Ceph cluster for an hour or two

The last time I tried to set up an AWS account for a quick test, it cost me ~$500 or so. I'd not noticed the various data-transfer fees, and when I *thought* I'd closed everything down, I'd missed something (can't remember what, now) so I paid a month's usage of that. These days I'm very leery of setting up things on time-and-data-based services. 

The suggestion has merit, if I can persuade someone else to foot any bills :)


[*] I have a bunch of UPSs that I had before the whole-house battery was installed. I deploy them judiciously.