App file format, large image texture blobs?
(1) By robnik on 2021-11-05 17:06:54 [link] [source]
I'm working on a drawing/painting app for iPadOS that currently uses the "pile of files" format. I've used SQLite for several other things and I like this idea of using it for an app file format:
But my one concern is large image blobs. According to this I might get worse performance:
So I'm wondering if people think that 2nd link is still current, or if there any tricks to matching the performance of a "pile of files" when you have larger blobs, like 1-2MB.
On an iPad Pro, a full screen bitmap saved as PNG and compressed is sometimes around 100K, but if there is a gradient or airbrush or something where most of the pixels are different, it can be more like 2MB. According to that second link, that is going to hurt performance.
(2) By Richard Hipp (drh) on 2021-11-05 17:19:06 in reply to 1 [source]
The document that best answers your question is probably:
That study was done to figure out if it is more performant to put 10KB thumbnails in an SQLite database or as separate files on disk. (The answer: an SQLite database.)
Your situation is different in that you are using 2MB blobs instead of 10KB blobs. I'm not sure where the cross-over point is. You should run experiments to see. The source code for the performance measurement software is linked in the article above.
Is raw performance your only consideration? Do power-safe transactions, a simple single-file storage format, and a powerful query language figure into your decision at all? If raw performance is the only factor, then I'm guessing direct-to-disk will be the best choice for 2MB blobs (but that is only a guess - you should verify). But if other factors come into play, then you will have an engineering tradeoff to consider. Only you can make that decision.
(3) By robnik on 2021-11-05 18:26:06 in reply to 2 [link] [source]
Okay, thank you. No, performance isn't the only consideration. It's a complex situation and I will need to time some code.
(4) By Warren Young (wyoung) on 2021-11-05 19:03:03 in reply to 3 [link] [source]
In the meantime, it’s easy to change the constants in the test program, build it for your target, and run it there.