litestream icon indicating copy to clipboard operation
litestream copied to clipboard

`litestream restore` cache objects on local file system for fast restore?

Open broccolihighkicks opened this issue 3 years ago • 1 comments

I want to use litestream restore to download a SQLite DB from cloud storage to run read queries against it.

Before my process runs I will run litestream restore to fetch and create the database file.

Is it possible to cache previously downloaded objects from cloud storage, seeing as they are immutable?

This would read them from the file system, and then just fetch the newest remote objects. This would theoretically reduce the speed of reconstructing the database.

An example is to run litestream restore at the start of a serverless HTTP handler. The first request may be a "cold boot" taking 10s to download 100% of the file, but subsequent requests would only need to read the 1% of newest changes, which would be much less than 10s?

broccolihighkicks avatar Feb 18 '22 17:02 broccolihighkicks

That's a good idea. It doesn't seem like it'd be too hard to implement. I probably won't be able to put it in before v0.4.0 but I can implement it afterward. 👍

benbjohnson avatar Feb 18 '22 21:02 benbjohnson