nelhage
Other clouds?
It looks like the current implementation is relatively tied to AWS, but in the long term is there any chance that llama could be extended to support GCP (Google Cloud Run + Google Cloud Storage look like a nearly drop-in replacement for lambda/s3) or even an arbitrary docker host/hosts and arbitrary s3-compatible object store?
You would have to do something like SQLite does and have a virtual file system. AWS S3 supports HTTP range queries, https://github.com/dacort/athena-sqlite/blob/master/lambda-function/vfs.py which can be a big performance win by not having to download entire files.
It's totally possible. S3 is already mostly hidden behind the store.Store interface, but you'd need to add a few shims/interfaces in other places. The actual surface area for interacting with AWS -- other than the initial configuration bootstrap -- is fairly small, so I don't think it would be a huge project.
gg reported substantially better results on Lambda than on Google Cloud Run, though, which is the main reason I chose to target it first.
