datafuselabs
feat: add object cache for stage parquet file
Summary
Now, the Databend object cache only supports the FUSE engine table. This query cannot be accelerated:
select * from 's3://aa/bb/cc/' (pattern => '.*.parquet')
However, we can enable it for the stage files with the parquet format.
Since you are working on the new parquet crate, cc @youngsofun @RinChanNOWWW
This is a low prioirty issue.
The high priority is finding a way to cache the metadata of the stage parquet (in memory?), similar to the approach used in DuckDB as discussed here: In-memory cache of Parquet data? Persistently cache Parquet metadata
It looks we have it already: FileMetaDataCache, but not support stage parquet file.
Hi @zenus ,
I apologize for any confusion caused by this issue. It's important to note that we should avoid caching for stage files. Attempting to control the caching area could lead to significant issues, particularly with some write operations.