Disk cache access range out of SST size

[jiacai2050]

Describe this problem

2023-09-15 10:41:18.673 ERRO [analytic_engine/src/compaction/scheduler.rs:537] Failed to compact table, table_name:xx, table_id:81909, request_id:379738764, err:Failed to build merge iterator, table:xx, err:Failed to build record batch from sst, err:Fail to read sst meta, err:Failed to decode sst meta data, file_path:0/81909/19008.sst, err:Parquet error: failed to get footer bytes, err:Generic DiskCacheStore error: Access is out of range, range:3979530..3979538, file_size:770151, last_modified:2023-09-12T08:03:07Z, file:0/81909/19008.sst.

I found above compaction error in one of our cluster, this is due to disk cache try to access SST with out-of-range requests, which is not expected.

            {
                "name": "19008.sst",
                "type": "Multipart",
                "size": 770151,
                "storage_class": "Standard",
                "last_modified": "2023-09-12T16:03:07"
            }

This is metadata of 19008.sst, as we can see, the last_modified time is equal to time in error log, which is to say file is not overwritten.

Server version

$ ceresdb-server --version CeresDB Server Version: 1.2.6-alpha Git commit: 179c4bd7a Git branch: refactor-skiplist-memory Opt level: 3 Rustc version: 1.74.0-nightly Target: x86_64-unknown-linux-gnu Build date: 2023-09-14T08:57:00.527676870Z

Steps to reproduce

Hard to reproduce now.

Expected behavior

Range should less than sst file size.

Additional Information

No response