Compactor: Clean up file is not block visit marker log when using S3 bucket

[yeya24]

Describe the bug When using S3 bucket client, there are excessive logs in Compactor saying file is not block visit marker during iter operation.

err="file is not block visit marker" operation="Iter

The error is thrown at https://github.com/cortexproject/cortex/blob/master/pkg/compactor/blocks_cleaner.go#L466. When trying to clean up partial blocks, we return this error when compactor finds a non visit marker file to terminate iteration early.

However, even though the error is expected, the error still got returned from the bucket client. When using S3 client, it will be retried and eventually the error will be logged at https://github.com/cortexproject/cortex/blob/master/pkg/storage/bucket/s3/bucket_client.go#L135.

To Reproduce Steps to reproduce the behavior:

1. Start Cortex 1.16.0 Compactor

Expected behavior There shouldn't be any logs for file is not block visit marker as it is not an error.

[yeya24]

I opened https://github.com/thanos-io/objstore/pull/103 to fix this issue

[kryachkov]

Hi @yeya24! Can this issue be affecting compactor performance? I get this log message a lot and it feels like compactor is not doing it's job as I often get "bucket index is too old" errors

[yeya24]

Hi @kryachkov, for this error specifically the answer is no. It is just annoying as it updates the failure counter even though the error is expected.

Regarding the issue you encountered, I think it is mainly relevant to your setting. Try increase your bucket index max stale time and block upload time. It could be also related to your object storage performance if you have a large amount of objects in your bucket. But I think the issue was fixed in the latest 1.17.0 release.

[friedrichg]

Still happening in v1.18.1

ts=2024-12-10T10:09:31.527783067Z caller=bucket_client.go:141 level=error msg="bucket operation fail after retries" err="file is not block visit marker" operation="Iter tenant-1/01JEMEWYHEEY2KEGSJ0Z1H5NHE"

[stale[bot]]

This issue has been automatically marked as stale because it has not had any activity in the past 60 days. It will be closed in 15 days if no further activity occurs. Thank you for your contributions.