The index files under the "locations" folder cannot be completely deleted.

[zifengmo]

BUG REPORT The index files with the .sst extension under the "locations" folder cannot be completely deleted.

Describe the bug

After upgrading Pulsar to version 3.0.1, I encountered an issue where the index files with the .sst files under the 'locations' folder in the Bookie directory cannot be completely deleted. In clusters with a small data volume, the behavior seems normal, with only a few .sst files for the current day. However, in clusters with high message throughput, such as 20,000 messages per second, I observe that files from several days ago are not being deleted. For instance, one of our clusters accumulated 40 GB of .sst index files over 20 days.After using the command 'bin/bookkeeper shell rebuild-db-ledger-locations-index' to rebuild the index, the overall space occupancy has reduced to 200 MB.

199M locations 46G locations.BACKUP-2023-12-06T03:26:40.383+0000

When examining the 'LOG' files, I found deletion records for some other files, but many files are still not getting deleted. The issue seems to be more pronounced in clusters with a large message volume.

2023/12/06-07:42:27.295286 140508326868736 [le/delete_scheduler.cc:78] Deleted file /data/bookkeeper/ledgers/current/locations/030778.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023/12/06-07:42:27.296158 140508326868736 [le/delete_scheduler.cc:78] Deleted file /data/bookkeeper/ledgers/current/locations/030776.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023/12/06-07:42:27.297293 140508326868736 [le/delete_scheduler.cc:78] Deleted file /data/bookkeeper/ledgers/current/locations/030774.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000

Could you please provide guidance or assistance on resolving this issue?

To Reproduce

Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior

A clear and concise description of what you expected to happen.

Screenshots

If applicable, add screenshots to help explain your problem.

Additional context

Add any other context about the problem here.