block file number increases dramatically after switching to parquet stoage

[alburthoffman]

Describe the bug

To Reproduce Steps to reproduce the behavior:

1. Start Tempo (SHA or version): main-3e003d6
2. Perform Operations (Read/Write/Others) READ/WRITE

Expected behavior Expect the queries run faster than before. But the queries become slower. And what I suspend is the number of block keeps increasing, while the old v2 storage is very stable and only requires less than 200 blocks.

The traffic is about 300K spans per second.

Environment:

• Infrastructure: kubernetes
• Deployment tool: helm

[annanay25]

Hi @alburthoffman, thanks for opening this issue.

Yes, we are aware and working on compaction improvements to ensure the blocklist length remains as low as possible. For now, the solution is to horizontally scale compactor replicas, but we will try adding performance improvements in this area over the next couple of weeks.

Meanwhile, if you could provide us with more information on what kind of queries are being executed against Tempo and the relative slowdowns compared to v2, it would help diagnose performance bottlenecks further. Thanks!

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had any activity in the past 60 days. The next time this stale check runs, the stale label will be removed if there is new activity. The issue will be closed after 15 days if there is no new activity. Please apply keepalive label to exempt this Issue.