thanos
thanos copied to clipboard
thanos-io
offset table size exceeds 4 bytes & symbol table size exceeds 4 bytes || Thanos Compactor error
Discussed in https://github.com/thanos-io/thanos/discussions/6455
Originally posted by abhishekgoel1410 June 18, 2023 Can anybody suggest the root cause & ways to handle these issues. Facing these 2 highlighted errors in thanos compactor component. I did remove the corrupted blocks to some backup s3 bucket and restarted the compactor process but it re-occurred after a few hours only. Due to this memory & storage factors are getting exhausted for storegateway component.
level=error ts=2023-06-06T06:27:00.33013803Z caller=compact.go:495 msg="critical error detected; halting" err="compaction: group 0@12094533570599070506: compact blocks [/data/compact/0@12094533570599070506/01H13xxxxx8XP /data/compact/0@12094533570599070506/01H16xxxxPXHD /data/compact/0@12094533570599070506/01H16xxxxxBT4F /data/compact/0@12094533570599070506/01H162xxxxx33A0N /data/compact/0@12094533570599070506/01H1xxxxxx3HAFT /data/compact/0@12094533570599070506/01H1xxxxxWPJBM]: postings offset table size exceeds 4 bytes: 4355208691"
level=error ts=2023-06-06T03:39:06.1064242Z caller=compact.go:495 msg="critical error detected; halting" err="compaction: group 0@15540937320163341612: compact blocks [/data/compact/0@15540937320163341612/01H19Hxxxxxx49T3 /data/compact/0@15540937320163341612/01H19JxxxxxG9RB /data/compact/0@15540937320163341612/01H19xxxx0WPB /data/compact/0@15540937320163341612/01H1xxxxADF03 /data/compact/0@15540937320163341612/01H1xxxxAXCN3 /data/compact/0@15540937320163341612/01H19xxxxx83VG8]: 2 errors: populate block: add series: symbol table size exceeds 4 bytes: 4634597500; symbol table size exceeds 4 bytes: 4634597500"
This is something we definitely need to address from Compactor. Right now users can only add no compact marker. Ideally Compactor should shard/partition the block somehow. Similar to the index size too big issue
hello, are there any other ways to approach this issue without using "no compact" marker?
Unfortunately I don't think there is any solution available now. Long term solution is Prometheus changes its index format to support it or Thanos has a compactor which supports vertical splitting blocks.
Users have to split their metrics into multiple shards manually somehow in order to produce smaller blocks