Yi Jin

Is there any progress on this one, we are seeing `404 not found` issue with kubernetes API for kubelet proxy: https://stackoverflow.com/questions/49275380/node-proxy-404-page-not-found, if we could disable keep alives for the connection...

Interesting, we did use our own base OS here Alpine Linux would it be reported by other thanos users? Also found another logs related to this: ``` goroutine 7613 [IO...

Report a different panic error stack trace related to HTTP2 client? ``` fatal error: unexpected signal during runtime execution panic during panic [signal SIGSEGV: segmentation violation code=0x80 addr=0x0 pc=0x437ffd] runtime...

ok, looks like this https://github.com/klauspost/compress/pull/867 is the root cause and are fixed in https://github.com/thanos-io/thanos/pull/6950, I saw thanos main picked up the newer go mod but not 0.32.5 nor [v0.33.0-rc.0](https://github.com/thanos-io/thanos/releases/tag/v0.33.0-rc.0). I...

actually it might be a false resolution, the panic seems still happening after i upgraded [email protected] but it was very infrequent, I am trying [email protected], i will let it run...

ok, it panic again with `/go/pkg/mod/github.com/klauspost/[email protected]/s2/writer.go:505 +0xb5` stack trace: ``` unexpected fault address 0x0 fatal error: fault [signal SIGSEGV: segmentation violation code=0x80 addr=0x0 pc=0x43e1f3] goroutine 6281927 [running]: runtime.throw({0x2654fb1?, 0x6d614e6472616873?}) /usr/local/go/src/runtime/panic.go:1077...

Confirmed after compile thanos with go 1.19 < 1.20, there is no panic happening ~ 10hrs, previously this would happen more than 20+ in 10 hrs for a deployments with...

it is reproducible, we are using IAM assume role in AWS instead of direct username/password, roll back to older version fixed this issue.

Speaking about the object storage layout, taking Grafana Mimir as an example, they have layout breakdown by tenant while thanos today put all tenants + raw resolution + 5m/1h downsampled...

hi @prymitive, any updates on this change, we hit this issue as well during WAL replay