Matt Wise

Thanks - next week I will try reproducing this again and run logdog. Is there a private place I can upload the logs to. Should I open an AWS ticket...

@bcressey can we get an updated BottleRocket release with these K8S versions: https://github.com/kubernetes/kubernetes/issues/118261#issuecomment-1790657622?

Yes Please!

> > The problem with how terraform does this, is that it makes refactoring extremely difficult, since the names are based on where they are located (how you compose your...

Seeing this as well - luckily the `--toolkit-stack-name` hack helped us get around it for now.

@jmazzitelli, The challenge is that it's not just that the metric name changes - the fundamental value changes. In this case, we're taking a `counter` with a ton of high...

Yes - I agree that it's not a terribly great situation to be in.. Unfortunately the docs at https://istio.io/latest/docs/ops/best-practices/observability/#federation-using-workload-level-aggregated-metrics do not address the issue with aggregating `counters`... but pre-calculating the...

Even the Istio team points out that this kind of optimization may be necessary at https://istio.io/latest/docs/ops/best-practices/observability/#optimizing-metrics-collection-with-recording-rules..

@jshaughn So - thinking more about this, it seems actually quite reasonable for Kiali to support these pre-aggregated metrics, but at the same time require that the operator include `rate1m`,...

Here's our current set of recording rules.. 1. For `_total` metrics we are recording them as `rate()`... 2. For Histograms and Gauges (`_bucket`, `_count`, `_sum`) we are leaving them as...