Thomas

It doesn't look like Thanos substitutes environment variables in its config which is what's making this hard. I could be wrong, and even would very much like to be wrong...

That is not correct. The example I posed is completely transparent, the env substitutions (`$(BUCKET_NAME)`) are part of the actual `Deployment` ```sh cue export deployment_list.cue | jq -r '.deploymentList.items[].spec.template.spec.containers[].args' [...

I understand. The problem in my case is the credentials for the object storage buckets are provisioned automatically by Rook. They are then available through a ConfigMap and Secret which...

It might be possible to use an init container to create `/aws-config` but that is once again, kind of hacky.

Yeah, this is tricky. Having to install an entire operator just to read environment variables for a Thanos config seems a bit much.

FWIW I've opened an issue with Thanos asking what they think is the best way to use environment variables with configs. https://github.com/thanos-io/thanos/issues/4293

Well, here's my hacky solution for the interim. It's using an init container to push the envsubst'd config into a shared emptyDir volume. https://github.com/uhthomas/automata/blob/6bfe6628ccdc14915e2cd7242db9029d8422cbce/k8s/pillowtalk/prometheus/prometheus_list.cue Thank you

I have also run across this today and it has completely halted a single node cluster. I guess I'll have to try the workaround of just removing the DB.

Unfortunately removing the DB breaks other things. ```sh 192.168.1.102 etcd Failed ? 2m38s ago Failed to run pre stage: failed to pull image "gcr.io/etcd-development/etcd:v3.5.16": 1 error(s) occurred: failed to pull...

The solution is to just delete the entire `/var/lib/containerd` directory.