Steve Brasier

The image `openhpc-220811-0842.qcow2` (here)[https://github.com/stackhpc/slurm_image_builder/pull/5/files] installs 9.0.3, then the upgrade task bumps to 9.0.7, then the pin to 9.0.3 at `environments/common/inventory/group_vars/all/grafana.yml:grafana_version` means the grafana role downgrades it again.

A note on certs: - The version faking for filebeat requires a custom `opensearch.yml` config file to be mounted into the container. - As we want to add an admin...

Note certs have a hardcoded 2yr life.

@m-bull I tried using `community.crypto.x509_certificate_info` to extract validity and delete if necessary, but as podman chowns everything in `certs/` the ansible loops/logic were just getting really messy. Put that on...

FIXED: that merge won't be right as we need an image using updated grafana etc.

Maybe 1s at most allowed, as per https://github.com/dun/munge/blob/77ff6823c423d19823d9259f8e0cae1fc98d9a7b/src/munged/dec.c#L984 ?

Example of failure after recreating OOD/login node: ``` @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: POSSIBLE DNS SPOOFING DETECTED! @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ The ECDSA host key for dev-login-0 has changed, and the key for the...

This commit works, but the problem is you need to know the login IP address, as seen from the OOD node: https://gitlab.com/nesi1/flexihpc-slurm/-/commit/0655b84535420b154cd33171e46cb3e579dd6f34. Might need `dig` to make it general e.g....

If just specifying the hostname, ssh will automatically add the IP to known_hosts. This fails if the IP remains the same but the hostkey changes, i.e. on rebuild

Maybe we SHOULD fix the internal (login) IP? We do I think for grafana root url anyway, as control needs to know login IP.