Docker image: Executable file not found in $PATH

[larkly]

When moving pods to new nodes (forcing a new image pull), the mattermost-operator deployment fails with ErrImagePull and CrashLoopBackoff, with events like:

Error: failed to start container "mattermost-operator": Error response from daemon: OCI runtime create failed: 
container_linux.go:349: starting container process caused "exec: \"mattermost-operator\": executable file not found in $PATH":
unknown

Specifying the image digest to the last known good image provided a workaround to the issue: image: mattermost/mattermost-operator@sha256:45fdb00b2e7623ab12f3b8023aa6966db6a7ebd1f4f89409692e657ed9f2d0d6

It seems like the latest built image available has an issue with its CMD.

Environment: Azure AKS 1.18.8, kubectl 1.18.2

[Szymongib]

Thank you for reporting the issue. I think that it is caused by Deployment not matching the image, as the Deployment changed slightly recently and the new image was pushed with the latest tag. I think using the tag v1.8.0 should work fine also with the existing Deployment. To use the latest again you would have to apply the latest yamls with the Operator from https://github.com/mattermost/mattermost-operator/blob/master/docs/mattermost-operator/mattermost-operator.yaml but I think the released version would be more stable.

I see that we are using the latest tag everywhere. I think it would be good to change it so that release yamls use the image tags matching the release (for example in v1.8.0 the image tag is latest). I also wonder if it would not be better to recommend installation from the latest release, not a master branch to prevent such situations in the future.

Any thoughts @jwilander, @gabrieljackson ?

[elijahomolo]

@Szymongib I'm running into the same issue as well. I'm using the tag v1.8.0 which also runs into the error.

Environment: DigitalOcean kubectl 1.19.3

[Szymongib]

Hi @elijahomolo, are you running the manifest from v1.8.0 Operator or an image with tag v1.8.0? Unfortunetly until recently we were releasing manifests with latest tag wich most likely caused the issue.

I would suggest verifying that the Operator image is not running tag latest and if so change it to v1.8.0 or update the whole manifest to the newer version.

[elijahomolo]

Hi @Szymongib the image tag is latest and the manifest from v1.8.0 Operator

[Szymongib]

Thanks for info @elijahomolo. That is probably what causes the issue as there is mismatch between the image and the rest of manifest. We have already fix an issue so that release manifest does no longer use the latest tag but it was not yet the case for v1.8.0.

To mitigate the issue I would recommend either updating whole manifest to the latest version (v1.11.1 at this time) or changing the image tag to v1.8.0.