gitops run: If dev-ks never becomes healthy, then eventually it goes missing

[ozamosi]

If I:

• Try to run gitops run against something broken
• Don't fix the reason it's breaking
• Wait 5 minutes or so

Then the dev-ks will be completely deleted, and the only way to recreate it is to turn gitops run off and on again.

I expect that I will be able to work to fix the reason it's failing for as long as I want, and whenever I fix it, everything will start to work.

[JamWils]

In good news, I have left a "healthy" gitops run running for days and it still works when I come back :)

[chanwit]

I think this bug is already fixed as a side effect of YAML validation.

[opudrovs]

@chanwit cool!

[chanwit]

Fixed by #2970

[makkes]

Tried to reproduce it in both session and non-session mode:

1. gitops run [--no-session] .
   

2. The Kustomization isn't ready:

   flux-system   run-dev-ks   3m24s   False   kustomize build failed: accumulating resources: accumulation err='accumulating resources from 'deployment.yaml': open /tmp/kustomization-3295192071/kaputt/deployment.yaml: no such file or directory': must build at directory: not a valid directory: eval symlink failure on '/tmp/kustomization-3295192071/kaputt/deployment.yaml' : lstat /tmp/kustomization-3295192071/kaputt/deployment.yaml: no such file or directory
   

3. Wait 10 minutes
4. Kustomization is still there:

   $ k -n flux-system get ks run-dev-ks  
   NAME         AGE     READY   STATUS
   run-dev-ks   10m   False   kustomize build failed: accumulating resources: accumulation [...]