"Could not update application spec: getwd: no such file or directory"

[empath-nirvana]

Describe the bug Image's stop updating, logs show the above error.

time="2021-08-18T22:50:40Z" level=info msg="git config user.name argocd-image-updater" dir=/tmp/git-395417317 execID=o1WQk time="2021-08-18T22:50:40Z" level=info msg="git config user.email [email protected]" dir=/tmp/git-395417317 execID=R06hZ time="2021-08-18T22:50:40Z" level=info msg="git checkout --force main" dir=/tmp/git-395417317 execID=kqU68 time="2021-08-18T22:50:40Z" level=info msg="git clean -fdx" dir=/tmp/git-395417317 execID=FgSqg time="2021-08-18T22:50:40Z" level=error msg="Could not update application spec: getwd: no such file or directory" application=

To Reproduce Not sure how to reproduce -- we have multiple apps using image updater with kustomize back to git

Expected behavior image updater should push back to git

Additional context This is intermittent and deleting the pod fixes the issue.

Version Please tell us about the version you encountered the issue with

Logs Please paste any relevant logs here

[jannfis]

Hey @empath-nirvana, does it stop updating a single application, or all of them?

[empath-nirvana]

It seems like all of them. It was at least more than one of them.

[jannfis]

Thanks for the update.

Is the following log line from your description truncated (e.g. did you remove the application name), or is it verbatim (e.g. no application name displayed at all)?

time="2021-08-18T22:50:40Z" level=error msg="Could not update application spec: getwd: no such file or directory" application=

[jannfis]

Also, are your application manifests in a mono-repo or separate repositories?

[empath-nirvana]

I removed the application name and they’re in different repos

[empath-nirvana]

It occurred again this morning -- i tried to exec into the container and look in /tmp and it's empty -- i guess it cleans up after a failure? In any case, deleting the pod fixed the issue again.

[ghost]

Thanks for the update.

Is the following log line from your description truncated (e.g. did you remove the application name), or is it verbatim (e.g. no application name displayed at all)?

time="2021-08-18T22:50:40Z" level=error msg="Could not update application spec: getwd: no such file or directory" application=

In my case the application name is displayed.

[empath-nirvana]

This is still happening about every two weeks, reboot always clears it up -- is it possible it's a disk space problem? how would i troubleshoot that.

[jannfis]

Hm, good question about the disk space. Could well be possible that there's a bug in cleaning up Git repositories. Will investigate.

[yogeek]

same issue but as mentionned in Slack it happens only on some applications in my case. Rebooting image-updater seems to solve it eventually

[lfdominguez]

happen to me, restarting solve the problem eventually... but this is strange... why is happening this?

[SVynnytska]

This is still happening to me

[halversonian]

This unfortunately is also still happening to us every 1-3 days - we were pondering if it had to do with the frequency of deployments (or potentially multiple at same time) as it seems to occur more often when both teams are testing their deployments.

As above, our workaround is restarting the pod.

[filiprafaj]

this started happening to me, when I added write-back-target: kustomization

appart from that, I use multiple images in one app and force-update

[exequielgop]

We have the same issue when use write-back-target: kustomization. As workaround we move to a CronJob the deployment, running every 5 minutes. @jannfis do you have news about this?

[srisek]

Any traction on this issue, we are running into the same issue in my Argocd prod environment.

time="2023-06-01T18:21:34Z" level=error msg="Could not update application spec: getwd: no such file or directory" application=petclinic-dev

[stevenyongzion]

I think we have got a bit more information about this issue.

Previously we host our ArgoCD project in just 1 repo and under this repo it creates applications for many systems. All is good.

Until 3 days ago we started to split them into different repos, about 10 of them.

We now start to see this issue quite frequently.

So this issue definitely has something to do with what @halversonian mentioned above: ...with the frequency of deployments (or potentially multiple at same time) as it seems to occur more often when both teams are testing their deployments.