codedeploy-agent gets stuck on the wrong deployment ID, incorrectly reports missing scripts

[gh-andre]

CodeDeploy agent keeps trying to access an invalid application revision and fails to find scripts to run in application events. The original configuration where this occurred is a CodePipeline instance, but now when I create a deployment just for this application and deployment group, it keeps selecting the same bad deployment revision.

It's happening on Ubuntu 18.04. Not sure how to obtain codedeploy-agent version, as it doesn't implement --version. The closest I can see is in the logs:

 Version file found in /opt/codedeploy-agent/.version with agent version OFFICIAL_1.4.1-2244_deb

The error I'm getting is this:

Notice that the deployment ID following the deployment group ID is different from the actual deployment shown in details. This is the problem - that other deployment ID isn't even for this application, but another application that was deployed to the same VM at a different time.

Checking this deployment, it is properly picking up correct build artifacts, which do contain the script that it reports to be missing:

The revision location shown in details does contain the script that is reported as missing, so codedeploy-agent is definitely trying to refer to a bogus deployment ID when it is trying to locate the application revision.

Looking at the codedeploy-agent logs, it just reports this error that is visible in UI and no other useful information. In verbose mode, it references the correct application revision and, on the next log line after this, is trying to open appspec.yml for the bogus deployment ID reported in the error.

I also noticed that deployment ID referenced in revision details in the screenshot above is different for manually-created deployments in that it may be different from the current deployment ID in the same screenshot and seems to be referring to the deployment ID that was active at the time CodePipeline was running, when it got a notification from the CodeBuild stage. Mentioning this because it sounds plausible that this is where the original deployment ID mix-up happened.