OctopusDeploy
Step Templates sourced from Git can cause a deployment to fail when deploying to multiple targets
Severity
Medium - No practical workaround
Version
Confirmed in 2023.4.8245
Latest Version
None
What happened?
Step Templates sourced from Git can cause a deployment to fail when deploying to multiple targets. It appears that Octopus is unnecessarily attempting to create a package/artifact from Git multiplied by the number of targets, all using the exact same file path and name.
Octopus should only create one artifact since this will be the same for all targets of the same deployment.
Based on my testing, self-hosted Windows-based Octopus Servers are affected by the file contention/race condition. On Linux/Octopus Cloud Octopus Servers, however, the duplicate packages/artifacts still occurs, just without error.
Reproduction
On a self-hosted installation of Octopus Server on Windows:
- Add a Step Template from Git to a Project process or Runbook process
- Deploy to multiple targets
- If the deployment doesn't fail, retry a few times
Error and Stacktrace
The process cannot access the file 'C:\Octopus\Packages\git-resources\b56df92d813bf9ab7988d6f8e92d2dbe8481294a@O7e24c0b27f526b3e21cf0ee16b651d05145acc73@da39a3ee5e6b4b0d3255bfef95601890afd80709.zip' because it is being used by another process.
More Information
2 Targets:
3 Targets:
Workaround
No response
Release Note: Fixed an issue where file access to the same git resource in parallel would cause a deployment failure.
The original fix for this issue resolved the problem within a single deployment. Parallel deployments across separate projects still experience this issue. We haven't prioritized this work but will update here shortly.
Hey @iseneirik I've re-raised this one internally to bump the priority up. It's not immediately on the list but if this is causing serious impacts for you that will help raise the priority. How frequently is this occurring for you?
Thanks for re-raising it, @IsaacCalligeros95.
In one of our spaces, four separate projects are triggered by the same build in TeamCity (they all push packages to Octopus Deploy at approximately the same time), and those projects failed with deployment every time (they all use the same Step Template). So for now, the Step Templates causing the issue are disabled as they were not critical in the deployment process (they logged some stuff, basically).
We have quite a few spaces is our organization which have a few common Step Templates. So to save (a lot of) time with maintenance of the Step Templates we have started putting them in scripts living in a common git repository which all spaces use.
The bug is not a production-breaking impact for us, but if there is no plan on fixing this in the near(ish) future, we will have to go back to inline scripts for the Step Templates, which will cause some overhead.
Hello, we just encountered this as well.
Octopus Deploy Version: 2023.4.8582
In our case we're deploying the same project across multiple environments in parallel. It fails under the Aquire packages with the error: The process cannot access the file '<REDACTED>.zip' because it is being used by another process.
This happened pretty much immediately after we started using git step templates for the canary project. I assume this will happen for a few environments every time we deploy.
We were planning to roll out git for a few step templates this week but this is a showstopper for us as we have a lot of environments for all our projects. We'll have to stick with inline source code for now.
[Internal Link] Customer experiencing this issue when running deployments in parallel https://octopus.zendesk.com/agent/tickets/201612
Currently in the process of investigating if this can be backported to 2024.2 versions
:tada: The fix for this issue has been released in:
| Release stream | Release |
|---|---|
| 2023.4 | 2023.4.8351 |
| 2024.1 | 2024.1.9772 |
| 2024.2 | 2024.2.1824 |
| 2024.3+ | all releases |
