Package gets cleaned up between parallel deployments to the same target

[veochen-octopus]

trafficstars

Severity

non-blocking

Version

any

Latest Version

I could reproduce the problem in the latest build

What happened?

When multiple deployments (e.g. tenanted) are happening at the same time on the same target, a package can get inadvertently cleaned up right before it's about to be registered for a deployment, resulting in a file not found error. Octopus performs package retention as part of package acquisition, which causes Calamari to remove packages if there's not enough disk space left. Between parallel deployments there's a chance that package retention from one of the earlier deployments will remove the referenced package in the middle of a later deployment's package acquisition.

Reproduction

Not tested. This is a race condition and requires a target with a relatively full disk.

• Create a deployment process that references a package.
• Make multiple tenanted deployments to the same target at the same time.
• Make sure one of the first few deployments triggers package cleanup.
• One of the later deployments should fail

Error and Stacktrace

01:55:09   Info     |       Unable to register package use.
01:55:09   Info     |       System.IO.FileNotFoundException: Could not find file 'C:\Octopus\Files\my-awesome-package@[email protected]'.
01:55:09   Info     |       File name: 'C:\Octopus\Files\my-awesome-package@[email protected]'
01:55:09   Info     |       at System.IO.FileInfo.get_Length()
01:55:09   Info     |       at Calamari.Common.Plumbing.FileSystem.CalamariPhysicalFileSystem.GetFileSize(String path) in C:\BuildAgent\work\e0cefbed4ad11812\source\Calamari.Common\Plumbing\FileSystem\CalamariPhysicalFileSystem.cs:line 230
01:55:09   Info     |       at Calamari.Commands.RegisterPackageUseCommand.RegisterPackageUse() in C:\BuildAgent\work\e0cefbed4ad11812\source\Calamari\Commands\RegisterPackageUseCommand.cs:line 64
01:55:09   Info     |       at Calamari.Commands.RegisterPackageUseCommand.Execute(String[] commandLineArguments) in C:\BuildAgent\work\e0cefbed4ad11812\source\Calamari\Commands\RegisterPackageUseCommand.cs:line 50
01:55:09   Error    |       Could not find file 'C:\Octopus\Files\my-awesome-package@[email protected]'.
01:55:09   Error    |       System.IO.FileNotFoundException
01:55:09   Error    |       at System.IO.FileInfo.get_Length()
01:55:09   Error    |       at Calamari.Common.Plumbing.FileSystem.CalamariPhysicalFileSystem.GetFileSize(String path) in C:\BuildAgent\work\e0cefbed4ad11812\source\Calamari.Common\Plumbing\FileSystem\CalamariPhysicalFileSystem.cs:line 230
01:55:09   Error    |       at Calamari.Commands.RegisterPackageUseCommand.RegisterPackageUse() in C:\BuildAgent\work\e0cefbed4ad11812\source\Calamari\Commands\RegisterPackageUseCommand.cs:line 64
01:55:09   Error    |       at Calamari.Commands.RegisterPackageUseCommand.Execute(String[] commandLineArguments) in C:\BuildAgent\work\e0cefbed4ad11812\source\Calamari\Commands\RegisterPackageUseCommand.cs:line 50

More Information

Logs added in shortcut

Workaround

Rerun the failed deployments

[LukeButters]

One possible cause of this issue is that during package acquisition we do a non atomic operation of:

1. finding a package
2. and later locking that package.

This means we have an opportunity for something else to sneak in and delete the package we are just about to use.

To reproduce this:

I created two projects: Project A: OctopusBypassDeploymentMutex=true and OctopusPackageRetentionPercentFreeDiskSpace=2 Project B: OctopusBypassDeploymentMutex=true and OctopusPackageRetentionPercentFreeDiskSpace=100 Both projects deploy the same package.

1. Run project A, this is important since packages are only cleaned up if and only if they are referenced in PackageRetentionJournal.json stored on the tentacle.
2. Run project A, again. Manipulating octopus with an artificial delay just before RegisterPackage.
3. Run project B, Let it run cleanup (it will clean up the package). Manipulate octopus such that for this deployment it gets delayed before staging the work
4. Let project A continue on with the register call which will fail, as in the github ticket.

[paraicoceallaigh]

another instance here: https://octopus.zendesk.com/agent/tickets/173045