Issues
Issues copied to clipboard
OctopusDeploy
Build information with a large number of change sets may cause OOM exceptions when decrypting variables
Severity
Sev 3
Version
2023.3
Latest Version
I could reproduce the problem in the latest build
What happened?
A customer's deployment process includes a very large number of change sets (20000+ commits). The build information is encrypted in the variables within the Bootstrap.Octopus.FunctionAppenderContext.ps1 script, at deployment time this variable is decrypted. In this customer's case the script is ~200Mb and sporadicall results in an OOM exception on a machine with 16GB of RAM, or very slow deployments. Deployments fail ~50% of the time.
The two variables PackageBuildInformation and PackageBuildMetadata cause the large file size.
Reproduction
Create a deployment process with 20000+ commits included in the build information.
Error and Stacktrace
NotSpecified: Object reference not set to an instance of an object.
At D:\Octopus\Work\20231016083038-905811-16856\Octopus.FunctionAppenderContext.ps1:210 char:1
+ Write-Verbose "Invoking target script $OctopusFunctionAppenderTargetS ...
More Information
This issue is specific to the deployment time, there are other known issues with Build Information from a performance perspective that may not be covered by this.
Workaround
The OCTOPUS__FeatureToggles__DisableBuildInformationOnReleasesAndDeploymentsFeatureToggle feature toggle has been enabled. This seems to have resolved the issue for now, but disabled build information.
Alternatively, stop pushing build information or limit the number of commits included.
We have also seen this manifest when using C# scripts with variables in with the error message:
ERROR: Script execution failed. [COMException] No logical space left to create more user strings. (Exception from HRESULT: 0x80131198).
This occurred due to the variable's size and disabling the build information using the environment variable feature toggle set to a value of True brought the variable file size down from 12Mb to 600Kb.
If you see this sort of error when running scripts with variables, please try the workaround in this GitHub issue and see if it helps; if you could let us know this has occurred via contacting [email protected] we can add it to our investigation on this issue.
Another report (private link): https://octopus.zendesk.com/agent/tickets/184091
More details provided in the above report:
This issue doesn't occur for every schema for the same release, just occurs for specific ones which are deployed to the same server, windows version, RAM size etc.., but occurs only to few schema releases and does not for others.
[Internal link] Another customer experiencing this issue https://octopus.zendesk.com/agent/tickets/186279
Release Note: Release changes in a single deployment record no longer contain duplicate commit and work items for a package.
:tada: The fix for this issue has been released in:
| Release stream | Release |
|---|---|
| 2024.2 | 2024.2.7747 |
| 2024.3+ | all releases |
