Tenant can't be deleted if it has a common variable value and was last connected to a deleted project and no others

[nharvey-octopus]

Severity

Few reports recently

Version

Repro'd in 2024.3.8336

Latest Version

I could reproduce the problem in the latest build

What happened?

"Sequence contains no elements" error is thrown if you attempt to delete a tenant with a common variable value where the only connected project was deleted.

Reproduction

1. Connect variable template to project
2. Connect tenant to same project with an environment
3. Update common variable value in tenant
4. Delete project
5. Attempt to delete tenant
6. Result is receiving error "Sequence contains no elements"

Error and Stacktrace

System.InvalidOperationException: Sequence contains no elements
   at System.Linq.ThrowHelper.ThrowNoElementsException()
   at System.Linq.Enumerable.Aggregate[TSource](IEnumerable`1 source, Func`3 func)
   at Octopus.Core.Security.Restrictions.ModelRestrictionMap.TenantVariableRestriction.TenantVariableRestrictionFilter.<>c__DisplayClass10_0.<ShouldInclude>g__HasPermissionWhenOwnerIsLibraryVariableSet|2(Tenant tenant) in ./source/Octopus.Core/Security/Restrictions/ModelRestrictionMap/TenantVariableRestrictionDefinition.cs:line 375
   at Octopus.Core.Security.Restrictions.ModelRestrictionMap.TenantVariableRestriction.TenantVariableRestrictionFilter.<>c__DisplayClass10_0.<ShouldInclude>b__4() in ./source/Octopus.Core/Security/Restrictions/ModelRestrictionMap/TenantVariableRestrictionDefinition.cs:line 364
   at Octopus.Core.Security.Restrictions.ModelRestrictionMap.TenantVariableOwnerProvider.UsingProjectOwner[T](TenantVariable variable, Func`1 whenOwnerIsProject, Func`1 whenOwnerIsLibraryVariableSet) in ./source/Octopus.Core/Security/Restrictions/ModelRestrictionMap/TenantVariableRestrictionDefinition.cs:line 41
   at Octopus.Core.Security.Restrictions.ModelRestrictionMap.TenantVariableRestriction.TenantVariableRestrictionFilter.<>c__DisplayClass10_0.<ShouldInclude>g__HasOtherViewPermissions|0() in ./source/Octopus.Core/Security/Restrictions/ModelRestrictionMap/TenantVariableRestrictionDefinition.cs:line 361
   at Octopus.Core.Security.Permissions.RestrictionResultExtensionMethods.And(RestrictionResult result, Func`1 otherRestrictionResultFactory) in ./source/Octopus.Core/Security/Permissions/RestrictionResult.cs:line 44
   at Octopus.Core.Security.Restrictions.ModelRestrictionMap.TenantVariableRestriction.TenantVariableRestrictionFilter.ShouldInclude(TenantVariable model, ISpacePartitionPrincipal principal, IOctopusQueryExecutor queryExecutor) in ./source/Octopus.Core/Security/Restrictions/ModelRestrictionMap/TenantVariableRestrictionDefinition.cs:line 357
   at Octopus.Core.Security.Restrictions.AccessChecker`1.CanView(TModel model) in ./source/Octopus.Core/Security/Restrictions/AccessChecker.cs:line 147
   at Octopus.Core.Security.Permissions.AccessCheckerExtensionMethods.AssertCanView[TModel](IAccessChecker`1 accessChecker, TModel model) in ./source/Octopus.Core/Security/Permissions/IAccessChecker.cs:line 28
   at Octopus.Server.Web.Infrastructure.OctopusQueryExecutor.LoadAsync[TDocument,TKey](TKey id, CancellationToken cancellationToken)
   at Octopus.Core.Persistence.Database.SlugDecorators.SlugQueryExecutorDecorator.LoadAsync[TDocument,TKey](TKey id, CancellationToken cancellationToken)
   at Octopus.Core.Persistence.Database.DatabaseDocumentStore`2.GetOrNullAsync(TKey id, CancellationToken cancellationToken)
   at Octopus.Core.Persistence.Git.SlugDocumentStoreDecorator`2.GetOrNullAsync(TKey id, CancellationToken cancellationToken)
   at Octopus.Core.Persistence.ChangeTracking.NevermoreAndGit.NevermoreAndGitChangeTrackingDocumentStoreDecorator`2.GetOrNullAsync(TKey id, CancellationToken cancellationToken)
   at Octopus.Core.Persistence.Auditing.AuditingDocumentStoreDecorator`2.GetOrNullAsync(TKey id, CancellationToken cancellationToken)
   at Octopus.Core.Persistence.Database.Deletion.DeleteRelatedDocumentsDocumentStoreDecorator`2.GetOrNullAsync(TKey id, CancellationToken cancellationToken)
   at Octopus.Core.Persistence.Database.Deletion.VetoDocumentStoreDecorator`2.GetOrNullAsync(TKey id, CancellationToken cancellationToken)
   at Octopus.Core.Persistence.Database.AccessCheckingDocumentStoreDecorator`2.GetOrNullAsync(TKey id, CancellationToken cancellationToken)
   at Octopus.Core.Persistence.DocumentStore`2.GetOrNullAsync(TKey id, CancellationToken cancellationToken)
   at Octopus.Core.Persistence.DeleteDocumentDocumentStoreAdapter`2.RemoveAsRelatedAsync(TDocument document, CancellationToken cancellationToken)
   at Octopus.Core.Persistence.Database.Indexes.DeleteRelatedDocumentsCommand.AsyncDeleteRelatedDocumentById[TRelatedDocument,TRelatedId](TRelatedDocument document, CancellationToken cancellationToken) in ./source/Octopus.Core/Persistence/Database/Indexes/DeleteRelatedDocumentsCommand.cs:line 226
   at Octopus.Core.Persistence.Database.Indexes.DeleteRelatedDocumentsCommand.DeleteRelatedDocumentAsync[TRelatedDocument](TRelatedDocument document, CancellationToken cancellationToken) in ./source/Octopus.Core/Persistence/Database/Indexes/DeleteRelatedDocumentsCommand.cs:line 214
   at Octopus.Core.Persistence.Database.Indexes.DeleteRelatedDocumentsCommand.ExecuteDeleteRuleAsync(RelationshipIdentifier relationship, IReferencingDocument reference, CancellationToken cancellationToken) in ./source/Octopus.Core/Persistence/Database/Indexes/DeleteRelatedDocumentsCommand.cs:line 188
   at Octopus.Core.Persistence.Database.Indexes.DeleteRelatedDocumentsCommand.DeleteAsync[TDocument,TKey](TDocument document, IOctopusQueryExecutor queryExecutor, CancellationToken cancellationToken) in ./source/Octopus.Core/Persistence/Database/Indexes/DeleteRelatedDocumentsCommand.cs:line 129
   at Octopus.Core.Persistence.Database.Deletion.DeleteRelatedDocumentsDocumentStoreDecorator`2.RemoveAsync(TDocument document, CancellationToken cancellationToken, Boolean enforceVetoRules) in ./source/Octopus.Core/Persistence/Database/Deletion/DeleteRelatedDocumentsDocumentStoreDecorator.cs:line 64
   at Octopus.Core.Persistence.Database.Deletion.VetoDocumentStoreDecorator`2.RemoveAsync(TDocument document, CancellationToken cancellationToken, Boolean enforceVetoRules) in ./source/Octopus.Core/Persistence/Database/Deletion/VetoDocumentStoreDecorator.cs:line 67
   at Octopus.Core.Persistence.Database.AccessCheckingDocumentStoreDecorator`2.RemoveAsync(TDocument document, CancellationToken cancellationToken, Boolean enforceVetoRules) in ./source/Octopus.Core/Persistence/Database/AccessCheckingDocumentStoreDecorator.cs:line 105
   at Octopus.Core.Persistence.DocumentStore`2.RemoveAsync(TDocument document, CancellationToken cancellationToken, Boolean enforceVetoRules) in ./source/Octopus.Core/Persistence/DocumentStore.cs:line 103
   at Octopus.Core.Features.Tenants.DeleteTenantCommandHandler.Handle(DeleteTenantCommand command, CancellationToken cancellationToken) in ./source/Octopus.Core/Features/Tenants/DeleteTenantCommandHandler.cs:line 28
   at Octopus.Core.Infrastructure.Mediator.AutofacMediator.Do[TCommand,TResponse](ICommand`2 command, CancellationToken cancellationToken)
   at Octopus.Core.Infrastructure.Mediator.Decorators.SystemComponentModelValidationDecorator.Do[TCommand,TResponse](ICommand`2 command, CancellationToken cancellationToken)
   at Octopus.Core.Infrastructure.Mediator.Decorators.FluentValidationsDecorator.Do[TCommand,TResponse](ICommand`2 command, CancellationToken cancellationToken)
   at Octopus.Core.Infrastructure.Mediator.Decorators.MessageBusSiphoningDecorator.Do[TCommand,TResponse](ICommand`2 command, CancellationToken cancellationToken)
   at Octopus.Server.Web.Controllers.Tenants.DeleteTenantController.DeleteTenant(DeleteTenantCommand command, CancellationToken cancellationToken) in ./source/Octopus.Server/Web/Controllers/Tenants/DeleteTenantController.cs:line 16
   at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.TaskResultExecutor.Execute(ActionContext actionContext, IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeActionMethodAsync>g__Logged|12_1(ControllerActionInvoker invoker)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeInnerFilterAsync>g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>g__Awaited|20_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Logged|17_1(ResourceInvoker invoker)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Logged|17_1(ResourceInvoker invoker)
   at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|7_0(Endpoint endpoint, Task requestTask, ILogger logger)
   at Octopus.Server.Web.Middleware.BoundaryTrailerRewriteMiddleware.Invoke(HttpContext context, IAutomationContext automationContext) in ./source/Octopus.Server/Web/Middleware/BoundaryTrailerRewriteMiddleware.cs:line 45
   at Octopus.Server.Web.Middleware.OpenFeatureUserContextMiddleware.InvokeAsync(HttpContext httpContext, IOctopusPrincipalInternal principal, IFeatureClient client) in ./source/Octopus.Server/Web/Middleware/OpenFeatureUserContextMiddleware.cs:line 31
   at Octopus.Server.Web.Infrastructure.Authentication.AuthorizationMiddlewareResultHandler.HandleAsync(RequestDelegate next, HttpContext context, AuthorizationPolicy policy, PolicyAuthorizationResult authorizeResult) in ./source/Octopus.Server/Web/Infrastructure/Authentication/AuthorizationMiddlewareResultHandler.cs:line 50
   at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
   at Octopus.Server.Web.UnitOfWorkMiddleware.InvokeAsync(HttpContext httpContext, IUnitOfWork unitOfWork) in ./source/Octopus.Server/Web/UnitOfWorkMiddleware.cs:line 31
   at Octopus.Server.Web.UnitOfWorkMiddleware.InvokeAsync(HttpContext httpContext, IUnitOfWork unitOfWork) in ./source/Octopus.Server/Web/UnitOfWorkMiddleware.cs:line 42
   at Octopus.Server.Web.Middleware.OctopusClientOldVersionWarningMiddleware.InvokeAsync(HttpContext context, IAutomationContext automationContext) in ./source/Octopus.Server/Web/Middleware/OctopusClientOldVersionWarningMiddleware.cs:line 24
   at Octopus.Server.Web.Middleware.DynamicContentHeadersMiddleware.InvokeAsync(HttpContext context) in ./source/Octopus.Server/Web/Middleware/DynamicContentHeadersMiddleware.cs:line 50
   at Octopus.Server.Web.Middleware.MaintenanceModeMiddleware.InvokeAsync(HttpContext context) in ./source/Octopus.Server/Web/Middleware/MaintenanceModeMiddleware.cs:line 58
   at Octopus.Server.Web.Middleware.OctopusAuthenticationMiddleware.InvokeAsync(HttpContext context, EnableSkipAuthenticationAttributeFeatureToggle enableSkipAuthenticationAttributeFeatureToggle, IUserAuthenticator userAuthenticator, IUserSessionService userSessionService, IWebAuthCache authCache, ILogger logger) in ./source/Octopus.Server/Web/Middleware/OctopusAuthenticationMiddleware.cs:line 71
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
   at Octopus.Server.Web.Middleware.TelemetryMiddleware.InvokeAsync(HttpContext context) in ./source/Octopus.Server/Web/Middleware/TelemetryMiddleware.cs:line 64
   at Octopus.Server.Web.Middleware.ErrorHandlingMiddleware.InvokeAsync(HttpContext context) in ./source/Octopus.Server/Web/Middleware/ErrorHandlingMiddleware.cs:line 51

More Information

This appears to be related to https://github.com/OctopusDeploy/Issues/issues/6526 but with a more specific method of reproduction.

Workaround

Connect any project and environment to the tenant then delete.

[nharvey-octopus]

Recent affected customers (Internal) - 196356 197528

[patrick-smergut-octopus]

Another report [internal] 207686

[nharvey-octopus]

Another instance of this [internal] 221540

[octoreleasebot]

Release Note: Fix issue where deleting tenants resulted in an error after all connected projects are removed while tenant variables still exist. When projects are deleted, now correctly removes the related tenant variables for that project.

[Octobob]

:tada: The fix for this issue has been released in:

Release stream	Release
2025.1	2025.1.8850
2025.2+	all releases