Azure
Feature: Support Deployment Stacks
Is this just "we want support for arbitrary resources" (tracking as #4028), or do you actually think that there's a use for the Stacks grouping mechanism in Kubernetes? AFAIK the problem Stacks is solving is "it's hard to delete resources in ARM because nobody likes Complete deployment mode". That same problem doesn't exist in Kubernetes (and by extension doesn't exist in ASO).
if you want to delete a resource, you just kubectl delete it (or delete it from Flux or whatever).
DeploymentStacks enables a different lifecycle boundary for an arbitrary set of resources. CRUD on the resources as a group regardless of where they reside (e.g. different subs/rgs) is part of it - RBAC/deny is another. But the grouping is the key.
I guess the question is, does that same grouping make sense in Kubernetes?
I'm not sure that it does because:
- The namespace already exists
- If you want grouping within a namespace, you would just do it with your own labels and/or YAML groupings locally, i.e.
group1.yamlandgroup2.yaml(== 2 Helm charts or 2 Kustomize roots), then you can manage those resources independently even though they go into the same namespace.
I don't know that "Azure keeps track of which resources are grouped" is all that compelling in a k8s native world. Something like hierarchical namespaces feels more native?
One of the challenges with Deployment Stacks is that they're pretty coupled to BICEP/ARM templates, which ASO doesn't use, and they don't (at least at first glance) compose well with the CRD model in k8s. I suppose that you could annotate a resource as belonging to a deployment stack and somehow do management of the resource through the stack, but surface the state on the individual CRs... but it would be difficult to coordinate.
If deployment stacks requires someone to create and maintain a BICEP/ARM template, then I'd have to lean towards this doesn't make the most sense. My thinking is there might be a way to represent a "stack" of resources via helm/K8s YAML provisioned by ASO and then put them into an Azure "deployment stack" to gain the top three why bullets as advertised.
Based on the API, do you think this would be reasonably possible?
e.g. I see the az stack group create --template-file parameter, but this parameter is not required. However, assuming you can provision resources in ASO and then assign them to a deployment stack the question then would be can you do anything useful/meaningful afterwards without a BICEP file?
I do think that there is a bit of an impedance mismatch between how ARM is doing things here versus how Kubernetes "wants" things/how ASO does them. We think it makes sense to keep this open as there may be a way to reconcile the two such as using an annotation on the existing CRs to "group" them into a stack somehow - but definitely the templating part of stacks, and the part that allows resources embedded directly in it, don't really make sense in the context of Kubernetes.
I would like this from a solution/platform perspective and being able to restrict an AKS cluster to really 'living' in one resource group.
If we have an AKS cluster, as much as possible for running it should be in that RG, so say the 'AKS' parts, get deployed as a stack, we use flux in AKS to configure it to point to a git repo that then would install azure-service-operator.
Ideally ASO would create its resources in the same RG too, so it's all contained, deployment stacks here would allow it to be grouped together and might also allow for say a cleaner 'uninstallation' if you decided to remove ASO. You can then clean-up the deployment stack.
Looking through most of the examples, ASO's philosophy is really creating a new resource group for new resources, but that does mean we need resource groups (and in some environments that may not be as easy to provision).
Looking through most of the examples, ASO's philosophy is really creating a new resource group for new resources, but that does mean we need resource groups (and in some environments that may not be as easy to provision).
We try not to be too opinionated here. You're correct that's what most of the example show, but that's just because it's easiest to show resource creation in a separate test resource group. You can create them in existing resource groups as well and we definitely have users doing that.
With that said, 100% hear your point about deployment stacks and the cleanup options they enable.
I wonder if we want an option at the namespace scope or ASO scope have a certain tag might accomplish the deletion scenarios. We don't currently support bulk or default tagging like that, but we probably could.