Feature: How to orchestrate values from other resources for things such as a FlexibleServersFirewallRule or FederatedIdentity

[buzzaII]

It would be great to give a end to end example of an orchestration with no manual steps required to deploy a infrastructure where some pieces are not know until 'manifest deployment' time - this is pretty close but still requires the manual input of the issuer OIDC url.

Some examples are if we need to add a firewall rule for an aks cluster to access a postgres sql, or provide the OIDC url for a managed identity ?

apiVersion: dbforpostgresql.azure.com/v1api20220120preview
kind: FlexibleServersFirewallRule
metadata:
  name: samplefirewallrule
  namespace: default
spec:
  owner:
    name: samplepostgresql
  startIpAddress: 1.2.3.4 <-- need to add the AKS clusters ip here 
  endIpAddress: 1.2.3.4

# from sample here: https://github.com/Azure-Samples/azure-service-operator-samples/blob/master/cosmos-todo-list-mi/cosmos-sql-demo.yaml
apiVersion: managedidentity.azure.com/v1beta20220131preview
kind: FederatedIdentityCredential
metadata:
  name: pod-identity
spec:
  audiences:
  # For Workload Identity, Audiences should always be "api://AzureADTokenExchange"
    - api://AzureADTokenExchange
  # For Workload Identity, Issuer should be the OIDC endpoint of the cluster. For AKS this will look like
  # https://oidc.prod-aks.azure.com/00000000-0000-0000-0000-00000000000/


  issuer: ${AKS_OIDC_ISSUER} <-  how would this be supplied ?


  # For Workload Identity, Subject should always be system:serviceaccount:<namespace>:<serviceaccount>
  subject: system:serviceaccount:cosmos-todo:cosmos-service-account

[theunrepentantgeek]

Hi @buzzaII, we're working to understand your actual requests.

For securely enabling access to a PostgreSQL server from an AKS cluster, the usual Azure solution would be to create a VirtualNetwork (aka VNet) with both the database server and your cluster as members.

Given that a VNet provides better isolation and control, why would you need to use IP addresses? Or was this just a motivational example? Either way, this sounds like a good scenario for a more detailed case study showing how things might work.

For FederatedIdentity, we can easily make the issuer and subject configurable, generating some additional support. You can see how this works on the ServersAdministrator object - look for the parallel properties sid and sidFromConfig.

Let us know if you'd like us to do this.

Or, are you looking for a general ability to define any resource property using some kind of expression language? We're aware of the way you can do this with ARM templates - and we're also well aware (through experience) of the configuration burden and maintenance issues that this has often created. We've previously been reluctant to go down that path because it adds an enormous level of complexity with relatively modest benefits. We can reconsider this position if it's a feature that you'd find highly valuable.

(To illustrate the complexity problem, consider the presence of an expression that can't be evaluated. Should this result in omission of that one property when we create the resource, or do we skip the resource until evaluation is possible? In earlier discussions we concluded that we'd likely need to decide this individually for every property, and that there may even be cases where users would want to decide this for themselves.)

Let us know what you're thinking of, and we can work to turn this into some concrete items on our backlog.

[buzzaII]

Thanks for the reply @theunrepentantgeek, it was more motivational as I would like to understand how we 'stich' together at operator execution time vs an upstream process collating these pieces of information.

Take the below example - the issuer property is actually the output parameter of the cluster issuerUrl , even the ability to export any arbitrary resource to a configmap (or secret) and then reference that in any required manifest would be ideal - but currently it looks like only select parameters are exported to configmaps or can be used from configmaps as described here.

# from sample here: https://github.com/Azure-Samples/azure-service-operator-samples/blob/master/cosmos-todo-list-mi/cosmos-sql-demo.yaml
apiVersion: managedidentity.azure.com/v1beta20220131preview
kind: FederatedIdentityCredential
metadata:
  name: pod-identity
spec:
  audiences:
  # For Workload Identity, Audiences should always be "api://AzureADTokenExchange"
    - api://AzureADTokenExchange
  # For Workload Identity, Issuer should be the OIDC endpoint of the cluster. For AKS this will look like
  # https://oidc.prod-aks.azure.com/00000000-0000-0000-0000-00000000000/

  issuer: ${AKS_OIDC_ISSUER} <-  how would this be supplied ?

  # For Workload Identity, Subject should always be system:serviceaccount:<namespace>:<serviceaccount>
  subject: system:serviceaccount:cosmos-todo:cosmos-service-account

[theunrepentantgeek]

Thanks @buzzaII, that's helpful.

currently it looks like only select parameters are exported to configmaps or can be used from configmaps

Correct, though it's relatively straightforward for us to add more to the list.

the ability to export any arbitrary resource to a configmap (or secret) and then reference that in any required manifest would be ideal

This is where things start to get complicated - I know it sounds simple on the surface, but it's not.

I'll bring this up for discussion at our next weekly sync meeting.

[buzzaII]

Just a shout out - this looks good team.

It also looks like some other k8s operators for infrastructure are looking at the same problem:

crossplane/crossplane#3393 (comment)

[comptonad]

@theunrepentantgeek I see some related work has been done on this but wanted to throw out a suggestion because I haven't seen it mentioned.

Resources like RoleAssignments have an ArbitraryOwnerReference and ManagedClusters leverage ResourceReference which can reference any resource via GVK. Could something like this be added to the FederatedIdentity resource as a reference to the source of the issuerUrl?

[matthchr]

I think that this ask is quite related to #2555, although that may not be generic enough for everything discussed here.

[matthchr]

#2555 has merged which makes this possible for situations where the input supports a FromConfig variant, but not all do.

There's some other stuff in the OSS community that may make these sorts of "DAG on CRD" easier going forward so we're holding off doing more in ASO specifically to see what the community effort looks like.

[matthchr]

There are other projects in the community that are also moving towards doing this in a more generic way. See https://kro.run/ (alpha) for example

[matthchr]

We've added FromConfig variants for issuer and subject on the FederatedIdentityCredential (see our docs). We support arbitrary exports via CEL expressions. We are still interested in feedback on specific fields which we should enable FromConfig variants for, so keep that coming.

For general answers, we defer to projects designed to support data flows like that (such as kro above, or Kustomize/Helm as well).

Closing this as I think there's not more we're planning to do here for the general case. New issues can be opened for specific fields folks feel were missed.