"existing" functionality for modules

[johnnyreilly]

Is your feature request related to a problem? Please describe.

We have a Bicep module registry which we use to wrap Bicep templates, such that they are delivered to match our organisations collectively agreed security posture. So to create a storage account we'd do something like this:

module deadLetterStorageAccount 'br:icebox.azurecr.io/bicep/ice/providers/storage/storage-accounts:v1.0' = {
  name: 'lens-dead-letter-storage-account-${branchHash}'
  params: {
    tags: tags
    location: location
    storageAccountName: opticalDeadLetterStorageAccountName
    skuName: 'Standard_ZRS'
  }
}

If we want to reference the resource created by that module later elsewhere, we can do that using existing like so:

resource opticalDeadLetterStorageAccount 'Microsoft.Storage/storageAccounts@2021-02-01' existing = {
  name: opticalDeadLetterStorageAccountName
  scope: resourceGroup(opticalStorageResourceGroupName)
}

But this is not a great developer experience as:

1. We have flipped from module syntax to resource syntax which is confusing
2. We have to go and look up what resource is used in the module to be able to write the module statement
3. If that module is updated to use a different resource, we have to find out the new one and then copy/pasta that across all our resource ... existing references

Describe the solution you'd like

What would be delightful is if there was some kind of existing syntax for modules. Imagine if, instead of:

resource opticalDeadLetterStorageAccount 'Microsoft.Storage/storageAccounts@2021-02-01' existing = {
  name: opticalDeadLetterStorageAccountName
  scope: resourceGroup(opticalStorageResourceGroupName)
}

we could do:

module deadLetterStorageAccount 'br:icebox.azurecr.io/bicep/ice/providers/storage/storage-accounts:v1.0' existing = {
  name: 'lens-dead-letter-storage-account-${branchHash}'
  params: {
    tags: tags
    location: location
    storageAccountName: opticalDeadLetterStorageAccountName
    skuName: 'Standard_ZRS'
  }
}

This would make upgrades much more straightforward / intuitive.

There's obviously a whole bunch of complexity in what I'm suggesting - but something like this would be amazing. Something that allows you to acquire the resources created by a module directly rather than indirectly. What do you think?

[jeskew]

Are you looking to have the resources within the module converted into existing resources (i.e., does it not matter if the resources were deployed via the module) or is it the module itself that you would want to refer back to?

For the latter, you can sort of do this today, since Bicep deploys modules as nested templates. This does require switching to resource syntax, so instead of

module deadLetterStorageAccount 'br:icebox.azurecr.io/bicep/ice/providers/storage/storage-accounts:v1.0' existing = {
  name: 'lens-dead-letter-storage-account-${branchHash}'
  params: {
    tags: tags
    location: location
    storageAccountName: opticalDeadLetterStorageAccountName
    skuName: 'Standard_ZRS'
  }
}

you could use

resource deadLetterStorageAccount 'Microsoft.Resources/deployments@2021-04-01' existing = {
  name: 'lens-dead-letter-storage-account-${branchHash}'
}

Outputs would be nested under properties, e.g., deadLetterStorageAccount.properties.outputs..., so it's not a drop-in replacement.

[johnnyreilly]

Are you looking to have the resources within the module converted into existing resources

Kind of this I think. It would be tremendous if we could have a way to reference the resources created by a module without explicitly having to know what exact resources they were.

In my head I think of the current situation as analogous to having a class which performs operations on your behalf which are abstracted away from you, but the class is presently unable to offer an API that allows reacquisition of same.

That's probably slightly confusing. But yeah, the thing you're suggesting as the other option is what we're already doing and it feels unintuitive.

Also, we run a Bicep registry in our organisation, so there's necessarily extra steps required to discover what the module is actually provisioning

[anthony-c-martin]

This is an interesting suggestion! Just to clarify - the module you're trying to establish a reference to is part of the same overall deployment, right? (e.g. you're not trying to access a module deployed by a totally independent deployment)

If we extended #2246 to be able to pass module references as params/outputs, would that solve the problem for you?

[johnnyreilly]

Just to clarify - the module you're trying to establish a reference to is part of the same overall deployment, right?

Yes I believe so. To be totally clear, in our present case we have:

main.bicep
storage.bicep
compute.bicep

main.bicep references storage.bicep and compute.bicep. storage.bicep creates a resource that we reference in compute.bicep as well. We achieve that presently using the resource ... existing syntax.

If we extended #2246 to be able to pass module references as params/outputs, would that solve the problem for you?

I think so - if that means you are able to pass around the reference to the resource then yes.

Having the ability to reference something from an independent deployment would also be amazing. But just module references would be a massive win. As a rough estimate, module references would cover 60% of my resource ... existing use cases. The other 40% being independent deployments.

[jeskew]

Referencing modules from independent deployments would be trickier, as ARM would look up the deployment by name and the Bicep compiler would look up type information from the referenced file path. This introduces the possibility that Bicep will block expressions that would resolve correctly during a deployment (and allow statements that fail at deploy time) due to drift between what outputs existed on the template when it was last deployed vs. what outputs exist in the template at compile time. There's also the possibility that the same name was used by a completely different deployment, in which case Bicep's understanding of what outputs would be available could be totally wrong.

[miqm]

+1 on this. Adding my use case:

Managing big infrastructure can be a challenge with bicep. There are parts, that you don't touch for months, and there are things that you change often. I'm testing a solution, where I'm treating a deployment as a resource. In Pipeline I've split IaC into different parts: core, network, firewall, etc. Each part is being deployed only if there were a change made to the code or there was a change in dependent part.

To pass parameters (i.e. log analytics id, deployed in "core" module) between parts I use an existing resource of Microsoft.Resources/Deployments. The downside is that I do not have any type checking for the "outputs" property.

Describe the solution you'd like I would like to be able to define a module with existing keyword. This syntax would be changed to reference to given Microsoft.Resources/Deployments and from referenced bicep file would extract output types for better user experience.

Additionally, the syntax would add .value to the output property automatically, so the expierience would be as close to a regular module as possible.

Also, it would be good to cover referencing existing deployment stacks.

[anthony-c-martin]

My 2 cents: I could see a feature like this existing for Stacks, because we have stronger lifecycle & stateful semantics associated with a Stack. For Deployments, I would be worried about the number of footguns associated with this - it would be easy for history to be overwritten or removed, and break things.

[miqm]

@johnnyreilly - a question to you and to all who upvoted the issue - would you be ok with having just a typed output? i.e.

//--- depModule.bicep

output someResource {
  id: string
  interestingProperty: string
} = {
  id: someResource.id
  interestingProperty: someResource.properties.interesting
}

//--- main.bicep
module dep 'depModule.bicep' existing stack = {
  name: 'stack_dep"
}

var xId = dep.outputs.someResource.id
var xProp = dep.outputs.someResource.interestingProperty

In main.bicep you would have completions based on the type defined for output in depModule.bicep and during validation phase if the stack_dep would not have those properties, you would receive an error?

[johnnyreilly]

To be honest, this wouldn't be that useful. It's not much more than syntactic sugar over having individual exports (which already exist). This solution would not resolve 2 and 3 from the original post:

2. We have to go and look up what resource is used in the module to be able to write the module statement
3. If that module is updated to use a different resource, we have to find out the new one and then copy/pasta that across all our resource ... existing references

Being able to directly reference a resource that is exported from a module is the aim.

[miqm]

so if we have a strongly typed outputs from a remote modules (not part of the current deployment but deployments or stacks), that also support resource references (as described in #2246), this would cover all your cases?

[miqm]

As for the discussion that we had on validating during runtime if the deployment/stack was deployed using the template we are referencing - We could follow the mentioned by Anthony Duck typing - if all of the parameter that we expect exist and have expected type - this means that this is the module we want.

For the technical solution, we could use user-defined functions for this as Johnny proposed. As I understood, there's existing functionality that verifies if the value referenced from the resource matches the type defined for the function, right?

From the user perspective, we would have a nice syntax:

//existingStack.bicep
output abc string = 'abc'
output num123 int = 123
output obj { prop: string } = {
  prop: 'value'
}
output arr string[] = [  'a',  'b' , 'c' ]

//main.bicep
module existingStack './path/to/module/existingStack.bicep' existing deployment = {
 name: 'existingStack'
 scope: resourceGroup()
}

var usageAbc = existingStack.outputs.abc

but behind the scenes, during template compilation, we would generate a user-defined function, which in bicep language would look like this:

func outputs_existingStack(outputs object) {
  abc: string
  num123: int
  obj: { prop: string }
  arr: string[]
} => {
  abc: outputs.abc.value
  num123: outputs.num123.value
  obj: outputs.obj.value
  arr: outputs.arr.value
}

plus the regular existing resource definition of Microsoft.Resources/deployments.

The usage would be converted to function call of the generated UDF (code in bicep, but this would be all done during compilation)

var usage = outputs_existingStack(existingStack.properties.outputs).abc

This way, all changes for the existing module functionality would be on the compiler side and we would leverage existing type-checking functionality.

A downside of this solution would be that since we would generate some internally-named function, and if the deployment "API" mismatches - there would be an error telling about some function, not about module version mismatch.

Also, as I did testing, this type check happens when we call the function. If we would like to do this on the template validation phase - then changes on the backend would be necessary.

Some other idea that came to me is that we could introduce module versioning, similar to what is in the Helm Charts. Each chart - in our case the "top-level" module that is being the base of the deployment (or stack, I'm treating them equally here, as stacks are "enriched" deployments from the output perspective) would need to define a version, and only modules with versions defined could be used in existing references. However I can imagine that all modules have version = 1.0.0 so this check would not give any advantage. The list of outputs and corresponding types feels more accurate.

Of course, since the generated existing resource and outputs function would have some internally fixed name pattern, the ARM backend could do the comparison and validity check in some earlier phase - but I feel that this would be improvement rather than core functionality.

To sum up - for strongly typed existing module outputs, I think we are capable of implementing this using existing functionalities. Fist step would be implementing conversion of the module sybolicName 'modulePath' existing [deployment | stack] = syntax into a function and a existing resource definition and use the combination of them when existing module symbolic name is being referenced.

Second step would be to introduce changes in the ARM backend to validate the type match in earlier stage than first usage and emit nicer error code on mismatch.

Lastly, if we implement passing resource references between modules within same deployment, this functionality would enable passing those references between modules that are not part of the same deployment - as the mechanism of passing resources would most likely be this same as for other types. We just need to agree how the resource type would look like, but as far as I remember, there were some ideas around it.

[miqm]

Regarding the possible syntax, we need to take into account that we should cover not only deployments, but also stacks. I see 3 main options:

Option A: add existing to module with descriptor deployment stack: Option A1:

module modSymbol 'modPath.bicep' existing deployment = {
  name: deploymentName
  scope: deploymentScope
}

module modSymbol 'modPath.bicep' existing stack = {
  name: stackName
  scope: stackScope
}

Option A2:

module modSymbol 'modPath.bicep' existing = {
  name: deploymentOrStackName
  scope: deploymentOrStackScope
  kind: 'deployment' | 'stack'
}

Pros:

• does not introduce new language constructs, using existing existing pattern known from resources. A2 moves selector to the properties so it's cleaner

Cons:

• Enables access to deployments - which we might not want to bring
• Option A1 requires additional syntax parsing

Option B: New type "stack"

stack stackSymbol 'stackSourceCodePath.bicep' existing = {
  name: stackName
  scope: stackScope
}

Pros:

• does not introduce existing on deployments (modules)

Cons:

• introduces new keyword
• Since we add existing users may expect there's option to create stack from deployment - although this is technically possible, I'm not sure if we should widely expose this possibility and I'm not sure about the implications (stack owning a stack).

Option C: use function

var existingStackSameScope = stack('stackSourceCodePath.bicep', stackName)
var existingStackDifferentScope = stack('stackSourceCodePath.bicep', stackScope, stackName)
var existingStackSubscription = stack('stackSourceCodePath.bicep', subscription(), stackName)
var existingStackMg = stack('stackSourceCodePath.bicep', managementGroup('mgName'), stackName)

resource stackScope 'Microsoft.Resources/resourceGroups@2024-07-01' existing = {
  name: stackRgName
  scope subscription(stackSubscriptionId)
}

Pros:

• scoping to stack only
• function similar to resourceId or ARM's resource(), where the bicep source can be considered as stacks "type".

Cons:

• Azure's existing resource in bicep code described as var
• multiple overloads taking special functions (subscription, managementGroup) can be tricky to implement and require special casing.

Each of the option has pros and cons. Happy to hear some feedback or perhaps some other proposals.

[johnnyreilly]

so if we have a strongly typed outputs from a remote modules (not part of the current deployment but deployments or stacks), that also support resource references (as described in #2246), this would cover all your cases?

I think so yes

[majastrz]

I don't think we're prepared at this point to add a new keyword to the language to support stacks.

Since this issue is really more about getting accurate types for the outputs object for an existing nested deployment or an existing stack, I am wondering if there's another option that we can consider.

How about this?

// stack case
@narrowOutputsBasedOn('./stackSourceCodePath.bicep')
resource stack 'Microsoft.Resources/deploymentStacks@2024-03-01' existing = {
  name: ...
  scope: ...
}

// deployment/module case
@narrowOutputsBasedOn('./moduleSourceCodePath.bicep')
resource deployment 'Microsoft.Resources/deployments@2025-03-01' existing = {
  name: ...
  scope: ...
}

The @narrowOutputsBasedOn(<bicep path>) decorator would be only allowed on known resource types like deployments or stacks. Once specified on a resource declaration, it would "override" the loose object type of the outputs property and replace it with an equivalent narrowed type based on the specified source file.

Thoughts?

[miqm]

@majastrz with this decorator (whatever the name would be), would we provide access without adding value - so the experience referencing those resources would be similar to referencing modules, or we keep the 1:1 parity with underlying json?

[majastrz]

@miqm You're talking about the objects with the value property? Yeah, it would have to understand that part.

[riosengineer]

Just to add sentiments on top of my existing (duplicate (now closed) issue - oops!) that this is becoming a much bigger ask for us now with stacks in the picture in most deployments we do.

It's becoming a very common pattern to reference an existing stack for output values downstream. So for us this is really needed to make authoring a better experience in code.