Proposal - simplifying resource referencing (part 2)

[anthony-c-martin]

Proposal - simplifying resource referencing (part 2)

Part 1 / Part 2

Problem statement

Passing around / obtaining references to resources in a type-safe manner is overly complex. Rather than inventing non-type-safe mechanisms to refer to resources or resource properties, we should provide a first-class syntax for doing so, with full type-safety and editor support.

Resources as params and outputs

A new type of resource will be accepted in param and output declarations to permit passing a reference to a resource as an input or output for a module. Supplying the type string for the resource would be optional, but functionality would be greatly reduced without it.

At compile-time, Bicep will type check for reference equality - it will ensure a valid resource reference is passed to a generic resource param, and it will ensure that a valid resource reference matching the expected type string is passed to a typed resource param.

Examples

Generic

// we haven't specified a resource type here
param lockableResource resource

resource lockResource 'Microsoft.Authorization/locks@2016-09-01' = {
  scope: lockableResource
  name: 'DontDelete'
  ...
}

Input/Output

// input a resource reference
param storageAcc resource 'Microsoft.Storage/storageAccounts@2021-01-01'

var myContainer = storageAcc.child('blobServices', 'default').child('containers', 'myContainer')

// output a resource reference - note the resource type can be omitted
output myContainer resource = myContainer

Property access

param storageAcc resource 'Microsoft.Storage/storageAccounts@2021-01-01'

// list keys
var myKey = listKeys(storageAcc.id, storageAcc.apiVersion).keys[0].value

// property access
output accountTags object = storageAcc.tags

Notes

1. If the param does not specify a resource type string, functionality will be greatly reduced - limited to using the resource as a scope for an extension resource, and accessing the resource id property. We want to encourage module authors to be specific about the type they accept to provide optimal type safety.
2. API versions do not need to match across module params and outputs, but types must match if the param has specified a type.
3. We will need to be careful when passing param references to resources at a different scope to the module, as they cannot be used for certain purposes (deploying children/extensions, for example).

Out of scope

1. This proposal requires both inputs and outputs to accept a resource reference, and there is no conversion between resourceId string and resource reference. The following would not be permitted:

   module myMod './module.bicep' = {
     name: 'myMod'
     params: {
       // resourceReference is a param of type 'resource'
       // resourceId is a string containing a resourceId
       resourceReference: resourceId
     }
   } 
   

Codegen

The most straightforward option for JSON codegen is to generate a string parameter or output in the template JSON, with some associated metadata.

[alex-frankel]

Will we accept a literal resource ID string for foo.params.res here?

foo.bicep

param res resource

main.bicep

module foo './foo.bicep' = {
  name: ...
  params: {
    res: '/subscriptions/.../resourceGroups/.../microsoft.rp/type/...'
  }
}

[majastrz]

My expectation would be for the string to be rejected because it's not a symbolic name. However, I think we do need some sort of interop gesture to bridge templates passing resource ID strings around with this way of passing parameters.

[majastrz]

@anthony-c-martin During our last discussion, we realized that API version match/mismatch semantics differ on inputs and outputs. I think it'd be worthwhile to explain more about for the community at large to offer feedback (if any).

In the generic resource case (note number 1), would we allow name and type property access as well? Just wondering... we can start small and add more later, of course.

Regarding code gen, we should consider moving some of the type safety down to the runtime so all tempaltes can leverage it. We could still compile down to a fully qualified resource ID but we could introduce a new parameter type in the JSON with some additional settings.

[alex-frankel]

My expectation would be for the string to be rejected because it's not a symbolic name.

Right, but then this would have to work I guess? So if you need to pass in a resource ID from an external source, you would expose that as a param of type resource.

params.json

{
  "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentParameters.json#",
  "contentVersion": "1.0.0.0",
  "parameters": {
    "foo": {
      "value": "/subscriptions/.../resourceGroups/.../microsoft.rp/type/..."
    }
  }
}

main.bicep

param foo resource

module foo './foo.bicep' = {
  name: ...
  params: {
    res: foo
  }
}

az deployment group create -f ./main.bicep -p params.json

[miqm]

My expectation would be for the string to be rejected because it's not a symbolic name.

Right, but then this would have to work I guess? So if you need to pass in a resource ID from an external source, you would expose that as a param of type resource.

params.json

{
  "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentParameters.json#",
  "contentVersion": "1.0.0.0",
  "parameters": {
    "foo": {
      "value": "/subscriptions/.../resourceGroups/.../microsoft.rp/type/..."
    }
  }
}

main.bicep

param foo resource

module foo './foo.bicep' = {
  name: ...
  params: {
    res: foo
  }
}

az deployment group create -f ./main.bicep -p params.json

in some other issue I've suggested using as keyword that could be used to cast string that we expect to be a resourceid to a biceps resource. perhaps worth revisiting that?

[majastrz]

@alex-frankel Yeah passing it through parameters would have to work. We do have to answer the question whether we need the ability to turn a string ID into a symbolic name within a Bicep file without the use of parameters. One scenario I can think of comes up with referencing a JSON file as a module (when we have the feature) when the template outputs a string resource ID.

@miqm Yeah as could be an option although we have so far resisted adding type casting so far. All the type conversions currently are done through converter functions like any() or string(). Another option would be a separate function or a resource() overload that accepts a string ID

[anthony-c-martin]

My expectation would be for the string to be rejected because it's not a symbolic name.

Right, but then this would have to work I guess? So if you need to pass in a resource ID from an external source, you would expose that as a param of type resource.

This is definitely a scenario we'll need to handle, and I think we have the options of either (with some rough examples):

1. Exposing as a string parameter in the JSON - simple, but potentially error-prone if people format the id incorrectly.

   "parameters": {
     "vmResource": {
       "type": "string"
     }
   }
   

2. Creating a new parameter type in JSON with enhanced validation (on server side, but possibly also client-side with psh/cli)

   "parameters": {
     "vmResource": {
       "type": "resource"
     }
   }
   

3. (sort of an in-between) Exposing as a string parameter in the JSON, with metadata that newer CLI utilities are able to understand (so as to not make it a breaking change, and allow better validation to be added gracefully).

   "parameters": {
     "vmResource": {
       "type": "string",
       "metadata": {
         "_typeinfo": {
           "resourceType": "Microsoft.Compute/virtualMachines"
         }
       }
     }
   }
   

[anthony-c-martin]

@alex-frankel Yeah passing it through parameters would have to work. We do have to answer the question whether we need the ability to turn a string ID into a symbolic name within a Bicep file without the use of parameters. One scenario I can think of comes up with referencing a JSON file as a module (when we have the feature) when the template outputs a string resource ID.

It makes sense to think about this, but for the purposes of this specific proposal, I'd like to treat turning a resourceId in a string into a symbolic reference out-of-scope. Would you be OK with me creating another issue specifically for that, and adding a note to this proposal to mention as such?

[majastrz]

Should be ok to include it in part 3.

[miqm]

I like Passing type in metadata rather than introducing new type. In addition we could do object or array typing in similar way.

[anthony-c-martin]

Will we accept a literal resource ID string for foo.params.res here?

@alex-frankel, @majastrz - FYI, I added a note to explicitly mention this scenario is not covered by this proposal.

[jamesongithub]

Will this proposal cover passing a resource directly as a parameter of another resource or just for module input/outputs

something like

resource disk 'disks' {
   ...
}

resource vm 'vms' {
  osDisk: disk
}

[anthony-c-martin]

Will this proposal cover passing a resource directly as a parameter of another resource or just for module input/outputs

This proposal is just covering module inputs/outputs.

[stan-sz]

As in #2163 the output of a module can contain secrets (e.g. output of a list*) function to store it's value in a KV. Is it planed to introduce secureString or secureObject (or other way to hide secrets) from module output?

[rynowak]

@anthony-c-martin - any ideas about how this would work for a resource with a discriminator field? (likely kind) property?

[SeidChr]

Just another suggestion: I can access my module outputs by <modulename>.outputs.<outputname>

I would love to access my module resources (fully type-safe of cause) by <modulename>.resources.<outputname>

The Idea brings up a question: Should all top level module elements be accessible in this way? Wouldn't it be nice to access <modulename>.modules.<modulename>.resources.<resourcename>.properties.<propertyname>?

[ptemmer]

Apologies as this is not the most appropriate place to ask, however this proposal would be a possible solution to my question, nonetheless as this feature hasn't been released yet, I'm wondering what the current most appropriate way is to achieve the following:

Say: main.bicep:

module rAppservice 'appService.bicep' = {
  name: appServiceName
  params: {
    appServiceName: appServiceName
}

resource vnetintegration 'Microsoft.Web/sites/networkConfig@2021-02-01' = {
  name: 'vnetintegration'
  parent: XXXXXXX
}

I cannot specify the module symbolic name as a parent as that is not a resource. Currently what I am doing is the following, however this feels wrong.

module rAppservice 'appService.bicep' = {
  name: appServiceName
  params: {
    appServiceName: appServiceName
  }
}

resource existingAppService 'Microsoft.Web/sites@2021-02-01' existing = {
  name: appServiceName
}

resource vnetintegration 'Microsoft.Web/sites/networkConfig@2021-02-01' = {
  name: 'vnetintegration'
  parent: existingAppService 
}

[anthony-c-martin]

@ptemmer your workaround looks like the best option that's currently available. The only thing to be careful with is to order the dependencies so that the networkConfig resource gets deployed after the module:

module rAppservice 'appService.bicep' = {
  name: appServiceName
  params: {
    appServiceName: appServiceName
  }
}

resource existingAppService 'Microsoft.Web/sites@2021-02-01' existing = {
  name: appServiceName
}

resource vnetintegration 'Microsoft.Web/sites/networkConfig@2021-02-01' = {
  name: 'vnetintegration'
  parent: existingAppService
  dependsOn: [
    rAppservice
  ]
}

[ptemmer]

Thanks @anthony-c-martin for the super quick reply. Good to know that workaround is ok, as I honestly had the feeling I wasn't using Bicep as it should be. I guess the current proposal will solve this altogether.

Off-topic: I'm starting out with Bicep so I due regularly run into these kind of issues/doubts. What is the best place to ask questions? This repo? Or is there a Slack bicep community? (wasn't able to find one).

Thanks again.

[anthony-c-martin]

@ptemmer - this repo is the best place to go. https://github.com/Azure/bicep/discussions is a good place to ask questions such as "What's the best way to do XYZ?" or for help with specific scenarios.

[thealanagrace]

Is there an ETA on this functionality? It would really help me make my deployment scripts so much sleeker!

[alex-frankel]

We are trying to get an initial implementation done in the next few months.

[rouke-broersma]

@alex-frankel

Parameter resources cannot be used with the .parent or .scope properties because it would allow you to bypass scope validation easily. The set of cases that we could actually provide validation for these use cases are really limited.

This is one of the use cases we would very much want to use this for, especially scope because of Authorization on resources. Is there a different issue for this use case or is this never going to be solved? I personally wouldn't consider this fixed until we can at least use resource params for scope.

[rynowak]

The first step of this (all the disruptive changes) are in but hidden behind an experimental flag. I plan to continue making progress on all of the scenarios described here and will keep this issue open until we're totally unblocked.

[johndowns]

@rynowak Awesome - is it possible to give some instructions on how to enable the experimental flag if we want to try some of this out now?

[rynowak]

Set the environment variable BICEP_RESOURCE_TYPED_PARAMS_AND_OUTPUTS_EXPERIMENTAL=true

I'd recommend also reading the PR description before trying this out https://github.com/Azure/bicep/pull/4971

Some of the scenarios described in this issue are explicitly blocked by the compiler at this stage. We've also run into limitations enforced by the deployment engine using resources as module outputs. That will require changes in Azure to unblock.

[bgawale]

Very much looking forward for this feature to be available, deployments would be much more simpler.

[xavierjohn]

If I had a module named west2 and it had two resources name myStorage and myEventHubNamespace, would using a . format be more natural? As a programmer in several languages, the natural format for me would be to use west2.mtStorage and west2.myEventHubNamespace without having to deal with module output.

[ds-evo]

Hello,

i just tried to set the resource param to null.

But this didn't work.

To reduce duplicate code i tried this:

param app_res resource 'Microsoft.Web/sites@2021-03-01' = any(null)
param app_slot_res resource 'Microsoft.Web/sites/slots@2021-03-01' = any(null)

param appIsSlot bool = app_res == null

var _res = (!appIsSlot) ? app_res : app_slot_res

@secure()
param cfg_conn object

@secure()
param cfg_app object

resource appConnStrs 'Microsoft.Web/sites/config@2021-01-15' = {
    parent: _res
    name: 'connectionstrings'
    properties: cfg_conn
}

resource appSettings 'Microsoft.Web/sites/config@2021-01-15' = {
    parent: _res
    name: 'appsettings'
    properties: cfg_app
}

But then i get this error:

Error BCP036: The property "parent" expected a value of type "Microsoft.Web/sites" but the provided value is of type "Microsoft.Web/sites/slots@2021-03-01 | Microsoft.Web/sites@2021-03-01"

There are so many cases, where i think i need to use a template language to auto-generate bicep files, only to get a generic solution.

And, i start to think, that the Rest-Api with a good ORM is more flexible and easier then bicep or ARM

[ds-evo]

UPDATE:

The above code has logic error - so rewrite it to this.

param app_res resource 'Microsoft.Web/sites@2021-03-01' = any(null)
param app_slot_res resource 'Microsoft.Web/sites/slots@2021-03-01' = any(null)

param appIsSlot bool = app_res != null

@secure()
param cfg_conn object

@secure()
param cfg_app object

resource appSlotConnStrs 'Microsoft.Web/sites/slots/config@2021-01-15' = if (appIsSlot) {
    parent: app_slot_res
    name: 'connectionstrings'
    properties: cfg_conn
}

resource appSlotSettings 'Microsoft.Web/sites/slots/config@2021-01-15' = if (appIsSlot) {
    parent: app_slot_res
    name: 'appsettings'
    properties: cfg_app
}


resource appConnStrs 'Microsoft.Web/sites/config@2021-01-15' = if (!appIsSlot) {
    parent: app_res
    name: 'connectionstrings'
    properties: cfg_conn
}

resource appSettings 'Microsoft.Web/sites/config@2021-01-15' = if (!appIsSlot) {
    parent: app_res
    name: 'appsettings'
    properties: cfg_app
}

Now i get: Error BCP229: The parameter "app_slot_res" cannot be used as a resource scope or parent. Resources passed as parameters cannot be used as a scope or parent of a resource.

This is my Solution for this problem. I ended with two modules, with same parameters and nearly same code....

module appCfgReal '../../../app/app_cfg_sets.real.bicep' = if (!dryRun && !appIsSlot) {
  name: '${appName}-appCfg-real-mod'
  params: {
    app: app
    cfg_conn: cfg_conn
    cfg_app: cfg_app    
  }
}

module appCfgSlotReal '../../../app/app_cfg_sets.real.slots.bicep' = if (!dryRun && appIsSlot) {
  name: '${appName}-appCfg-real-slot-mod'
  params: {
    app: app
    cfg_conn: cfg_conn
    cfg_app: cfg_app    
  }
}

PS: And it would be nice, if string interpolation in resource types would work.