[Feature Request]: Ability to consume CARML modules directly

[fblix]

Description

Is there currently a way to consume CARML modules directly without having to clone the repo and to publish each module again?

[JPEasier]

Would be nice for one of my customer projects to consume CARML modules from a public registry.

[AlexanderSehr]

Hey @fblix,

great question and something which comes up every now and then. Originally the plan was to publish to the Public Bicep Registry on-scale, but until there is no way to automate this for us, it is on hold.

To operate our own Registry (with a public endpoint), we'd need to satisfy certain conditions:

• new, empty registry deployed (lock protected)
• clarify funding for the registry (the current registry resulted in about 40$ of consumption since its creation - in storage costs)
• establish a strict versioning process in CARML (i.e., maintain the version.json files on our end too)
• maintain a table of module versions that point to specific commits (to allow backtracking to an oder module code & readme) and expose the URL to a given version (for reference in a solution template)

  For example

  • 1 table with all latest module versions:

    Module	Source	Url
    Key Vault	1.5	carml.azurecr.io/bicep/microsoft.keyvault.vaults:1.5
    Storage Account	1.7	carml.azurecr.io/bicep/microsoft.storage.storageaccounts:1.5
    ...	...	...

  • 1 table per resource type exposing all published versions

    Module	Source	Url
    Key Vault	1.5	carml.azurecr.io/bicep/microsoft.keyvault.vaults:1.5
    Key Vault	1.4	carml.azurecr.io/bicep/microsoft.keyvault.vaults:1.4
    Key Vault	1.3	carml.azurecr.io/bicep/microsoft.keyvault.vaults:1.3
    Key Vault	1.2	carml.azurecr.io/bicep/microsoft.keyvault.vaults:1.2
    Key Vault	1.1.1	carml.azurecr.io/bicep/microsoft.keyvault.vaults:1.1.1
    Key Vault	1.1.0	carml.azurecr.io/bicep/microsoft.keyvault.vaults:1.1.0
    Key Vault	1.0	carml.azurecr.io/bicep/microsoft.keyvault.vaults:1.0

[rahalan]

Team need to create a document to describe what is needed to create a CARML registry.

[segraef]

Further points to be elaborated in addition

• Find out current status of Bicep in ACR, are there any new preview features that can be used?
• Analyze publishing process from Bicep Registry Modules
• Hard requirements on versioning process
• Version map table in order to reference PR/commit for troubleshooting
• Private vs Public Bicep Registry
  • Outline advantages and disadvantages for organisations in regards to Inner Source and Open Source as well as Security and IP protection (Share Bicep modules by using private registries)

Not too much relevant for this topic but might be considered in future

• Can HC TF Registry patterns taken into consideration where each module gets decoupled into individual repositories?
  • Would it increase maintenance and complexity?
  • Would it simplify dependencies rather than having all Modules in a single repository like it is currently?
  • Separating Modules from the main repository into would increase maintenance around future publishing processes?
  • Moving documentation to a public webspace like GitHub Pages https://carml.github.io/
    • General Documentation can be decoupled from main repository
    • Public webspaces like GitHub Pages can be used to establish a proper version release documentation available for different versions see https://registry.terraform.io/modules/Azure/network/azurerm/3.5.0

Let's assume each Module is stored and maintained in its own repository leveraging publishing CI, where will this CI be stored, within each single repository or centrally? If centrally stored, would it make sense to leverage GitHub Actions/ADO tasks to publish Bicep modules into an ACR instead of using utilities and to decrease dependencies and complexities?

Outlining the approach for the POC to elaborate pitfalls/issues/efforts to move to the public registry

1. Create registry and make it publicly accessible

• Create new and empty registry (lock protection optional, only relevant internally)
• Test new preview features like anonymous public access
• Test manual publishing and consumption of Bicep modules (internally and externally)
• What options do we have if there's currently no way to make it publicly accessible?
• What are the waiting times

2. Establish strict versioning process

• versioning scheme: semantic (1.0.2), integer (v1,v2), dates (2022-09-13)
• Find out options how versioning can be achieved, dependent or independent to CARML
• If versioning dependent on CARML
  • version.json as version reference
  • commit as version reference
  • repo release as version reference
• If versioning independent of CARML version map table is required
• Establish a strict versioning process
  • Example process: CARML module changed > (version.json changed) > PR > commit as reference in repo module and
  • Module change and version bumping see Updating an existing module as reference

3. Establish publishing process

• Evaluate publishing out of CARML > ACR
• Evaluate publishing forked CARML repo > ACR
• How do contributors publish Modules? More details see https://github.com/Azure/bicep-registry-modules/blob/main/CONTRIBUTING.md
• Bicep Registry Modules GitHub Workflow Publish Module might be leveraged.

4. Consume and call Bicep Modules from Public Registry

• Test it!

5. Cost Estimation

• Costs for a growing registry over years with average monthly growth of 10MB

References

Terraform Registry GitHub Pages Publish and call Bicep modules Create Private Bicep Modules Registry Anonymous Pull Access Bicep Registry Modules Public Registry Details MCR Bicep Share Bicep modules by using private registries

[segraef]

Added #2064 for testing.

[segraef]

@fblix, this can be achieved via https://docs.microsoft.com/en-us/azure/container-registry/anonymous-pull-access, we're currently elaborating options on how to implement it, please stand by. Test was done via #2064 which is basically just this az acr update --name myregistry --anonymous-pull-enabled. But we still need to elaborate the incorporation of versioning as well as the upcoming documentation versioning #2025 .

[segraef]

I think we can close this one.