Proposal: Structured component metadata info requirement

[berndverst]

Describe the proposal

This is a proposal to capture and require component metadata information to be provided alongside each component in a structured manner according to a defined schema.

Why

This has several advantages:

• Automatically generate component documentation to be in line with components-contrib code. No more manual creation of docs PRs for component metadata.
• We can generate a schema of all known components which can be used to provide better developer tooling, e.g. this could enable the validation of component.yaml files.

How

Alongside the code of each component a metadata.yaml should be included which describes all metadata options, whether they are required, what type of values they accept, etc A schema definition additional enables authoring of these metadata.yaml files with tools such as VS Code.

An example example schema for how component metadata can and should be described can be found here: https://github.com/ItalyPaleAle/dapr-components-contrib/blob/component-info-sample/.build-tools/pkg/metadataschema/schema.go

Here are some specific examples of what this would look like.

Example state store with multiple connection options:

schemaVersion: v1
type: state
name: azure.cosmosdb
version: v1
status: stable
title: "Azure Cosmos DB (SQL API)"
urls:
  - title: Reference
    url: https://docs.dapr.io/reference/components-reference/supported-state-stores/setup-azure-cosmosdb/
capabilities:
  # If actorStateStore is present, the metadata key actorStateStore can be used
  - actorStateStore
  - crud
  - transactional
  - etag
  - ttl
  - query
authenticationProfiles:
  - title: "Master key"
    description: "Authenticate using a pre-shared \"master key\"."
    metadata:
      - name: masterKey
        required: true
        sensitive: true
        description: "The key to authenticate to the Cosmos DB account."
        example: '"my-secret-key"'
  - title: "Azure AD: Managed identity"
    description: "Authenticate using Azure AD and a managed identity."
    metadata:
      - name: azureClientId
        required: false
        description: "Client ID (application ID). Required if the service has multiple identities assigned."
        example: '"c7dd251f-811f-4ba2-a905-acd4d3f8f08b"'
  - title: "Azure AD: Client credentials"
    description: "Authenticate using Azure AD with client credentials, also known as \"service principals\"."
    metadata:
      - name: azureTenantId
        required: true
        description: "ID of the Azure AD tenant"
        example: '"cd4b2887-304c-47e1-b4d5-65447fdd542b"'
      - name: azureClientId
        required: true
        description: "Client ID (application ID)"
        example: '"c7dd251f-811f-4ba2-a905-acd4d3f8f08b"'
      - name: azureClientSecret
        required: true
        sensitive: true
        description: "Client secret (application password)"
        example: '"Ecy3XG7zVZK3/vl/a2NSB+a1zXLa8RnMum/IgD0E"'
  - title: "Azure AD: Client certificate"
    description: "Authenticate using Azure AD with a client certificate (in PFX/PKCS#12 format). One of azureCertificate and azureCertificateFile is required."
    metadata:
      - name: azureTenantId
        required: true
        description: "ID of the Azure AD tenant."
        example: '"cd4b2887-304c-47e1-b4d5-65447fdd542b"'
      - name: azureClientId
        required: true
        description: "Client ID (application ID)."
        example: '"c7dd251f-811f-4ba2-a905-acd4d3f8f08b"'
      - name: azureCertificate
        required: false
        sensitive: true
        description: "Certificate and private key (in PFX/PKCS#12 format)."
        example: |
          "-----BEGIN PRIVATE KEY-----\n MIIEvgI... \n -----END PRIVATE KEY----- \n -----BEGIN CERTIFICATE----- \n MIICoTC... \n -----END CERTIFICATE-----"
      - name: azureCertificateFile
        required: false
        sensitive: true
        description: "Path to PFX/PKCS#12 file on disk, containing the certificate and private key."
        example: "/path/to/file.pem"
      - name: azureCertificatePassword
        required: false
        sensitive: true
        description: "Password for the certificate if encrypted."
        example: "password"
metadata:
  - name: url
    required: true
    description: "The Cosmos DB url."
    example: '"https://******.documents.azure.com:443/"'
  - name: database
    required: true
    description: "The name of the database."
    example: '"db"'
  - name: collection
    required: true
    description: "The name of the collection (container)."
    example: '"collection"'

Example binding

schemaVersion: v1
type: bindings
name: http
version: v1
status: stable
title: "HTTP"
urls:
  - title: Reference
    url: https://docs.dapr.io/reference/components-reference/supported-bindings/http/
binding:
  output: true
  input: false
  operations:
    - name: create
      description: "Alias for \"post\", for backwards-compatibility."
    - name: get
      description: "Read data/records."
    - name: head
      description: "Identical to get except that the server does not return a response body."
    - name: post
      description: "Typically used to create records or send commands."
    - name: put
      description: "Update data/records."
    - name: patch
      description: "Sometimes used to update a subset of fields of a record."
    - name: delete
      description: "Delete a data/record."
    - name: options
      description: "Requests for information about the communication options available (not commonly used)."
    - name: trace
      description: "Used to invoke a remote, application-layer loop-back of the request message (not commonly used)."
capabilities: []
metadata:
  - name: url
    required: true
    description: "The base URL of the HTTP endpoint to invoke"
    example: '"http://host:port/path", "http://myservice:8000/customer"'
    # If omitted, uses the same values as "<root>.binding"
    binding:
      output: true

For more information please see this draft PR https://github.com/ItalyPaleAle/dapr-components-contrib/pull/3

Next steps: @dapr/maintainers-components-contrib and @dapr/approvers-components-contrib should agree on this strategy. Once this is done @berndverst and @ItalyPaleAle will work on finalizing the schema and adding schema validators that can be used to ensure every component has a metadata info yaml.

Going forward approvers and maintainers should ensure that whenever components are created or updated the associated metadata info yaml file is also updated. In the future we will power our component documentation from these files after all.

[berndverst]

@yaron2 @artursouza as co-maintainers, do you have any objections to this?

This would really help with keeping documentation in sync in the future -- instead of requiring a separate docs PR, we can just enforce that the metadata info is provided in this structured format and then generate the component docs from there.

FYI @msfussell

[yaron2]

+1

[1046102779]

On the basis of the metadata defined above, we can add a metadata key validation. In this way, in the non-init phase, we can restrict and constrain the incoming dynamic metadata through enumeration values such as required, optional and denied`.

metadata:
      - name: masterKey
        validation: required # optional , denied.  Instead of  required: true.
        sensitive: true
        description: "The key to authenticate to the Cosmos DB account."
        example: '"my-secret-key"'

Relate issue: https://github.com/dapr/dapr/issues/5040

[ItalyPaleAle]

@1046102779 We originally proposed required as a boolean because we assumed that a metadata property would either be required or optional. What is the use case for denied?

Also as for validation, want to clarify that at this stage we do not plan to have support for "complex" validation rules (we will only have support for marking a property as required and for specifying its type, such as string/number/enum). We will look into adding more complex validation rules (regex matching, min/max bounds, etc) in the future, although that can get very complex, very quickly (it will soon turn into an ask for a Turing-complete language!)

[1046102779]

@1046102779 We originally proposed required as a boolean because we assumed that a metadata property would either be required or optional. What is the use case for denied?

Also as for validation, want to clarify that at this stage we do not plan to have support for "complex" validation rules (we will only have support for marking a property as required and for specifying its type, such as string/number/enum). We will look into adding more complex validation rules (regex matching, min/max bounds, etc) in the future, although that can get very complex, very quickly (it will soon turn into an ask for a Turing-complete language!)

In the financial payment business scenario, we use the pubsub component. We hope to use the delayed msg deliverAfter param of pulsar to control duration that the delayed message stays in the queue. This time is fixed and cannot be changed. So we don't want MicroLogic to dynamically pass this param through metadata. So with denying this action.

In other scenarios, we may restrict MicroLogic more strictly to prohibit the use of metadata, and only allow to use all metadata stored in the YAML initialization phase. make it mandatory to regulate developers.

[1046102779]

@berndverst @ItalyPaleAle

1. authenticationProfiles[i].metadata.required
2. authenticationProfiles[i].metadata.sensitive
3. metadata[i].required

Here are a few questions to ask:

1. Is jsonschema officially defined, or is it defined by the MicroLogic developers themselves?
2. Where are all defined jsonschema files loaded? And the configurations such as required and sensitive exist in the object of the Component (components-contrib repo) or the object of Dapr Runtime (dapr repo)? Or somewhere else?
3. Does the required and sensitive params validation occur in the initialization phase of the component or in the non-initialization phase?
4. If dynamic verification of metadata is performed in the non-initialization phase of a component, is it necessary for all component objects to provide a method for obtaining metadata?

[ItalyPaleAle]

I opened the PR here: https://github.com/dapr/components-contrib/pull/2077

The schema is officially defined: https://github.com/dapr/components-contrib/blob/190a304365c1c9f645a84a414b0b7d568977595d/component-metadata-schema.json (auto-generated from https://github.com/dapr/components-contrib/blob/190a304365c1c9f645a84a414b0b7d568977595d/.build-tools/pkg/metadataschema/schema.go ).

The component schema files will be defined in components-contrib: see example here: https://github.com/dapr/components-contrib/blob/190a304365c1c9f645a84a414b0b7d568977595d/bindings/azure/servicebusqueues/metadata.yaml

Currently, required and sensitive parameters are not enforced at runtime. We are only planning to use them for docs at this time. Your PR may implement that in the runtime.

[1046102779]

Thanks. @ItalyPaleAle .

Currently, required and sensitive parameters are not enforced at runtime. We are only planning to use them for docs at this time. Your PR may implement that in the runtime.

My PR should require the components-contrib repo to provide a method for obtaining metadata, including metadata.Base data in the object.