Class-based resource capabilities in module manifest

[ThomasNieto]

Prerequisites

• [x] Write a descriptive title.
• [x] Make sure you are able to repro it on the latest version
• [x] Search the existing issues.

Summary

There is an option in the module manifest to set which capabilities are present for a resource however that applies to all resources in the module and not to individual resources.

https://github.com/PowerShell/DSC/blob/7ed105eef99ab73b05da14759f4f2cab01e2546d/powershell-adapter/psDscAdapter/powershell.resource.ps1#L90-L100

The schema should be updated to allow individual resources to specify their capabilities.

PSData = @{
  DscCapabilities = @{
    TestClassResource = @('get', test', 'export)
    NoExport = @('get', test')
  }
}

The ideal state is to allow class-based resources to not have to implement all methods. This could be done by checking the AST if the method contains a throw [NotImplementedException]::new() so that the code and capabilities match at all times.

Steps to reproduce

n/a

Expected behavior

n/a

Actual behavior

n/a

Error details

Environment data

n/a

Version

3.1-preview6

Visuals

No response

[michaeltlombardi]

I think it would be useful to consider an all-up re-approach to indicating metadata about DSC resources in module manifests. For example, something like:

@{
  PSData = @{
    DscResources = @{
      MyResource = @{
        Capabilities = @('get', 'test', 'set')
        Description = '<synopsis of the resource purpose and usage>'
        Tags        = @('Windows') # Optional
        Kind        = 'resource'   # Default if not specified
        Exceptions  = @(
          @{
            Type           = 'System.UnauthorizedAccessException' # The exception thrown
            MessagePattern = 'requires elevation'                 # Optional pattern to match
            ExitCode       = 2                                    # Exit code to propagate
          }
        )
      }
    }
  }
}

Right now, we don't have a way to surface most of this information. In the future, with the resource development kit (RDK), we'll be able to surface more of this information easier - but I think it's probably still a good idea to define a model to shortcut the adapter needing to analyze resources in full, particularly for use cases like searching for resources or integrating into editors and other higher-order tools.

[michaeltlombardi]

Thought about this a bit more, and I think what I would propose looks more like this:

# MyModule.psd1
@{
    PSData = @{
        'Microsoft.DSC' = @{
            Resources = @(
                @{  # Bundled into manifest for class resource
                    Type = 'MyModule/MyClassResource'
                    Description = 'description'
                    Tags = @('pwsh', 'Windows', 'Linux', 'macOS', 'foo')
                    Kind = 'resource'
                    Capabilities = @('get', 'set', 'export')
                    Schema = @{
                        # JSON Schema
                    }
                }
                @{ # Reference to external resource file (relative to manifest):
                    Manifest = 'MyModule.MyClassResource.dsc.resource.pwsh.yaml'
                }
            )
            # Alternately, point to a JSON or YAML resources file (relative to manifest):
            Resources = 'MyModule.dsc.resources.pwsh.json'
            Extensions = @(
                # Defines similar structure but for extensions implemented in PowerShell
            )
            Configurations = @(
                @{
                    Path = 'Example.dsc.config.yaml' # Relative to manifest
                    Description = 'Explanation of config purpose'
                    Tags = @()
                }
            )
        }
    }
}

Notes:

1. We use the Microsoft.DSC key instead of DscResources.
2. This key relates to how DSC understands the resources. It has no relation to the DscResourcesToExport key or PSDSC. Resources may be PSDSC compatible, but that's handled by existing manifest structure.
3. We could use this to support defining extensions in PowerShell (if we extend the adapter to surface/invoke extensions). We could also support surfacing shared configuration documents. Both are entirely excluded when we use DscResources as the top-level key.
4. We could support directly defining the data in the module manifest or referencing a separate data file, which has the same structure and schema. This is useful for enabling integrating tools to review a resource without needing to execute PowerShell or implement their own PSD1 parser.
5. We could support defining the data files one-per-resource or as a larger blob, which is just an array of definitions.
6. While hand-crafting this is very not-fun, we could provide DevX tooling to generate this information by inspecting the module. With that in place, we would advise authors to incorporate that tooling into their build pipeline instead of trying to ensure the implementation and data representation stay in sync.
7. While we can't (currently) provide much of an enhanced DevX for authoring this data in the module manifest, we can provide a pretty functional DevX for viewing/editing the data files with the enhanced authoring schemas for VS Code.

[gaelcolas]

I think that's a great way to expose the required metadata to the adapter, but is it the best?

If we put aside the adapter implementation for now, could DSC discover PowerShell class-based resources shipped within a module just by looking at a json (i.e. without even spinning up pwsh).

Maybe adding the complete metadata object in the module manifest is a bit too much, and would still require to read the psd1 (spin up pwsh). Couldn't we have a convention that if there's a ModuleName.Microsoft.DSC.json (by convention) along the module PSD1 or PSM1 it would just find it and read the resources capabilities & info? Then the right way to have the metadata would be in the class definition in the form of an attribute (and some could be derived/deduced from the class definition).

The RDK (or anyone) could create the attribute, and provide the tools used at compilation to generate the json. Attribute would be in a third module that has to be Required/Nested for the parser.

Let's discuss, and we could discuss that during the community call tonight.

[gaelcolas]

As discussed with Mikey, discovery needs to look into $Env:PSModulePath, not Path... So it should still be done from the adapater.

[SteveL-MSFT]

WG discussion is to have a new .dsc.adaptedresource.json (or yaml) file which is the current DscResource schema (not the manifest) with the addition of a new schema property that holds the JSONSchema. A new discover extension would traverse $env:PSModulePath to look for these files and the discover extension would be enhanced to allow extensions to return a list of manifests or a list of these new adapted manifests.

This will make discovery much faster than currently which needs to import the module or parse the module manifest. These resources will also just show up via dsc resource list without specifying the adapter making discovery better in addition to being faster. The current engine logic will fallback to querying adapters if the resource isn't found so existing PS class based resources will still be found the "old way".

[ThomasNieto]

Is .dsc.adaptedresource.json created at build time with the module author or is it generated at runtime with the adapter/discovery extension?

Runtime built .dsc.adaptedresource.json files in the module manifest has a few issues. The user has to have write access to $env:PSModulePath which works for user installed modules but not machine installed without escalation.

Module shipped .dsc.adaptedresource.json requires an extra step from the module author and can lead to a mismatch between resource and schema.

Adapter flow

flowchart TD
    A[Resource Lookup]
    B[Adapter]
    C[Adapter Cache]
    D{Resource Found?}
    F[Return Resource Metadata]
    H[Search $env:PSModulePath]
    I[Read Module Manifest]
    J[AST Parse Module]
    K[Return Resource Metadata]

    A --> B
    B --> C
    C --> D
    D -->|Yes| F
    D -->|No| H
    H --> I
    I --> J
    J --> K
    K --> F

Discovery with pre-built manifests

This method is required for resources and extensions using this model like PSResourceGet: https://github.com/PowerShell/PSResourceGet/pull/1852

This method is already in scope in issue https://github.com/PowerShell/DSC/issues/913

flowchart TD
    A[Resource Lookup]
    B[Search $env:PSModulePath]
    C[Return Resource Manifest Paths]

    A --> B
    B --> C

Discovery with runtime built manifests (new adapter)

Proposed discovery extension without needing adapters.

flowchart TD
    A[Resource Lookup]
    B[Discovery Extension]
    C[Discovery Resource Cache]
    D{Resource Found?}
    F[Return Resource Manifests]
    H[Search $env:PSModulePath]
    I[Read Module Manifest]
    J[AST Parse Module]
    K[Build full manifest]

    A --> B
    B --> C
    C --> D
    D -->|Yes| F
    D -->|No| H
    H --> I
    I --> J
    J --> K
    K --> F

Example manifest

Leveraging the newly added single manifest with multiple resources (https://github.com/PowerShell/DSC/pull/1187) we can accomplish the following.

{
  "manifests": [
    {
      "$schema": "https://aka.ms/dsc/schemas/v3/bundled/resource/manifest.json",
      "type": "AnyPackageDsc/Package",
      "version": "0.1.0",
      "get": {
        "executable": "pwsh",
        "args": [
          "-NoLogo",
          "-NonInteractive",
          "-NoProfile",
          "-ExecutionPolicy",
          "Bypass",
          "-Command",
          "$Input | ./psDscAdapter/powershell.resource.ps1 Get"
        ],
        "input": "stdin"
      },
      "set": {
        "executable": "pwsh",
        "args": [
          "-NoLogo",
          "-NonInteractive",
          "-NoProfile",
          "-ExecutionPolicy",
          "Bypass",
          "-Command",
          "$Input | ./psDscAdapter/powershell.resource.ps1 Set"
        ],
        "input": "stdin"
      },
      "schema": {
        "embedded": {
          "$schema": "https://json-schema.org/draft/2020-12/schema",
          "type": "object",
          "properties": {
            "name": {
              "type": "string"
            },
            "version": {
              "type": "string"
            },
            "latest": {
              "type": "boolean"
            }
          },
          "required": [
            "name",
            "version"
          ],
          "additionalProperties": false
        }
      }
    },
    {
      "$schema": "https://aka.ms/dsc/schemas/v3/bundled/resource/manifest.json",
      "type": "AnyPackageDsc/Source",
      "version": "0.1.0",
      "get": {
        "executable": "pwsh",
        "args": [
          "-NoLogo",
          "-NonInteractive",
          "-NoProfile",
          "-ExecutionPolicy",
          "Bypass",
          "-Command",
          "$Input | ./psDscAdapter/powershell.resource.ps1 Get"
        ],
        "input": "stdin"
      },
      "set": {
        "executable": "pwsh",
        "args": [
          "-NoLogo",
          "-NonInteractive",
          "-NoProfile",
          "-ExecutionPolicy",
          "Bypass",
          "-Command",
          "$Input | ./psDscAdapter/powershell.resource.ps1 Set"
        ],
        "input": "stdin"
      },
      "schema": {
        "embedded": {
          "$schema": "https://json-schema.org/draft/2020-12/schema",
          "type": "object",
          "properties": {
            "name": {
              "type": "string"
            },
            "location": {
              "type": "string"
            },
            "trusted": {
              "type": "boolean"
            }
          },
          "required": [
            "name",
            "location"
          ],
          "additionalProperties": false
        }
      }
    }
  ]
}

What does using discovery get us over adapters?

• Consistent interface between engine and adapters, discovery extensions and individual resources. (Everything returns a resource manifest)
• New features can be added to the engine and added to discovery extension later without adding new capabilities into the adapter interface. See https://github.com/PowerShell/DSC/issues/872 for adding schema to adapters.
• Resources are not hidden behind the adapter parameter.
• Runtime built manifests returned via discovery do not require complex adapter logic such as understanding configuration documents. The only thing needed is a thin translation layer from JSON to the PowerShell resource and back to JSON.

If you have any questions or comments feel free to post and I'll answer them to the best of my ability.

This is quite a lot to describe over text, I plan on creating a proof of concept and can describe over the working group call.

[ThomasNieto]

In the working group call last week it was mentioned that there would be a requiredAdapters property where the adapted resource declares which adapters are supported.

The adapted resource should not assert which adapters are supported but rather the discovery/adapter system that should make that assertion. Here is an example of the implications, lets say for example someone forks the PowerShell adapter to either add new features or to use pwsh-preview.exe In that case if the resource author declares which adapters are supported forking of adapters or custom built adapters that support existing paradigms are not allowed.