Add ability to have extensions to `dsc.exe`

[SteveL-MSFT]

Summary of the new feature / enhancement

We should allow for community extensions to dsc.exe so they show up as sub-commands. For example we may want to have dsc cache --clear which is supported by PS-Adapter extension.

Proposed technical implementation details (optional)

Extensions should be executables (or maybe an extension manifest?) that have -dsc suffix to the command, like cache-dsc.exe

[michaeltlombardi]

Proposals:

1. Define a set of extension points for where/how the extension can interact with DSC.
2. Require a manifest, with a similar data structure as resources to indicate how the extension integrates.

1. Extension Points

I think we probably want to define a few common extension points to enable hooking into existing DSC commands. Separately, we should consider whether we want to support something similar to the GitHub CLI extension model, where integrating developers can define new commands for DSC.

Proposed built-in extension points

• dsc cache <clear|refresh|init> (Cache) - A cache extension defines three commands - one to initialize a cache, one to refresh it, and one to clear it. Users can invoke the cache commands directly (e.g. dsc cache refresh. Before DSC executes a configuration operation, it follows the code path for initializing all known caches. The init command for a cache extension should return as soon as it knows whether the cache is already initialized. I'm not sure what input, if any, these commands should receive. They should exit with code 0 on success and can send messages, but not output, back to DSC.
• dsc config resolve (Resolver) - A resolving extension defines a single command that operates on and returns a DSC configuration document. DSC calls this hook at the beginning of any dsc config operation, before final validation of the resolved document by DSC. This would enable extensions to pre-process a configuration - for example, you could define a resource instance generator or fill in non-specified default values, or whatever. Theoretically, you could also use this extension to enable authoring custom configuration functions. The only input and output for these extensions is a DSC configuration document.
• dsc <command> [<subcommands>] (Command) - a command extension defines a new command for DSC at the top-level. The names of these commands must be unique and can't conflict with built-in commands.

Theoretically, an extension could do more than one of these, but it's probably better to have separate manifests, even if the app is shared across the three kinds.

Example command extensions

# Extension that shows a node graph for a configuration document or result
cat ./app.config.dsc.yaml | dsc graph # Show node graph for config document
dsc config get | dsc graph            # Show node graph for get result

# Extension that shows terminal-help for resources
dsc explain --resource Microsoft/OSInfo                     # Show full help
dsc explain --resource Microsoft/OSInfo --property codename # Show property help only
dsc explain --manifest ./tstoy.dsc.resource.json            # Inspect manifest for help

# Extension that helps with resource manifest DevX
dsc manifest new --type TailspinToys/tstoy --capabilities Get, Set, Export # Scaffold manifest
dsc manifest new --interactive                                             # Create with prompts
dsc manifest validate ./tstoy.dsc.resource.json                            # Test manifest
dsc manifest show ./tstoy.dsc.resource.json                                # Pretty-print manifest
dsc manifest show ./tstoy.dsc.resource.json --schema                       # Pretty-print schema

# Extension that handles packaging, installing, updating resources
# Validate that the MyResource folder has everything needed to publish a resource package
dsc package validate --folder ./MyResource
# Package the resource for publishing
dsc package build --folder ./MyResource --output-folder ./packages
# Publish the resource to the default repository
dsc package publish  ./packages/MyOrg.MyResource.dsc.package.json
# Search the GitHub container registry for resources
dsc package find resource
# List all locally installed packages
dsc package list
# Update locally installed packages
dsc package update

2. Manifests

I think that we should either require standalone manifests for an extension or extend the resource manifest to enable defining extensions. I'm hesitant to support both for the complexity/ambiguity.

I prefer a standalone manifest from an integration, testing, and packaging perspective.

Cache extension

$schema: https://aka.ms/dsc/schemas/v3/bundled/extension/manifest.json
type: Microsoft.DSC/PowerShellCache
description: Manage caching adapted PowerShell resources
tags:
- PowerShell
cache:
  init:
    executable: pwsh
    args:
    - -NoLogo
    - -NonInteractive
    - -NoProfile
    - -Command
    - './psDscAdapter/powershell.cache.ps1'
  clear:
    executable: pwsh
    args:
    - -NoLogo
    - -NonInteractive
    - -NoProfile
    - -Command
    - './psDscAdapter/powershell.cache.ps1 -Clear'
  refresh:
    executable: pwsh
    args:
    - -NoLogo
    - -NonInteractive
    - -NoProfile
    - -Command
    - './psDscAdapter/powershell.cache.ps1 -Refresh'

Example resolver extension manifest

$schema: https://aka.ms/dsc/schemas/v3/bundled/extension/manifest.json
type: Microsoft.DSC/InstanceGenerator
description: Pre-process a configuration to generate instances from a configuration function.
tags:
- DevX
resolve:
  executable: dsc-igen
  input:      stdin

Example custom command extension manifest (args)

$schema: https://aka.ms/dsc/schemas/v3/bundled/extension/manifest.json
type: Microsoft.DSC.Docs/Explain
tags:
- Docs
- Help
customCommands:
  explain:
    executable: dsc-explain
    help:
      arg:   --help
      command: help
    args:
      - name: --resource
        type: string
        description: The type of the resource to show help for.
      - name: --property
        type: string
        description: The property of the resource to show help for.
      - name: --manifest
        type: string
        description: The path to the resource manifest to show help for.
    argSets:
      - required: --resource
        args: [--resource, --property]
      - required: --manifest
        args: [--manifest, --property]
    # In this case, we can be relatively lazy and just indicate it returns a Markdown string;
    # If the extension returned structured data, we would need to have the full schema here
    # for users and integrating tools.
    outputSchema:
      $schema: https://json-schema.org/draft/2020-12/schema
      $id:              https://raw.githubusercontent.com/PowerShell/DSC/main/schemas/2024/04/extensions/explain/output.json
      type:             string
      contentMediaType: text/markdown
      title:            Resource Documentation
      description:      The Markdown documentation for a resource or a property of the resource.

    # Has no subcommands, but use the same definition structure recursively.
    subcommands: []

This is probably the roughest example, as we need to be able to determine the argument definitions from the manifest. This requires much more thorough design and investigation. An alternative would be to require custom commands to specify a JSON schema for input and output both, then generate the CLI arguments from the schema properties (but always send the data to the extension as JSON either over stdin or to the JSON input argument).

Example custom command extension manifest (input schema)

$schema: https://aka.ms/dsc/schemas/v3/bundled/extension/manifest.json
type: Microsoft.DSC.Docs/Explain
tags:
- Docs
- Help
customCommands:
  explain:
    executable: dsc-explain
    help:
      arg:   --help
      command: help
    args: []
    input: stdin
    schemas:
      input:
        $schema: https://json-schema.org/draft/2020-12/schema
        $id:              https://raw.githubusercontent.com/PowerShell/DSC/main/schemas/v3/extensions/explain/input.json
        oneOf:
        - required: [resource]
        - required: [manifest]
        properties:
          resource:
            type:        string
            description: The type of the resource to show help for.
          property:
            type:        string
            description: The property of the resource to show help for.
          manifest:
            type:        string
            description: The path to the resource manifest to show help for.
      output:
        $schema:          https://json-schema.org/draft/2020-12/schema
        $id:              https://raw.githubusercontent.com/PowerShell/DSC/main/schemas/2024/04/extensions/explain/output.json
        type:             string
        contentMediaType: text/markdown
        title:            Resource Documentation
        description:      The Markdown documentation for a resource or a property of the resource.

[michaeltlombardi]

Circling back on this proposal, I want to highlight the value of the extension model.

Given the dsc itself will end up on client machines and servers as well as invoked by higher-order tools:

Whatever ships in dsc itself should be minimal - as small as possible to accomplish its work with as limited an attack surface as possible. The core dsc executable is going to be leveraged heavily in various contexts and the easier it is to ship and patch, the better for everyone.

Enhancing the interactive experience for DSC is extremely useful for development and test scenarios, but may include dependencies and risk vectors you don't want to ship in your higher order tool or install to your production servers.

Examples of improved interactive experiences

• Adding a TUI to DSC for interactive/exploratory usage:

  Calling dsc tui at the CLI could open a user interface enabling a user to select the command they want from the command hierarchy and interactively define the parameters, see the output and messaging streams separately, export the command or results to their clipboard, etc. This would make it much easier for a new user to gain familiarity with DSC, but isn't something I would want on a server by default.

• Adding an enhanced viewer for documentation:

  Right now, the only docs you can review for DSC are the CLI help and browsing the docs on the Learn platform. It would be useful to be able to retrieve and review documentation locally - as Markdown text or a file, in the terminal or the browser, etc. There are, again, good reasons not to want to have an enhanced experience on every server.

• Adding an interactive review command to pipe output to:

  Calling a command like dsc config set ./config.dsc.yaml | dsc review could enable a user to explore the result data, either in a TUI or a REPL. They could step through the results, compare before/after state diffs for a specific instance, review messages, etc. This would be helpful as a debugging/investigation tool, but not necessarily one that higher order tools would want to bundle.

• Adding a lint command:

  When writing or updating a configuration document or resource manifest, it would be helpful to check that document for best practices, suggested improvements, errors, etc. While we could build that functionality directly into DSC, it's primarily an author-time concern.

While the initial work required to support extensions is not minimal, I think it pays off in the long term, both for maintaining DSC itself (we can progressively enhance the DevX/UX without needing to ship new versions of the engine everyone relies on) and for the broader ecosystem.

The proposal I'm working on for the resource development kit(s) would benefit enormously from reusing the extension model and is a useful candidate for an early prototype of this proposal.

[SteveL-MSFT]

Similar to how we have different kinds of resources, we would have different kinds of extensions that would implement our required "interface" (just like how get, set, test, export are exposed). This would be a similar but separate foo.dsc.extension.json file that would also be discovered via PATH env var.