Have metadata in config to have dsc engine ignore missing resources

[SteveL-MSFT]

Summary of the new feature / enhancement

Some resources may be installed at runtime via a package manager resource. Currently, DSC engine before spawning resources will attempt to discover all used resources. If any one is missing, the config operation fails before any resources are called.

Proposed technical implementation details (optional)

Have new metadata to declare to DSC engine to ignore missing resources:

metadata:
  Microsoft.DSC:
    missingResources: ignore
resources:
- name: Install Foo
  type: Microsoft.WinGet/Package
  properties:
    id: foo
- name: Use Foo
  type: Foo/bar
  properties:
    hello: world

missingResources would accept ignore, warn, error where if not specified defaults to error. warn emits a warning message, but proceeds.

[denelon]

This is a great solution for WinGet as a "high-order" tool to still allow a configuration to be executed even if WinGet and DSC.exe can't validate the resources that will end up getting installed as some subset of the configuration document.

[SteveL-MSFT]

Thinking about this a bit more, there's actually a bit more complication to this.

What happens now is that when given a config, the DSC engine will first validate it is valid in terms of resources. So it does a complete discovery of all resources and determines if the ones used are in that list. If any are missing, an error is returned and deployment stops.

In this scenario, a resource may install other resources used in the config. So ignore won't work because discovery only happens at the start. So instead, I think what we need is:

metadata:
  Microsoft.DSC:
    resourceDiscovery: [ preDeployment | duringDeployment ]

preDeployment is current behavior and the default if not specified. It attempts during validation stage to find all resources and reconcile them with the ones specified in the config. Any missing results in no deployment attempt.

duringDeployment will defer discovery until the first resource is used and re-discovery has to happen for each new resource used.

[SteveL-MSFT]

Thinking about this further, I think I might prefer having resources like WinGet/Package declare in their manifest a capability like resource installation. Then, any resources that dependsOn that resource won't be flagged as missing and after execution of WinGet/Package, it would force rediscovery of resources. However, this does require the config author to denote potentially not installed resources as potentially being installed at deployment time. This makes it more specific than the global metadata above which would skip over resources that aren't there and won't be installed.

[ThomasNieto]

Adapted resources would also need to be able to specify they can install resources.

[SteveL-MSFT]

Adapted resources would also need to be able to specify they can install resources.

That's true. As our future apt resource would be built on the Python adapter. This shouldn't be a problem if we extend the schema since an adapter list operation is expected to return instances of DscResource objects. But we do need to make sure this scenario is covered.

[michaeltlombardi]

I would propose that:

1. Resources that (may) install other resources should indicate this functionality in the resource manifest.
2. Resources that (may) install other resources should use a canonical resource property to tell DSC when they have installed a resource. For command resources, this should either:
   • Work similarly to discovery extensions, returning the path to any manifests for the newly-available resource.
   • Indicate that DSC should invoke discovery again to update the installed resource list.
3. Resources that use dependsOn to reference a resource that may install a resource can be deferred for validation error.
4. We should probably define a method for package management resources to retrieve the metadata for an instance from the repository, rather than the system (query or find maybe?).

Ideally, it would be nice to be able to rely on the following order of operations:

1. Check the node graph from the configuration document for missing resources.
2. If an instance has a missing resource but defines the dependsOn field that points to a package manamement resource, invoke the find method for that resource against the defined dependency resource instance properties.
3. If the find method indicates that the package would install the missing resource type name, don't error or warn. Otherwise, either fail the execution (default behavior) or defer until after the dependent resources execute.
4. If any resource invocation indicates that it installed one or more resources, update the resource list as needed.
5. If an instance is being processed for a resource that still doesn't exist, raise an error.

This would enable users to define configurations that install resources and use those resources in a single run in a way that we could validate before actually changing the system. Without being able to invoke a resource or extension method to discover installable resources, I don't see how we could effectively validate whether a configuration using not-yet-installed resources will fail at execution time. This is still limited to using package management resources that implement this capability, but I think that's okay. The alternative is a two-stage application where you install prerequisite software and then configure the machine for the remaining items.

[SteveL-MSFT]

For 3.2 we'll do https://github.com/PowerShell/DSC/issues/1170#issuecomment-3387751807 and later the other explicit manifest declarations to validate more concisely at deployment time