WIP: Add support for JSON Settings Files

[liamjpeters]

PR Summary

Very much a work in progress, but almost all tests pass (new cmdlet needs help for Get-Help) and json files are usable wherever .psd1 settings file can be used.

Json is a widely used format with lots of great existing tooling. It could offer some nice UX in future (could publish JSON Schemas to offer inline validation in tools such as VS Code etc).

Along with changes aimed at making Json (and potentially other formats) viable I wanted settings to be more discoverable. To that end I've:

• Proposed a new cmdlet New-ScriptAnalyzerSettingsFile which would allow creation of settings file in desired format. One option is -All which populates file with all rules and all configurable rule options (set to their defaults).

• Added a list of options to the RuleInfo class. For configurable built-in rules, the options, their types, and default value are populated.

  Get-ScriptAnalyzerRule -Name 'PSAvoidUsingPositionalParameters' | Select -expand Options
  
  Name             OptionType      DefaultValue
  ----             ----------      ------------
  CommandAllowList System.String[] {}
  Enable           System.Boolean  True
  

Additional change detail:

• Internal location of Settings Presets moved from Engine\Settings -> Engine\SettingsPresets so Settings folder can be used for all Settings logic and parsers. Presets are still shipped in the built module's Settings folder.

• Internal location of Command Data Files moved from Engine\Settings -> Engine\CommandDataFiles. Now separated out from the Settings Presets, as presets could now be in .json format - which is what previously distinguished the command data files from presets. They are shipped with the module in a new folder CommandDataFiles.

• GetShippedSettingsDirectory() abstracted to GetShippedModuleSubDirectory(string subDirectoryName) as there are now two folders, Settings, and CommandDataFiles, which consumers may care about. Usages updated in AvoidOverwritingBuiltInCmdlets and UseCompatibleCmdlets which load and use the Command Data Files.

• All state about settings data moved to SettingsData class. Settings class becomes a pure static helper. It's responsible for the supported file formats, preset resolution, auto-discovery, and orchestrating the deserialization of a settings file into a SettingsData and vice-versa.

• An interface ISettingsFormat describes a Settings Format. It can serialize a SettingsData to a string representation of the intended format and deserialize that string representation into a SettingsData. 2 Implementations of this are Psd1SettingsFormat and JsonSettingsFormat.

• The order of the Settings Formats in the Settings class (s_formats) defines the order in which formats are checked during Auto-Discovery and Preset Resolution. For instance if JsonSettingsFormat precedes Psd1SettingsFormat:

  • During Auto-Discovery, if both PSScriptAnalyzerSettings.json and PSScriptAnalyzerSettings.psd1 exist, PSScriptAnalyzerSettings.json will be used.

  • During Preset resolution, if the preset CmdletDesign is used and both CmdletDesign.json and CmdletDesign.psd1 exist in the presets folder, the settings in CmdletDesign.json would be used. Note: In practice, this should be avoided. A test to ensure preset uniqueness should be added. Presets should only be shipped in a single format.

• When passing values in via -Settings:

  • When nothing supplied and no settings file in cwd:

    PS > Invoke-ScriptAnalyzer -Path . -Verbose
    VERBOSE: Settings object could not be resolved.
    

    Same as current behaviour. Auto discovery fails. Built-in defaults used

  • When nothing supplied and a psd1 settings file in cwd:

    PS > Invoke-ScriptAnalyzer -Path . -Verbose
    VERBOSE: Settings not provided. Will look for settings file in the given path C:\Scratch.
    VERBOSE: Found C:\Scratch\PSScriptAnalyzerSettings.psd1. Will use it to provide settings for this invocation.
    

    Different from current which omits the path. Note the space between path and period below which should contain the cwd path.

    PS > Invoke-ScriptAnalyzer -Path . -Verbose
    VERBOSE: Settings not provided. Will look for settings file in the given path .
    VERBOSE: Found C:\Scratch\PSScriptAnalyzerSettings.psd1. Will use it to provide settings for this invocation.
    

  • When nothing supplied and a json settings file in cwd:

    PS > Invoke-ScriptAnalyzer -Path . -Verbose
    VERBOSE: Settings not provided. Will look for settings file in the given path C:\Scratch.
    VERBOSE: Found C:\Scratch\PSScriptAnalyzerSettings.json. Will use it to provide settings for this invocation.
    

    Different from current which doesn't know about the json format.

    PS > Invoke-ScriptAnalyzer -Path . -Verbose
    VERBOSE: Settings object could not be resolved.
    

  • When incorrect type supplied:

    PS > Invoke-ScriptAnalyzer -Path . -Settings $false
    Invoke-ScriptAnalyzer: Settings should be either a file path, built-in preset or a hashtable.
    

    Differs from current behaviour which proceeds with built-in settings, noting only that settings cannot be found. This is inconsistent with when an invalid path is supplied when execution doesn't proceed.

  • When Preset name supplied:

    PS > Invoke-ScriptAnalyzer -Path . -Settings CmdletDesign -Verbose
    VERBOSE: Using settings preset CmdletDesign. File found at C:\Home\PSScriptAnalyzer\out\PSScriptAnalyzer\1.24.0\Settings\CmdletDesign.psd1.
    

    Differs from current behaviour - calls out that a preset was used.

  • When explicit valid psd1 file path supplied:

    PS > Invoke-ScriptAnalyzer -Path . -Settings C:\Scratch\PSScriptAnalyzerSettings.psd1 -Verbose
    VERBOSE: Using settings file at C:\Scratch\PSScriptAnalyzerSettings.psd1.
    

    Same as current behaviour.

  • When explicit valid json file path supplied:

    PS > Invoke-ScriptAnalyzer -Path . -Settings C:\Scratch\PSScriptAnalyzerSettings.json -Verbose
    VERBOSE: Using settings file at C:\Scratch\PSScriptAnalyzerSettings.json.
    

    Different from current behaviour which fails with:

    PS > Invoke-ScriptAnalyzer -Path . -Settings C:\Scratch\PSScriptAnalyzerSettings.json -Verbose
    VERBOSE: Using settings file at C:\Scratch\PSScriptAnalyzerSettings.json.
    Invoke-ScriptAnalyzer: Settings file 'C:\Scratch\PSScriptAnalyzerSettings.json' is invalid because it does not contain a hashtable.
    

  • When explicit invalid file path supplied:

    PS > Invoke-ScriptAnalyzer -Path . -Settings C:\Scratch\NoSuch.psd1 -Verbose
    Invoke-ScriptAnalyzer: Cannot resolve settings file path 'C:\Scratch\NoSuch.psd1'.
    

    Same as current behaviour.

• New Cmdlet New-ScriptAnalyzerSettingsFile. Intent is to make it easy to create settings files and discover available settings/options without having to directly resort to docs. It creates settings files based on presets (or All available settings set to their defaults). Creates file in nominated format at desired path. Path defaults to cwd and file format defaults to highest precedence file format in Settings.cs (currently .json). Has argument completers for FileFormat and Base.

  Gist of json and psd1 files created by cmdlet here.

Still to look at:

• [ ] Lots of testing. Plenty of coverage needed to ensure no regression. Testing around loading formats and that each format produces the same SettingsData object for equivalent input.
• [ ] Need to check on other (case sensitive) platforms that there are no issues.

PR Checklist

• [ ] PR has a meaningful title
  • Use the present tense and imperative mood when describing your changes
• [ ] Summarized changes
• [ ] Change is not breaking
• [ ] Make sure all .cs, .ps1 and .psm1 files have the correct copyright header
• [ ] Make sure you've added a new test if existing tests do not effectively test the code changed and/or updated documentation
• [ ] This PR is ready to merge and is not Work in Progress.
  • If the PR is work in progress, please add the prefix WIP: to the beginning of the title and remove the prefix when the PR is ready.

[bergmeister]

@liamjpeters Will there still be back-compatibility for existing setting files, which I think will be a must? I know JSON is common nowadays but the current format is powershell hashtable format so as long as one works with powershell, reading and editing those setting files should actually be easier in powershell than if it was json file. Json doesn't have a well defined standard and not clear on edge cases like whether keys with different casing aren't allowed or not so we'd open PSSA up for those edge cases. What is the use case? Is an application reading/writing those files?

[liamjpeters]

👋 @bergmeister - I hadn't put much thought into JSON's edge cases - especially with duplicate keys - so that's really valuable feedback - thank you 🙂.

This is WIP and I will hopefully go back to it at some point - unless this is something the project feel strongly isn't right for PSSA.

Backwards-compatibility and not breaking any existing setups is a must, I agree.

The PR, as I left it, passes all existing tests (barring a help documentation one, unrelated to the json/settings stuff). All existing psd1/hashtable usages for -Settings continue to work as they did before - including presets and auto-discovery.

I've not touched the PR in a while as I'm still learning/improving with C# - I'm not sure about the abstractions/structure I've put in place for settings. If you ever felt like giving it a high-level look and giving me some general guidance - I'd really appreciate that - though I know you're a super busy person, so no worries if not.

---

My motivations for this PR actually focus around discoverability of rules and their settings, as well as for my own fun/learning on how settings were implemented in PSSA.

Currently you can use Get-ScriptAnalyzerRule to get an idea of what rules are out there and then you need to resort to docs pages to find out what configuration options are available for each rule.

The PSD1 settings file experience isn't amazing. I appreciate that if you're a PowerShell person then the syntax is familiar to you - it's just a hashtable after all. But that doesn't help you with the structure; there's no tooling to offer validation of a settings file beyond the parser validating that it's legal PowerShell.

Imagine supporting a JSON settings format: We could generate and ship a JSON schema alongside the module. The VS Code extension could use that JSON Schema to offer intelisense and validation with very minimal effort I would hope. If it were published somewhere then most other popular IDEs could do the same too. I notice that Test-Json also has a -Schema and -SchemaFile parameter too!

[TMA-2]

@liamjpeters I've created a JSON schema for PSScriptAnalyzer configuration if it helps at all. There's a lot of room for improvement, obviously, but it does have descriptions and properties for all current rules for what it may be worth.

[liamjpeters]

@TMA-2 That's a really great start and will be a big help when I get back to this! Thanks so much 🙂