Generate template APIs from YAMLs

[premun]

As a partial replacement for #150, we could also generate the template APIs from YAML files.

Goal

User will give us YAML directory, we will produce APIs for each template that could be called from C#.

Example

User gives us install-dotnet.yml template:

parameters:
- name: version
  type: string

- name: fullSdk
  type: boolean
  defaultValue: true

We generate something like this:

public static class Templates
{
    public static Template<Step> InstallDotNetTemplate(string version, bool fullSdk = true);
}

which would return a YAML like referencing the template (we can get the path and everything), something like:

  template: {path}
  parameters:
    version: {version}
    fullSdk: {fullSdk}

[premun]

I made some progress, it's in the branch but it will require quite a lot of work still unfortunately

[premun]

@radical I returned to this after some time and made good progress to the point where this is something useful maybe even. You can try it out. It's a lot of hacky code assembled together very fast but it does give nice results.

Instructions

PowerShell on Windows, but will work on other platforms too:

git clone https://github.com/premun/sharpliner
cd sharpliner
git checkout template-tools
cd sharpliner\src\tools\Sharpliner.Tools
dotnet run -- template-api -o RuntimeTemplates.cs `
  D:\runtime\eng\pipelines\common\build-coreclr-and-libraries-job.yml `
  D:\runtime\eng\pipelines\common\download-artifact-step.yml `
  D:\runtime\eng\pipelines\common\download-specific-artifact-step.yml `
  D:\runtime\eng\pipelines\common\evaluate-changed-darc-deps.yml `
  D:\runtime\eng\pipelines\common\evaluate-changed-paths.yml `
  D:\runtime\eng\pipelines\common\evaluate-default-paths.yml `
  D:\runtime\eng\pipelines\common\evaluate-paths-job.yml `
  D:\runtime\eng\pipelines\common\global-build-job.yml `
  D:\runtime\eng\pipelines\common\internal-variables.yml `
  D:\runtime\eng\pipelines\common\macos-sign-with-entitlements.yml `
  D:\runtime\eng\pipelines\common\perf-variables.yml `
  D:\runtime\eng\pipelines\common\platform-matrix-multijob.yml `
  D:\runtime\eng\pipelines\common\platform-matrix.yml `
  D:\runtime\eng\pipelines\common\restore-internal-tools.yml `
  D:\runtime\eng\pipelines\common\upload-artifact-step.yml `
  D:\runtime\eng\pipelines\common\upload-intermediate-artifacts-step.yml `
  D:\runtime\eng\pipelines\common\variables.yml `
  D:\runtime\eng\pipelines\common\xplat-setup.yml

This will produce a file RuntimeTemplates with the C# APIs that you can include in your pipelines and they will include the right - template: ... reference.

Example output

using System.Collections.Generic;
using Sharpliner.AzureDevOps;
using Sharpliner.AzureDevOps.ConditionedExpressions;
using Sharpliner.AzureDevOps.Tasks;

namespace Pipelines;

public static class RuntimeTemplates
{
    // /eng/pipelines/common/download-artifact-step.yml
    public static Template<Step> DownloadArtifactStep(
        string unpackFolder = "",
        bool cleanUnpackFolder = true,
        string artifactFileName = "",
        string artifactName = "",
        string displayName = "") => new("/eng/pipelines/common/download-artifact-step.yml", new()
    {
        { "unpackFolder", unpackFolder },
        { "cleanUnpackFolder", cleanUnpackFolder },
        { "artifactFileName", artifactFileName },
        { "artifactName", artifactName },
        { "displayName", displayName },
    });
}

You can then reference this somewhere in your C# pipeline:

Problems

There are some problems with how the runtime YAML templates are written that make this non-100%. I will explain briefly.

In YAML, there are two ways to define what parameters the template expects.

1. Lazy way, just names and default values

   parameters:
     unpackFolder: ''
     cleanUnpackFolder: true
     artifactFileName: ''
     artifactName: ''
     displayName: ''
   

2. Fully specified parameters with types and other

   parameters:
   - name: myString
     type: string
     default: a string
   
   # no default makes it required which is checked by AzDO before the pipeline runs
   - name: myBoolean
     type: boolean
   

Unfortunately, Arcade and Runtime both use the first and sometimes they specify in comments what is required. This way, the C# API cannot tell what needs a default value and what doesn't which makes it imperfect. For instance above you can see that all string parameters have default values "" which is probably not true in how the template behaves. This also means that we cannot always infer the type, e.g.:

parameters:
  jobs: []