Update azure pipelines download step

[ajdean]

I have tried to complete the implementation around the AzurePipelinesDownloadStep so that artifacts can be consumed from other jobs/targets. Now generates a DownloadBuildArtifacts@0s task that looks something like:

- task: DownloadBuildArtifacts@0
  inputs:
    buildType: 'current'
    downloadType: 'specific'
    itemPattern: |
      artifacts/*.zip
    downloadPath: '$(Build.Repository.LocalPath)'

~~As an aside I did seem to run into an issue when trying to use the updated package in anger. When running the build I got the following error:~~

Error output:
   Unhandled exception. System.TypeInitializationException: The type initializer for 'Nuke.Common.NukeBuild' threw an exception.
    ---> System.Exception: Resolving argument 'Host' failed.
   Value 'AzurePipelines' could not be converted to 'Host'
   Arguments were:
     [0] = --generate-configuration
     [1] = AzurePipelines
     [2] = --host
     [3] = AzurePipelines
      at Nuke.Common.Assert.Fail(String message, Exception exception) in C:\Dev\github\nuke\source\Nuke.Utilities\Assert.cs:line 28
      at Nuke.Common.ArgumentParser.ConvertArgument(String argumentName, String[] values, Type destinationType, Nullable`1 separator) in C:\Dev\github\nuke\source\Nuke.Utilities\ArgumentParser.cs:line 140
      at Nuke.Common.ArgumentParser.GetNamedArgument(String argumentName, Type destinationType, Nullable`1 separator) in C:\Dev\github\nuke\source\Nuke.Utilities\ArgumentParser.cs:line 81
      at Nuke.Common.ParameterService.GetCommandLineArgument(String argumentName, Type destinationType, Nullable`1 separator) in C:\Dev\github\nuke\source\Nuke.Build\Execution\ParameterService.cs:line 172
      at Nuke.Common.ParameterService.<GetParameter>g__TryFromCommandLineArguments|18_0(<>c__DisplayClass18_0& ) in C:\Dev\github\nuke\source\Nuke.Build\Execution\ParameterService.cs:line 143
      at Nuke.Common.ParameterService.GetParameter(String parameterName, Type destinationType, Nullable`1 separator) in C:\Dev\github\nuke\source\Nuke.Build\Execution\ParameterService.cs:line 162
      at Nuke.Common.ParameterService.GetFromMemberInfo(MemberInfo member, Type destinationType, Func`4 provider) in C:\Dev\github\nuke\source\Nuke.Build\Execution\ParameterService.cs:line 136
      at Nuke.Common.ParameterService.GetParameter[T](MemberInfo member, Type destinationType) in C:\Dev\github\nuke\source\Nuke.Build\Execution\ParameterService.Statics.cs:line 41
      at Nuke.Common.ParameterService.GetParameter[T](Expression`1 expression) in C:\Dev\github\nuke\source\Nuke.Build\Execution\ParameterService.Statics.cs:line 29
      at Nuke.Common.NukeBuild..cctor() in C:\Dev\github\nuke\source\Nuke.Build\NukeBuild.Statics.cs:line 33
      --- End of inner exception stack trace ---
      at Nuke.Common.NukeBuild.Execute[T](Expression`1[] defaultTargetExpressions) in C:\Dev\github\nuke\source\Nuke.Build\NukeBuild.cs:line 78
      at Build.Main() in C:\Dev\Racti.SampleWebApp\build\Build.cs:line 204

~~I tracked it down to an issue with the Host.AvailableTypes property which didn't seem to be returning any types. Can only assume the issue was perhaps introduced when the project was updated to target .net6.0. Not sure if the fix I have put in is the best, but seems to work.~~

Edit: Have removed the changes for the above fix. Not sure if it is really an issue or just me. Seems better to just stick with the original intent of this PR.

I confirm that the pull-request:

• [x] Follows the contribution guidelines
• [x] Is based on my own work
• [x] Is in compliance with my employer

[matkoch]

I think I stopped working on this because it wasn't clear to me whether to use

• DownloadPipelineArtifact@2 or
• DownloadBuildArtifacts@1

Could you give some information why DownloadBuildArtifacts should be used?

PS: I rebased your branch.

[ajdean]

That's a good question. Seems Microsoft recommend using DownloadPipelineArtifact@2, so I'll have a look into using that task instead.

[ajdean]

@matkoch I have had a look into using DownloadPipelineArtifact@2 and it appears to work OK, but has raised a few questions for me. It would appear to get the most out of the new publish pipeline artifacts tasks (compression and de-duplication etc), you also need to be using the related PublishPipelineArtifact@1, which would obviously expand the scope of the PR.

I guess the question is, would you rather:

1. Just leave the PR as-is and use the DownloadBuildArtifacts task. Can obviously look into switching to the pipeline artifacts at a later date.
2. Update the PR to use the newer DownloadPipelineArtifact@2, but leave the publishing using PublishBuildArtifacts@1. I think if you went down this path you would need to revisit the configuration of the DownloadPipelineArtifact@2 task if you eventually moved to using the PublishPipelineArtifact@1 task.
3. Expand the scope of the PR and wwitch to using both PublishPipelineArtifact@1 and DownloadPipelineArtifact@2. Have got this working, but is obviously a bigger change.

[matkoch]

Publish Pipeline Artifacts is not supported in on-premises. Please use Publish Build Artifacts if you're using Azure DevOps Server or TFS 2018.

[matkoch]

It would still be interesting how these two differ. What are the pros and cons.

But for now I guess DownloadBuildArtifacts should be used.

[ajdean]

Yeah Microsoft's documentation on the difference between the 2 is pretty light. There is a comment from one of the devs here that gives some interesting insight into how the new one functions (the TLDR is basically it can perform much better with large artifacts, other than that not sure there is a great deal of difference?).

[matkoch]

I tested this on the repository but it didn't work... How did you test it?

[ajdean]

In terms of testing, I just built it locally, then updated my nuget.config to have the \output\packages folder as a package source, then added the package to a test project I had.

When I built the test project it generate the azure-pipelines.yml file as expected and once committed, it built successfully in azure devops ops (using a cloud based runner).

What particular issue are you having?

[matkoch]

For instance with this one:

          - task: PublishBuildArtifacts@1
            displayName: 'Publish: test-results'
            inputs:
              artifactName: test-results
              pathToPublish: 'output/test-results'

          - task: DownloadBuildArtifacts@0
            displayName: Download Artifacts
            inputs:
              buildType: 'current'
              downloadType: 'specific'
              itemPattern: |
                output/test-results/*.trx
                output/test-results/*.xml
              downloadPath: '$(Build.Repository.LocalPath)'

The problem is that the file structure isn't preserved... so none of the patterns matches, because output is not part of the published artifact.

[ajdean]

This is a bit of a pain really. I think the download task should really be something like:

- task: DownloadBuildArtifacts@0
  inputs:
    artifactName: 'test-results'
    itemPattern: |
      *.trx
      *.xml
    downloadPath: '$(Build.Repository.LocalPath)/output/test-results'

that way it will work regardless of how many folders deep the artifacts are (I was only testing with a single artifacts folder).

Unfortunately I then ran into this issue where it downloads the artifacts into a folder named after the artifact name (i.e. /output/test-results/test-results), so you still have the issue of it not preserving the paths properly. In this instance I guess you could change the downloadPath so it was just $(Build.Repository.LocalPath)/output, but that would only work if the artifact name was always the name of the last folder in the path (I have a feeling the artifact name should perhaps be something more unique across the various stages and jobs the pipeline may have, but I haven't totally thought it through).

There is a work around on that issue that involves using the CopyFiles@2 to copy the files to the correct play which seems to work ok, but obviously isn't a perfect/most optimal solution. It ends up producing something like:

- task: DownloadBuildArtifacts@1
  displayName: 'Download Build Artifacts'
  inputs:
    artifactName: 'test-results'
    downloadPath: '$(Agent.TempDirectory)'
- task: CopyFiles@2
  #Workaround https://github.com/Microsoft/azure-pipelines-tasks/issues/6739
  displayName: 'Copy Build Artifacts'
  inputs:
    sourceFolder: '$(Agent.TempDirectory)/test-results'
    targetFolder: '$(Build.Repository.LocalPath)/output/test-results'
    contents: |
      *.trx
      *.xml

Anyway, let me know if you think the work-around is acceptable (also perhaps worth noting the issue doesn't seem to exist with the newer Pipeline Artifacts, so that could be a future solution for those users who are able to use them)

[SuperJMN]

Is there any update on this? I've just noticed that Consumes produces nothing for Azure Pipelines.