Move release package layout creation to the release infrastructure

[MilenaHristova]

Context

The first step of the .NET staging pipeline is to gather all artifacts associated with a installer BAR ID using darc gather-drop. The command gathers shipping and non shipping assets. Currently it also copies the shipping NuGet packages, associates them with owners and arranges them in a folder structure

publish_files\
    nuget\
      identities\ #csv files containing package name + version, named with short name of repo
      nupkgs\
        repo-shortname\
                 packages\  #shipping packages from repo     
        sympkgs\
            repo-shortname\
                 assets\
                    symbols\  #symbol packages from repo  
         nupkgs-all-just-for-reference.txt   # file list of all packages, 
         nupkgs-aspnet.txt                          # file list of all packages that aspnetcore owns on nuget.org
         nupkgs-core.txt                              # file list of all packages that dotnet core owns on nuget.org
         nupkgs-core-manifest.txt              # file list of all packages that are workload manifests owned by dotnet core on nuget.org
         sympkgs-all.txt                               # file list of all symbol packages

The layout is a requirement for the NuGet Packages publishing stage in the release pipeline

There are a few problems in this implementation:

• Creating a release layout is not part of the main functionality of gather-drop
• It requires darc to know about the release process and makes the gather-drop specific for that purpose
• Any change in the layout require a rollout of arcade-services
• All packages in the publish_files folder are also in the shipping folder so we have to maintain copies of about 12GB of data
• Everything in the publish_files is published and downloaded multiple times in the staging pipeline without being used - this is described in Release layout doc

Goal

The goal is to remove the release layout creation from darc and move it to the release infrastructure.

• Check with the release team if the layout is needed anywhere in the staging process - currently it's being copied to dotnetstage blob storage and to a file share
• In case it's not needed in staging remove all uses of publish_files artifact from the staging pipeline
• Add a stage in the release pipeline that downloads all packages, associates them with owners (using the manifest.json) and separates them into folders
• Make sure workload manifests are published after any package that they reference

[mmitche]

I think that even if the layout is used in the release pipeline, it could potentially be removed and replaced with a manifest. We could source packages from blob storage or the original drop location (signed). The manifest of packages to release, their locations, and their owners would be generated by staging/release.

The key requirements would be:

• Need a workflow to augment/change/etc. the list of packages to replace (e.g. we may need to replace a package, add a new one, or remove one we would release) that doesn't require rerunning the build or staging.
• Minimize file copying