Feature request: Push to multiple registries

[Danielku15]

Currently we can only push the image to a single output registry which is fine for most cases but in some cases you might want to push into multiple registries (e.g. company internal registry + cloud registry). For those usecases it would be great to have not a single OutputRegistry but rather a OutputRegistries and we would push to all of them.

In some cases you might be able to build this using classical docker commands (let sdk-container-builds push to local daemon and take it further from there) but this requires a fully operational docker daemon and quite some own scripting (docker tag and docker push).

But we have hardware and software level restrictions on our build agents which prevent us to follow this path. On some we have (older) AMD hardware which doesn't support nested virtualization on Windows and others are in a new V-Sphere cluster which also lacks support of nested virtualization in combination with Hyper-V which is also not really fully supported.

Therefore: would be great to have pushes to multiple registries out-of-the-box.

[baronfel]

Couple things to consider:

• should the repository name be required to be the same across all registries, or can it vary per-registry?
• should the tag list be required to be the same across all registries, or can it vary per-registry?

[Danielku15]

I think for this feature request we could keep everything consistent, same container name and tag (aka. repository) per registry.

The use case of different names and tags per registry can be fulfilled with calling dotnet publish multiple times with respective properties set. This will result in a different container ID though, but could be a temporary workaround to progress step-wise and see if this is becoming really a market requirement.

Update: There is one additional detail to the repository name that I had missed. There is a "namespace" portion that might be used by the registry to place the image correctly. Those namespaces might be different on each registry. e.g. on a local registry it might go into a "department" namespace while on the cloud registry the namespace might be omitted or related to the whole "company".

[baronfel]

I struggle with how this should be presented to users. None of the contemporaries of this project (notably Jib and ko) support multi-registry push from a single command. Is that because it is technically hard? Not ergonomic to use? As proposed in dotnet/sdk#32132, all namespace/name/tag/etc data must be the same across all registries being pushed to - is that even generally what would be expected from a command like this? Especially given that in https://github.com/dotnet/sdk-container-builds/issues/381 we were considering adding something like ContainerRepositoryPrefix.

Allowing for different 'destination' metadata would probably mean that we would need to refactor to an Item instead of simpler Properties, but that's another leap in user knowledge that we'd require. I'm not entirely convinced that this is more useful than leaning into more local caching and requiring more publish calls. In fact, you could probably simulate this with an MSBuild-batched call to the PublishContainer target, with different values of ContainerRegistry set (just tested, yes you can):

  <ItemGroup>
    <DestinationRegistry Include="sdk-container-demo.azurecr.io" />
    <DestinationRegistry Include="ghcr.io" />
  </ItemGroup>

  <Target Name="MultiPush">
    <ItemGroup>
      <_SingleRegistryPublish Include="$(MSBuildProjectFullPath)" AdditionalProperties="ContainerRegistry=%(DestinationRegistry.Identity)" />
    </ItemGroup>
    <MSBuild Projects="@(_SingleRegistryPublish)" Targets="PublishContainer" BuildInParallel="true" />
  </Target>

and of course other properties could be set in the same way.

[Danielku15]

Talking on use case level

We'd like to build the software (container image) once and the publish/push it in the right way to the locations we want. And we want this without involving any container daemon because we do not have it available everywhere in the right configurations (windows/linux). Some builds are pushed to a single registry only (pre release) while other builds (release versions) are pushed to multiple registries. The registry list is coming from the CI system as input.

So where do we have a problem today?

sdk-container-builds is doing the whole workflow at once: It takes the publish folder, packages a layer, bundles an image (if needed) and pushes it to a single target.

Making multiple calls results in different layers and images being generated. There is hardly a possibility to have the same layer (even hash-wise) published to multiple locations with the provided tooling.

It requires external tooling to publish the same layer to multiple locations. e.g. push to local daemon, tag it additionally and push it further to remote registries.

I personally want to avoid that we are making "different" builds and images just for different pushes to multiple locations. It wastes resources and makes things potentially prone to errors due to differences across systems. Devs internally pull the images from the in-house registry while in production there is a cloud registry.

How sdk-container-build could solve this problem

I see personally following options:

a) There is a good way to tell the framework all locations to push to (with potentially different targets like different registries, local outputs etc.). This is the proposal in https://github.com/dotnet/sdk/pull/32132 b) There is a way to separate the two operations and customize the workflows as needed. One action would be the generation of the image (layers and manifests) and the other one is pushing it to daemons and registries.

While a) gives an easier out-of-the-box experience to devs and is likely easier to setup when it comes to tooling. b) gives more flexibility but also means quite a rework of the toolkit and would mean more work for devs to use it. Combining could be possible.

[Danielku15]

@baronfel What do you think about the proposal I made here? Shall I try to prepare a change where we can specfiy registries and tags like in Docker as a unified solution to the different issues we have open?

[bmcclory]

A variation on this scenario: "Promoting" already published container images to additional registries.

We're using the .NET container build tools to publish multiple container images from a solution with a single dotnet publish <sln> command. This has eliminated a lot of complexity from our pipelines and solution source: No more hand-rolled Dockerfiles, .NET publish/docker build/docker push pipeline tasks all collapsed into a single task, and the pipelines don't need to maintain any static list of Docker projects. The tooling discovers all "container publishable" projects in the solution. Fantastic!

But later in our pipelines, we need to "promote" those previously published images from a development registry to a production registry. With Docker CLI this is just a sequence of pull/tag/push operations. But which images to promote? We're back to statically enumerating the full image URIs of all container image projects in our solution.

It'd be great if there was some target in here that could handle this.

[baronfel]

In general we don't have a design for what using the .NET SDK as a general-purpose image manipulation utility might look like. I personally view our tech as the 'entrypoint' into the broader space of container-management tools, and after generating containers with the SDK I'd be looking at more broadly used tools like skopeo, regclient, or dredge to move the containers around. To that end, we do return some data about the generated images from the PublishContainer Target:

• GeneratedContainerManifest - the JSON manifest describing the container layers and config
• GeneratedContainerConfiguration - the JSON configuration file that describes how to run the container, as well as image metadata
• GeneratedContainerDigest - the unique identifier of the newly-created container. with this you should be able to do commands like docker tag to re-tag the image, docker push to push it to remote registries, etc.
• GeneratedArchiveOutputPath - if you chose to push the image to a tar.gz archive, this is the path to that file so you know where to find it

Given these outputs,

• find the projects to publish
• publish them
• containerize them
• grab data about the containerized projects and then do...something else with it? write it to a file, Exec docker commands, etc.

If that sounds interesting, I do have some sample code that shows how you might do this here. In this case I'm containerizing the same application for multiple architectures, and then using the docker CLI to create a multi-image manifest using the generated image names, but you could vary this slightly using the steps I wrote above to get the same effect.

Looking at this I think it's reasonable that the PublishContainer task should output the full name of the generated container image as well, so I'll log an issue for that.

[bmcclory]

I personally view our tech as the 'entrypoint' into the broader space of container-management tools

Fully agree.

I'm mostly looking for guidance -- which you've generously provided -- to ease the transition from the .NET tooling "entrypoint" into other specialized container management tools / tasks. My first stumbling block with the tooling was figuring out how hook into the MSBuild outputs from the targets so I could

grab data about the containerized projects and then do...something else with it

:)

I'll look at your samples more closely and follow-up with any questions!

[baronfel]

@bmcclory sounds good - and of course I'm happy to keep adding scenarios to the baronfel/sdk-container-demo repo to clarify specific use cases for doing more with the SDK Container tech. Right now it's fairly barebones (mostly building different kinds of applications), but I could definitely see a section of 'cookbook'-style samples that illustrate practical use cases like this.

[bmcclory]

Just to close the loop here, and for anyone else with similar use case, I went with the simple approach of a custom PromoteContainer target that depends on ComputeContainerConfig (which sets properties like ContainerImageTag and ContainerRepositoryName). That's the target I was looking for. My custom target execs a few simple Docker CLI commands:

    <Target
        Name="PromoteContainer"
        DependsOnTargets="ComputeContainerConfig"
        Condition="'$(IsPublishable)' == 'true' AND '$(EnableSdkContainerSupport)' == 'true'">

        <PropertyGroup>
            <DevImage>$(DevContainerRegistry)/$(ContainerRepository):$(ContainerImageTag)</DevImage>
            <ProdImage>$(ProdContainerRegistry)/$(ContainerRepository):$(ContainerImageTag)</ProdImage>
        </PropertyGroup>

        <Message Importance="high" Text="Promoting $(ContainerRepository):$(ContainerImageTag)" />
        <Exec Command="docker pull $(DevImage)" />
        <Exec Command="docker tag $(DevImage) $(ProdImage)" />
        <Exec Command="docker push $(ProdImage)" />
    </Target>

Not the most robust -- all based on tags and not image digests, assumes a publish to DevContainerRegistry was done previously, etc. -- but assumptions that work for my use case. I can now do dotnet msbuild <solution> /t:PromoteContainer in a pipeline and it "promotes" all the images that were published (/pushed) in an earlier pipeline stage (from dotnet publish <solution> /t:PublishContainer).

Thanks for the help.