Multi-architecture support

[astefanutti]

At the moment, Camel K binaries and images are available for amd64 architecture only.

Following up on #1215, it would be useful to publish multi-architecture images, so that Camel K can be used on a wider range of architectures. Based on a recent PoC, I've identify the following tasks:

• Camel K:
  • [ ] Use a multi-architecture default Java base image
  • [ ] Build and publish multi-architecture images: BuildKit could be a good candidate to ease the task of building images for all target architectures, as it automatically picks the right base image and creates the multi-architecture manifest
  • [ ] The https://github.com/scylladb/go-set dependency doesn't compile on 32 bits architectures
• Kaniko:
  • [ ] Multi-architecture based image support: GoogleContainerTools/kaniko#646, GoogleContainerTools/kaniko#980
  • [ ] Multi-architecture warmer and executor images

[github-actions[bot]]

This issue has been automatically marked as stale due to 90 days of inactivity. It will be closed if no further activity occurs within 15 days. If you think that’s incorrect or the issue should never stale, please simply write any comment. Thanks for your contributions!

[robertonav20]

Any updates for this task? Can I help with something?

[squakez]

@robertonav20 you may have a look at the tasks in the issue description if you're willing to have a look at the implementation. A draft PR with some PoC may be useful to experiment and reason about the pros and cons.

[robertonav20]

Hi @squakez, I build and execute camel k operator and a camel quarkus application inside raspberry. I used docker buildx and specify the target platform.

I must modify makefile and dockerfile for camel-k-operator and use the correct java base image for camel quarkus application.

I apologize but I haven't strong skill with go language, I try to do my best. Can you give me an advice?

[squakez]

I think you can start having a look at the builder package which is the one in charge to take care for building images and pushing to registry.

[robertonav20]

Hi @squakez, i've tried to import BuildKit library like suggest @astefanutti inside the builder package but there is a problem of dependency

go: finding module for package github.com/go-logr/logr/funcr
github.com/apache/camel-k/pkg/builder imports
        github.com/moby/buildkit/client imports
        go.opentelemetry.io/otel/sdk/trace tested by
        go.opentelemetry.io/otel/sdk/trace.test imports
        github.com/go-logr/logr/funcr: module github.com/go-logr/logr@latest found (v1.2.3, replaced by github.com/go-logr/[email protected]), but does not contain package github.com/go-logr/logr/funcr

Now I'm trying to use buildah library instead buildKit, it's more friendly than buildkit, but it's the same I got error during build

# github.com/docker/docker/pkg/archive
/home/rob/go/pkg/mod/github.com/moby/[email protected]/pkg/archive/archive_unix.go:84:13: undefined: system.RunningInUserNS

you can find the my code here https://github.com/robertonav20/camel-k/... i will update you if i fix the problem

[robertonav20]

Hi, i have an update with this tasks, below the description for each point>

• Use a multi-architecture default Java base image

I use the graalvm base image in order to use for amd64/arm64 like this FROM --platform=$BUILDPLATFORM ghcr.io/graalvm/graalvm-ce:ol8-java11-22.1.0

I thinked to use quarkus-mandrel-base-image for arm64 but isn't available for now (https://github.com/quarkusio/quarkus-images/commit/21e25a3c3ae3e3f4a4bd02b1e76f6051c31729c0)

• Build and publish multi-architecture images: BuildKit could be a good candidate to ease the task of building images for all target architectures, as it automatically picks the right base image and creates the multi-architecture manifest

I modified the Makefile to use BuildKit with correct platform parameter in base of host's architecture

docker buildx rm --all-inactive --force
docker buildx create --append --name builder
docker buildx build --platform=$(IMAGE_TARGET_PLATFORM) -t $(CUSTOM_IMAGE):$(CUSTOM_VERSION) -f build/Dockerfile .

So, for the first part can i submit pr?

For the integration-kit image, I cannot find a solution to build and push a working image because there are a problem dependency (read the previous comment). Do you have an idea how to fix?

[squakez]

Thanks for running those experiments and reporting the feedback. About the multi-architecture Java image, I guess we may wait for that to be officially available. We will align to Quarkus platform, so, that one will become eventually available. As for the operator container, I think we cannot proceed so easily to change the make images. We depend on that for the release process and likely the docker buildx is not available in the github actions we use for the official release.

However, you can create a new action beside images, ie, images-arch which can be run manually by anyone who wants to run that. We may eventually use that once we have the rest of pieces in place. @oscerd @astefanutti @phantomjinx wdyt?

[oscerd]

It makes sense to me. +1.

[robertonav20]

So, @squakez suggest a command makes-arch to build for other platform, we can make the same thing with dockerfile because the base image isn't available. It makes sense if that image doesn't available soon, are you right? I don't know if can be a good idea, maybe there are other constraints for quarkus.

I checked the GitHub action, here https://github.com/marketplace/actions/build-and-push-docker-images can be found the official docker GitHub action of docker which support buildx, so I think it's possible use it.

Anyway, It's a pleasure help the community.... For now I try to build an integration kit multi arch with buildah strategy.

It's possible just update buildah version to 1.23.3 because the latest version support arm64 arch... Stay tuned I will give some feedback soon

[robertonav20]

As i say previously, i build and execute an integration kit inside rpi4 8gb with architecture arm64 with the use of buildah, below the modifies:

• Update buildah at 1.23.3 version
• Add the parameter platform

bud := []string{
        "buildah",
        "bud",
        "--storage-driver=vfs",
        "--platform",
        platforms.DefaultSpec().OS + "/" + platforms.DefaultSpec().Architecture + "/" + platforms.DefaultSpec().Variant,
        "--pull-always",
        "-f",
        "Dockerfile",
        "-t",
        task.Image,
        ".",
}

All image are available here https://hub.docker.com/repository/docker/robnav24241/camel-k operator and integration-kit

[squakez]

Nice work. Ideally those architecture values should be "parameter-ized" so that the user can provide that information somewhere (ie, in the IntegrationPlatform configuration).

[robertonav20]

@squakez sorry, I don't understand why should be parameterized? The integration kit are builded inside kubernetes so with architecture of the node amd64, arm64 etc.... this thing can't be modified by the user.

Can you describe me?

[squakez]

Yeah, I am thinking on a generic situation where you have multiple nodes in your cluster, and the operator is running on any non-ARM architecture. Then you have a special node in ARM architecture where you are willing to run the Integrations. In such case, having that value as a parameter would help.

[robertonav20]

Ok Great! Thanks! But you are limited by buildah architecture support, for others solutions it's necessary fix the dependency problem which i describe some comment ago

Anyway, i saw the IntegrationPlatform struct is based entirely by external attributes from [email protected], so i suggest add the Architeture field inside builderTrait and PublishTask like this

// The builder trait is internally used to determine the best strategy to
// build and configure IntegrationKits.
//
// +camel-k:trait=builder.
type builderTrait struct {
	BaseTrait `property:",squash"`
	// the architecture of image
	Architecture string `json:"baseImage,omitempty"`
	// Enable verbose logging on build components that support it (e.g. Kaniko build pod).
	Verbose *bool `property:"verbose" json:"verbose,omitempty"`
	// A list of properties to be provided to the build task
	Properties []string `property:"properties" json:"properties,omitempty"`
}

// PublishTask image publish configuration
type PublishTask struct {
	// can be useful to share info with other tasks
	ContextDir string `json:"contextDir,omitempty"`
	// Architecture of image
	Architecture string `json:"baseImage,omitempty"`
	// base image layer
	BaseImage string `json:"baseImage,omitempty"`
	// final image name
	Image string `json:"image,omitempty"`
	// where to publish the final image
	Registry RegistrySpec `json:"registry,omitempty"`
}

Then can be map Architecture when necessary just like this

	case v1.IntegrationPlatformBuildPublishStrategyBuildah:
		if t.Architecture == "" {
			t.Architecture = platforms.DefaultSpec().OS + "/" + platforms.DefaultSpec().Architecture + "/" + platforms.DefaultSpec().Variant
		}

		e.BuildTasks = append(e.BuildTasks, v1.Task{Buildah: &v1.BuildahTask{
			BaseTask: v1.BaseTask{
				Name: "buildah",
			},
			PublishTask: v1.PublishTask{
				Architecture: t.Architecture,
				Image:        getImageName(e),
				Registry:     e.Platform.Status.Build.Registry,
			},
			Verbose: t.Verbose,
		}})

PublishTask struct is shared by all builders for this reason i choosed her

@squakez @oscerd @astefanutti @phantomjinx do you agree?

[squakez]

You can have a look at some refactoring we've done recently here https://github.com/apache/camel-k/pull/3032 - We've introduced a generic PublishStrategyOptions in the IntegrationPlatform that can be used for any strategy extra option (we're using for Kaniko there). I think we could leverage that and from that onward cascade the parameter up to the Buildah Task. Ideally we don't want a specific parameter to slip into an abstract concept (such as the Builder). Also, feel free to create any draft PR and we can discuss over there any idea you have around this or other developments.

[robertonav20]

Ok @squakez, i understand your advice but i dont know how to use PublishStrategyOptions.

Can you give me and example to fill that structure from cmd?

[squakez]

Sure, you can have a look at #3032 where we used that for the "Kaniko" options. We may mimick that and include any "Buildah" option as well.

[robertonav20]

Hi @squakez, yes i have a look yesterday, below i describe the code:

Inside the package builder i add buildah.go file to manage buildah constants properties

package builder

const BuildahPlatform = "BuildahPlatform"

Here you can find the mapping to buildah builder

	case v1.IntegrationPlatformBuildPublishStrategyBuildah:
		var architecture string
		var found bool
		if architecture, found = e.Platform.Status.Build.PublishStrategyOptions[builder.BuildahPlatform]; !found {
			architecture = platforms.DefaultSpec().OS + "/" + platforms.DefaultSpec().Architecture + "/" + platforms.DefaultSpec().Variant
		}

		e.BuildTasks = append(e.BuildTasks, v1.Task{Buildah: &v1.BuildahTask{
			BaseTask: v1.BaseTask{
				Name: "buildah",
			},
			PublishTask: v1.PublishTask{
				Architecture: architecture,
				Image:        getImageName(e),
				Registry:     e.Platform.Status.Build.Registry,
			},
			Verbose: t.Verbose,
		}})

Then map architecture attribute to buildah command.

I see a difference between https://github.com/apache/camel-k/pull/3032 and this task, that task introduce a command parameter for install, but in our case we must manage the architecture of build for integration kit, so it's necessary introduce this parameter inside run command

My questions are:

• Where is the point of the code ? I suppose run.go inside pkg/cmd and here i can add the new buildah parameter (exmaple: buildah-platform)
• The variable platform inside run.go method createOrUpdateIntegration is empty, How to fill it ?
• How to check if that map PublishStrategyOptions contains that parameter?

I'm hope you understand my questions.

Thanks in advance

[squakez]

Any build configuration should be included into the IntegrationPlatform, not exposed in the kamel run command. I think you can extend the BuildahTask to include the new needed parameters https://github.com/apache/camel-k/blob/fe997b9ac1297884840ffb42bb41302bb5040ee5/pkg/apis/camel/v1/build_types.go#L107 and passing them from IntegrationPlatform.PublishStrategyOptions as we're doing for Kaniko settings. Once they are in the BuildahTask, you should be able to use them.

[robertonav20]

Ok thanks, i understand your advice and i fix the code just like this.

type BuildahTask struct {
	BaseTask    `json:",inline"`
	PublishTask `json:",inline"`
	// The platform of build image
	Platform string `json:"baseImage,omitempty"`
	// log more information
	Verbose *bool `json:"verbose,omitempty"`
}

case v1.IntegrationPlatformBuildPublishStrategyBuildah:
	var platform string
	var found bool
	if platform, found = e.Platform.Status.Build.PublishStrategyOptions[builder.BuildahPlatform]; !found {
		platform = platforms.DefaultSpec().OS + "/" + platforms.DefaultSpec().Architecture + "/" + platforms.DefaultSpec().Variant
	}

	e.BuildTasks = append(e.BuildTasks, v1.Task{Buildah: &v1.BuildahTask{
		Platform: platform,
		BaseTask: v1.BaseTask{
			Name: "buildah",
		},
		PublishTask: v1.PublishTask{
			Image:    getImageName(e),
			Registry: e.Platform.Status.Build.Registry,
		},
		Verbose: t.Verbose,
	}})

In this way you can manage the platform parameter if you want, just fill PublishStrategyOptions with the correct paramater from outside.

But, i don't understand how to works PublishStrategyOptions, for kaniko i found this code inside install.go

kanikoBuildCacheFlag := cobraCmd.Flags().Lookup("kaniko-build-cache")
if kanikoBuildCacheFlag.Changed {
	platform.Spec.Build.PublishStrategyOptions[builder.KanikoBuildCacheEnabled] = strconv.FormatBool(o.KanikoBuildCache)
}

If i undertstand well, with install command you can pass kaniko-build-cache parameter and to set the PublishStrategyOptions, right? There is another way to fill PublishStrategyOptions? How do you thing to manage the platform parameter for buildah?

Thanks in advance! 😄

[squakez]

Nice, that fits perfectly into the design. I think those parameters must be provided by the user editing the IntegrationPlatform (manually) ie kubectl edit ip. A nice addition would be to transform that kaniko-build-cache into something generic, so we can manage any PublishStrategyOptions option. I'm creating a follow up issue for that. For now, I guess that the manual editing of the IntegrationPlatform would be enough for the scope.

[robertonav20]

Ok great! So you leave to the user edit the manifest of integration platform under Camel K CRD.

Can i submit a PR?

[squakez]

Sure, it's more than welcome.

[robertonav20]

You can find here https://github.com/apache/camel-k/pull/3309 the pr

[barius]

I'm able to build a arm64 camel-k-operator image by first creating a arm64 quarkus-mandel image, then base camel-k on this. However, integration kits built in this arm64 operator is still amd64/x64. I'm running Docker Desktop on an M1 Macbook, camel-k 1.9.1.

Looked into camel-k builds and saw the Spectrum builder. Don't know if it supports arm64. Is there any way I can get a arm64 image? Maybe after the above PR is merged? Or is there anyway I can help with that?

[robertonav20]

Hi @barius to build integration kits arm64 it's necessary use buildah as build-strategy you can follow the pr open from this ticket

[barius]

@robertonav20 Okay I'll try it! Good work!

[barius]

Confirm that the PR works! Now my integrations are arm64, yeah! Saw ~4x performance on my mac :) I'll still using my local arm64 quarkus-mandrel image though, didn't find the manual command to build kamel operator to support arm architecture :(

[robertonav20]

Ok great you can find the full guide here https://github.com/robertonav20/camel-k/blob/main/docs/modules/ROOT/pages/configuration/multi-architecture.adoc