beam
beam copied to clipboard
[Task]: Migrate from archived kaniko for building SDK container images
What needs to happen?
GoogleContainerTools/kaniko was recently publicly archived. https://github.com/GoogleContainerTools/kaniko/pull/3502
I remembered that Apache Beam uses kaniko for its SDK builds, so I checked the repository and confirmed it is still being used in one place for building the SDK container image: https://github.com/apache/beam/blob/v2.65.0/sdks/python/apache_beam/runners/portability/sdk_container_builder.py#L250-L252
While I haven't heard anything about the Kaniko container repository being deleted, it might be a good idea to migrate to a different tool sooner rather than later IF the pipeline is still in use.
For reference, the kaniko community has suggested podman, buildah, or buildKit as potential migration paths. Both are tools for building OCI images and seem like viable alternatives to kaniko. https://github.com/GoogleContainerTools/kaniko/issues/3348
BuildKit provides a build environment within a container image (Note: it requires privileged access). https://hub.docker.com/r/moby/buildkit
Next steps:
- Confirm if this pipeline is still actively in use.
- Outline the requirements for the replacement tool.
Issue Priority
Priority: 2 (default / most normal work should be filed as P2)
Issue Components
- [ ] Component: Python SDK
- [ ] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam YAML
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [x] Component: Infrastructure
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner