apache
Publish Docker images to Apache repo in DockerHub
Is your feature request related to a problem or challenge? Please describe what you are trying to do. I would like users to be able to download Ballista docker images instead of having to build them themselves.
Describe the solution you'd like Publish Docker images to Apache repo in DockerHub. Based this on how Spark does this?
Describe alternatives you've considered None
Additional context None
I 1000% support this. Every other major software has pre-packaged docker image you can just pull and run. Pushing it to docker hub is easy, the harder part is that we need an apache account with credentials, then to put that secret in CI. Even if it is in CI, I don't know how we could prevent it leaking from PRs, but I could look into it.
I filed a ticket with INFRA: https://issues.apache.org/jira/browse/INFRA-23709
It looks like the easiest path is to use GitHub Container Registry. This doesn't rule out looking at publishing to Dockerhub in the future, but I recommend that we try this first.
We can use GITHUB_TOKEN within GitHub actions for authentication.
https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry
My understanding is that the Docker image URLs would be
ghcr.io/apache/arrow-ballista-scheduler
ghcr.io/apache/arrow-ballista-executor
We may also want to publish some client images to make it easier for users who don't already have appropriate Rust or Python setups.
ghcr.io/apache/arrow-ballista-cli
ghcr.io/apache/arrow-ballista-python
Here is the updated proposal for how this will work:
We can use a GitHub action to push to ghcr.io when we push release tags to the repo.
When we tag the repo with a release candidate tag e.g. 0.10.0-rc1, we start a vote that covers both the source release (tarball) and also the published Docker image release candidate.
Once the vote passes, we push the 0.10.0 tag, and this will cause the image to be published with the 0.10.0 version, and we can then announce that as an official release.
Thanks @avantgardnerio. I assume that this is too much to try and get into 0.10.0 (planning on releasing this in the next few days), so I have tagged this for 0.11.0. Let me know if do you want to try and do this for 0.10.0 (we an always delay the release to accommodate this).
Adding for reference: currently blocked by https://issues.apache.org/jira/browse/INFRA-23897