Move from dockerhub to quay.io (and consider we got an 800$ charge!)

[yuvipanda]

@choldgraf's card is apparently attached to the jupyterhub org on dockerhub, and it just got an 800$ charge!

This is not supposed to happen, as we have 'sponsored oss' status (I'm not sure who got that for us, but THANK YOU to whoever that was).

While I guess this charge is a mistake on their part, it at least feels sufficient motivation for me to move us completely over to quay.io. We already moved repo2docker (https://github.com/jupyterhub/repo2docker/issues/1076#issuecomment-920700356), and while I can't find the issue right now, the 'sponsored OSS' status was why we didn't move everything else. Time to move it now I think.

Repos to move

• [x] z2jh https://github.com/jupyterhub/zero-to-jupyterhub-k8s/pull/3254
• [x] jupyterhub https://github.com/jupyterhub/jupyterhub/pull/4612
• [x] configurable-http-proxy https://github.com/jupyterhub/configurable-http-proxy/pull/508
• [x] repo2docker
• [x] docker-image-cleaner
• [x] binderhub https://github.com/jupyterhub/binderhub/pull/1789
• [x] kubespawner default https://github.com/jupyterhub/kubespawner/pull/800
• [x] dockerspawner default https://github.com/jupyterhub/dockerspawner/pull/504
• [ ] mybinder.org-deploy

[sgibson91]

• https://github.com/jupyterhub/team-compass/issues/559

Cc @consideRatio and @mathbunnyru who lead that work

[consideRatio]

Was @choldgraf's card charged and is it now removed so it can't be charged going onards?

[minrk]

👍 to moving everything to quay and leaving docker. I don't think anything's been charged, but let's make sure we don't give them a chance to charge in the future.

[mathbunnyru]

I see this charge on the jupyter DockerHub account as well (also unsuccessful), so we will also have to move it.

I think I understand what's going on.

https://www.docker.com/community/open-source/application/

Core contributors of your project namespace get access to a year-long Docker Team subscription

https://github.com/jupyterhub/team-compass/issues/559#issuecomment-1281606540

And, based on this message, it was exactly one year ago when the upgrade to Docker Team happened. I didn't think that it would only last a year, and then Docker would charge us for the next year - doesn't really sound like a "Docker-Sponsored Open Source Program", more like "Docker Team One Year Trial". I missed it a year ago, I'm so sorry. Hope no one was charged (at least DockerHub says the payment failed, which is good).

[mathbunnyru]

👍 to moving the jupyter account to quay as well.

I will work on jupyter/docker-stacks move this or next week (I suppose it will be the most difficult repo to move, so it will take a bit of time).

[manics]

Could we push the important images (mostly jupyterhub) to both docker hub and quay.io for a release cycle or two, but update all docs, config and example repos to use quay.io? That'll ensure anyone pulling/checking latest will still get updates in the short term.

[choldgraf]

I've sent a support request to Docker asking them WTF is going on. I'll let you all know if I hear back. I definitely agree we should just leave the platform because this is too much hassle if there are alternatives that work just as well.

I'd suggest we:

• Try to get one more year of service.
• But plan to migrate off ASAP.
• Use that year as buffer to give us some breathing room.
• Cancel service as soon as we're migrated off.

[manics]

I don't think we need another year of service (though there's no harm if someone wants to apply). It means people may hit the rate limits when pulling, but hopefully that means they'll look at the readme and see that they should move to quay.io

[yuvipanda]

I built a small tool that'll migrate some but not all tagged images across the registries! https://github.com/yuvipanda/move-off-dockerhub.

I've moved k8s-hub image's released versions only over to quay.io now, so if people do end up being throttled by dockerhub, they can simply point to quay.io and get the exact same image (as it's copied).

I'll do this for all the z2jh and chp images, and the JupyterHub ones as well.

@manics we can do this in reverse for a bit to dockerhub as well.

[mathbunnyru]

@yuvipanda please, change the email of the jupyterhub quay.io account to [email protected]. It will only change the icon for the organization (and the images).

[mathbunnyru]

After 2 days of work with ~25 commits, I've finally switched http://github.com/jupyter/docker-stacks/ to Quay.io. An example: https://quay.io/repository/jupyter/base-notebook Only 1 of these 25 commits is actually renaming (mostly, sed-like), all the others are gonna stay if we ever decide to change the registry again.

Cool features work fine too - automatic registry description update, docker manifest to create multi-arch images, testing contributed recipes, and automatic wiki update.

Nice features of Quay.io I noticed:

• the tags page visually shows which tags have the same content. As we push many tags for one image, this is useful
• Security scan is built-in for free on the same page (remote Docker Scout is not free, as far as I know)
• Having usage logs for free is nice as well
• the tag history page shows when a tag was moved - it might be helpful during debugging (and lots of our tags are moving)

Things I don't like/don't work:

• The first opening of Quay.io where you need to set which cookies you allow is a banner. It doesn't look/feel nice and is so slow.
• I don't think there are badges with the number of stars/pulls/size of an image (currently, I use shields.io with Docker Hub values)

[choldgraf]

I went ahead and applied for another year of Open Source account status with DockerHub, and got an e-mail that it was approved today. Could somebody confirm that we do in-fact have this status now?

[consideRatio]

Not yet, but I recall that this wasn't reflected instantly last time either.

[manics]

I received an email yesterday:

Hello jupyterhub!

We’re sad to see you go. This email confirms that your Docker Team - Annual subscription for your account has been canceled.

🤷🏽

[consideRatio]

Dockerhub status looks like this:

I received this email:

[yuvipanda]

At least once the JupyterHub image moves are complete, we should go to all the READMEs in the jupyterhub dockerhub and write them out as redirects.

[yuvipanda]

Ok, everything that needs a PR now has a PR :) Reviews would be appreciated!

[mathbunnyru]

Ok, everything that needs a PR now has a PR :) Reviews would be appreciated!

Done. Please, ping me in PRs if I missed something.

[Carreau]

I would also suggest contact the Jupyter Media strategy team to have them broadcast on various channels that you are moving to Quay

[chris-bateman]

This is obviously still a WIP but this just caused issues on a few of our pipelines. There really needs to be a widespread announcement and migration guide put out.

[minrk]

@chris-bateman we're working on it, but I'm curious how this caused any issues for you. It shouldn't have affected any existing images as far as I can tell.

[mathbunnyru]

We'll have to search our repos for read-usages of Docker images and update them as well. An example: jupyterhub/jupyterhub-deploy-docker: https://github.com/jupyterhub/jupyterhub-deploy-docker/pull/133

[choldgraf]

I wonder if the issues that @chris-bateman ran into were because of the momentary lapse in our DockerHub account status, because of the rejected credit card payment the tried to make.

Either way, right now we should just have another year of DockerHub's service, so nothing should be changed for the time being. Agreed that we should communicate the decision to migrate (within our capacity to sustainably do this...)

[chris-bateman]

I'm sorry for the lack of detail in my above comment. The issue did turn out to be something else related to this PR https://github.com/jupyter/docker-stacks/pull/2006 Resolved on our end due to some hard coded assumptions. But we did see a clone issue of the docker image during one of our builds. It might have been due to the above DockerHub status but I can't comment on the exact cause.

[mathbunnyru]

I applied for open source program for jupyter Docker Hub organization a few days ago (https://github.com/jupyterhub/team-compass/issues/694) and today received an approval.

I see that our images still show a Sponsored OSS badge, but we're on a free Docker Free Team plan.

[yuvipanda]

Great to hear that both jupyter and jupyterhub are still part of the program.

I do think we should still migrate the 'official' source to be quay.io - so we don't have to do this every year, and at the risk that at some point soon we'll just get back a 'no'

[yuvipanda]

ok, everything has moved now! \o/

Next:

• [ ] Make a note in each dockerhub description that the image has moved to quay.io
• [ ] Copy the descriptions for these images from dockerhub to quay.io

[yuvipanda]

Spoke too soon - mybinder.org-deploy also needs to move, but there are no external consumers of that image.

[mathbunnyru]

ok, everything has moved now! \o/

Next:

• [ ] Make a note in each dockerhub description that the image has moved to quay.io
• [ ] Copy the descriptions for these images from dockerhub to quay.io

@yuvipanda I highly recommend not doing it manually, because we might delete/add some badges in the READMEs and sometimes text is changed. There is a update-container-description-action. It works with both Docker Hub and Quay.io with slightly different setup. Here is an example how Jupyter Docker Stacks use it: https://github.com/jupyter/docker-stacks/blob/main/.github/workflows/registry-overviews.yml

[miraculixx]

ok, everything has moved now! \o/

I appreciate the move to quay.io and everyone's effort to make this happen. I just came across this update by chance, noticing the reference to quay.io in the docs, and I'm wondering about the scope of the migration strategy.

Specifically I noticed that, as of writing this, only the images tagged with python-3.11 are available from the quay.io repo. For example, docker.io/jupyter/datascience-notebook:python-3.10.11 does not seem to be available on quay.io.

Is this intended? If yes, what is the policy going forward as to which Python versions will be built/provided on quay.io v.s. docker hub, or will the latter be shut down, that is all previous images would be removed? Thank you