mediacloud
Move from GCR back to Docker Hub
Hey James,
Can you move Docker images from Google Cloud Registry back to Docker Hub?
We used to use Docker Hub for storing all of our Docker images. Then at some point Docker remembered that they're a business and decided to rate limit public images thus breaking our deploys. Their premium plan was $35 but it required managing authentication credentials on the host so I figured I'll just migrate everything to GCR as it would be cheaper.
It turned out that GCR isn't that cheap at all as the data outgress costs us $200+/month. So, I think we should just give up and move back to Docker Hub.
The task involves:
- [x] Signing up with Docker Hub, creating an organization that we'll all share (think up of a good short and sensible username; if you need to choose one, go for short over the sensible), adding our card to the organization (do note that this is a public issue so don't post credentials and such here), buying a premium plan or whatever they call it for the organization, sending the invoice to Lorrie
- [ ] Updating
./dev/scripts and the rest of the codebase to pull / build / push to the new organization on Docker Hub - [ ] Figuring out a way to securely push Docker Hub authentication credentials to our servers (otherwise if Docker doesn't authenticate, we'll get rate limited in the middle of a deployment)
- [ ] Letting our partners at CfA know that they might want to sign up for Docker Hub premium too if they don't want to get rate limited
There's a commit somewhere making the move from dockermediacloud (too long of an username!) on Docker Hub to GCR, that might serve as a good reference for what needs to be done.
Or maybe there are better alternatives to Docker Hub; what we're looking for is paying a fixed price per month for unlimited public pushes and pulls.
https://quay.io/plans/ also worth looking at, although they do rate limiting and I'm not quite sure what that limit is.
Steps ahead:
- [ ] Encrypt Docker credentials and get them on our servers via Ansible Vault so that they're automatically authenticated to Docker Hub
- [x] Make a user (e.g.
mc_systems, email[email protected]) in our Docker Hub org for the servers to use - [ ] Make sure pull rate-limiting only applies to users in our org (i.e. public, non-MC users don't count towards the limit)
- [ ] This authentication scheme must also apply to Docker Swarm
- [x] Make sure Docker monthly invoices are automatically forwarded to Lorrie
- [x] Update image URLs in https://github.com/mediacloud/production-docker-config/blob/master/docker-compose.yml
Make sure pull rate-limiting only applies to users in our org (i.e. public, non-MC users don't count towards the limit)
It's more like "make sure that we get our $35 worth, i.e. we don't get rate limited by authenticating via an user added to the organization".