Prepare for end of gke.mybinder.org

[minrk]

Google cloud credits are ending at the end of this month. Some tasks to track dealing with this:

• [x] https://github.com/jupyterhub/team-compass/issues/646
  • Decide if this is enough funds to keep the "core" of Binder going for a while
• [x] https://github.com/jupyterhub/team-compass/issues/643
• [x] allow mybinder.org to deploy just the federation redirect without binderhub, so we can keep mybinder.org up without gke.mybinder.org (@minrk)
• [x] reduce image cache size in GKE to start reducing costs (@minrk)
• [x] https://github.com/jupyterhub/mybinder.org-deploy/issues/2556
• [x] turn off gke.mybinder.org on April 30, and make another federation member 'prime', leaving reduced self-funded project in place for mybinder.org

[choldgraf]

How much of our current capacity can we re-direct to OVH? If we assume that no other services are spun up (e.g., on AWS), what is the total expected drop in capacity?

I'll ask NumFocus about our Binder funds now and see what they say.

[manics]

As previously mentioned I've got some Terraform for deploying an AWS EKS cluster, and I've deployed a BinderHub example using ECR. I haven't looked at scaling it up, but it should be enough to get started. Happy to help once we've got an admin access to an AWS account.

[choldgraf]

@manics I've opened an issue to track deploying on AWS at the link below (and added it above), wanna discuss and track there?

• https://github.com/jupyterhub/mybinder.org-deploy/issues/2556

[minrk]

OVH hasn't been as robust, I think largely due to issues in the Harbor registry, which may be fixed (https://github.com/jupyterhub/mybinder.org-deploy/issues/2514#issuecomment-1506693566). It seems fairly stable now with ~100 users, and ought to cover everything we have on GKE (funding-wise), but it seems to struggle with larger load. I've bumped capacity to 200 in https://github.com/jupyterhub/mybinder.org-deploy/pull/2557 so we can see how that goes.

Current typical peak users is ~500, so it's a pretty big drop.

We've also had some disk-filling-up issues, which can be solved short-term by recycling nodes, but needs to be addressed somehow.

Two possible ways we can address capacity in OVH, since funding isn't short right now:

• add another identical OVH cluster, if the issue is in the cluster itself. THis should be relatively low maintenance, as it would necessarily be identical, other than URLs and credentials.
• move registry to quay.io, like @sgibson91 is doing for AWS, if the registry is indeed the core issue and it can't keep up. This may be an issue generally, because the official max size of an OVH private registry is 5TB, and the GKE registry is 15 TiB.

[sgibson91]

Also, I have access to Scott H's AWS account that I have been testing https://github.com/jupyterhub/mybinder.org-deploy/pull/2467 on

Update: Pangeo have actually pulled down their AWS deployments, announced in this Discourse https://discourse.pangeo.io/t/aws-pangeo-jupyterhubs-to-shut-down-friday-march-17/3228 Stating the reason "Credits have depleted" - so potentially Scott's account is no longer the best place to be testing/host the new AWS deployment

[minrk]

I realized another significant change that we may have to deal with more directly: staging is also run on GCP. We probably need to shut this down as well to save costs, though it's drastically less costly than running prod.

We could also launch a staging cluster on OVH, since we have room there, but I think it can't run everything it currently does (analytics publisher, matomo). Essentially, everything that we do want to keep on GKE after shutting down gke.mybinder.org because it isn't easily relocated.

Perhaps notably, we don't actually run tests for these components, so the staging tests will pass just fine if it's only a single federation member and not a full staging 'federation'

[choldgraf]

Do we have an estimate of how much staging would cost on GKE?

[minrk]

Cost calculator suggests it should be around ~$250 for the current cluster and cloud sql (I'm assuming registry costs are negligible, since registry costs are ~all pulls, and staging hardly does any pulls). I could probably save ~$100 further by using a smaller cluster (e.g. 4 cores with 200GB disks or even less). I think I can comfortably fit it in $200/month, and if we push it, fit it in $100.

[minrk]

Without the prod BinderHub, I think I can probably aim for a fairly similar cost for running only the non-Hub parts. Those are all pretty low-resource.

We could even run both staging and the prod entrypoint on the same cluster in different namespaces. That's probably not the best idea, though.

[choldgraf]

So is this right then:

• Non-hub core infrastructure: ~$100-200/mo
• Staging hub infrastructure: ~$100-200/mo

Or around $2,400 - $4,800 / yr ?

And then user clusters are all fully modular?

As a stop-gap, maybe this is what we use the funds that @arnim mentioned from the GESIS work that we are doing (especially if we could get it on the lower end). That'd keep the core going for at least a year.

[minrk]

I believe we can get to that, yes.

[choldgraf]

In that case, my feeling is that this should be our target for "core funding" that we pay for with cash rather than bunches of credits, and try to run all of this on GKE. Then we can boost capacity of mybinder.org in a more opportunistic way, where we just spin up a cluster somewhere and connect it to the federation.

AKA, credits can power user capacity, but we should have $$ to cover at least 1 year's worth of core operation, or something like that.

[arnim]

Ensuring that at least the core-infrastructure can continue to operate is important. As stop-gap we can use for this 5k USD from the GESIS-2i2c contract.

[rgaiacs]

I'm trying to complete the setup of BinderHub on GESIS premises. Trying to navigate the sea of Helm chart.

@minrk and @sgibson91 do you have 1h to help me with online pair programming debugging next week? Maybe Tuesday?

[minrk]

Yes, I think I can do Tuesday. Any time between 10-15 CEST.

[rgaiacs]

Thanks @minrk. I sent an invitation to your email.

[manics]

We could even run both staging and the prod entrypoint on the same cluster in different namespaces. That's probably not the best idea, though.

What are your main concerns with running staging on the same cluster? It might not be "best-practice" but it'd save $74/month, with more savings from not paying to run an additional set of K8s management pods.

The Prod cluster has a 2TB volume for prometheus which is around $80/month. I presume this was originally setup for long term storage of data- in which case we could save a lot by archiving it to an object store ($40/month for 2TB, even cheaper for the long-term archive class)?

However if this was the intention we're not doing it! We're only storing the metrics for 30 days, and we're using < 50G:

$ kubectl -nprod exec $(kubectl -nprod get pod -lapp=prometheus -o name) -c prometheus-server -- df -h /data
Filesystem                Size      Used Available Use% Mounted on
/dev/sdc                  1.9T     46.5G      1.9T   2% /data

https://github.com/jupyterhub/mybinder.org-deploy/blob/5a710c1159b18fadc56402b27aaa853626f1fb21/config/prod.yaml#L121-L125

[minrk]

It might not be "best-practice" but it'd save $74/month, with more savings from not paying to run an additional set of K8s management pods.

It's mainly the 'best practice' bit. We do deploy one cluster-wide resource (cert-manager), which would become shared. So any upgrades related to that could only be deployed to prod, not tested on staging before updating prod. It actually wouldn't save any on management costs, because staging is a zonal cluster, which costs $0, while prod is regional (benefits from zero downtime upgrades). It would mostly save on node costs, which is around $100-$150 for staging.

The Prod cluster has a 2TB volume for prometheus which is around $80/month. I

I think prod prometheus was never meant to be long-term. With more than a couple months of data, performance gets too slow and too memory intensive. It's also going to dramatically reduce its needs when the prod binderhub stops, because there will be no launch events or running notebooks, which is the main resource prometheus tracks.

[minrk]

The simplest way to reduce the cost of prod prometheus is to reduce the size and just delete the current prometheus data and deploy a new PV. This should be fine, since prometheus data is by definition ephemeral, but it might be good to do that sooner rather than later, so we have some data from before the transition.

[minrk]

What do folks think of deploying staging in a namespace on the prod cluster? I think this would likely save ~$100/month, but it would increase the risk of downtime on a bad deploy to staging.

[manics]

What do folks think of deploying staging in a namespace on the prod cluster?

For the cost saving I think that's justified. Ultimately we're asking are we better off spending $100/month on staging to reduce the chance of an outage (do we know how often that happens in practice?), or to spend it on ensuring the production cluster keeps running for longer. Given the current situation I think saving the $100 is worth it, we can always switch back later.

We can reduce the risk by enforcing a namespace resource limit https://kubernetes.io/docs/tasks/administer-cluster/manage-resources/quota-memory-cpu-namespace/

[minrk]

Having looked at it, I think the effort of merging staging into prod safely (things like ingress controllers, node exporters, etc. might have issues), plus the fact that I think we can trial running staging on spot instances for a total of ~$20/month, makes the effort/cost not worth merging them this week. We can look at it later, though, since I think it would be on the order of saving $1/day.

[choldgraf]

2i2c is ready to use funds to pay for the core Binder infrastructure

We discussed this in a team meeting yesterday and agreed that 2i2c can use its funds to power the "core" mybinder.org infrastructure, and pass through this cost to GESIS up to $5,000.

Here are the things I think we need to do:

• Set up the "core infrastructure" project/cluster/etc
• Connect 2i2c's GCP billing account with mybinder.org's "core infrastructure" project
• 2i2c needs some kind of process to track the spend and send a monthly invoice to GESIS

@minrk can you provide guidance on the proper projects to look for to connect 2i2c's credit card?

I'd appreciate if we could mention this at least in a blog post or something, so that this arrangement and the work going into it isn't invisible to the outside world (for both GESIS and for 2i2c). Are others OK with that?

[manics]

Sounds great. In addition to the blog we could give the funders more prominence on https://mybinder.readthedocs.io/ ?

For the avoidance of doubt, it'd be good to mention somewhere that the core funding provided by GESIS (for GKE) is in addition to the compute provided by GESIS (https://notebooks.gesis.org/binder/) as I was quite confused at first- I thought the core infra was moving to GESIS 😄

[choldgraf]

Yep good point, maybe we should make all of this explicit in those docs. e.g. a page that breaks it down by:

• Core Binder infrastructure
  • funding: (GESIS + 2i2c)
  • infrastructure: (GKE, run by Binder team)
• Federation member: GESIS
  • funding: GESIS
  • infrastructure: GESIS
• Federation member: Future AWS cluster
  • funding: Some generous person
  • infrastructure: AWS, run by Binder team

and so on?

[minrk]

That sounds like a great plan! It's all in project binderhub-288415, where everything has been this year. I believe this is currently already backed by @choldgraf's 2i2c card (billing account: gke.mybinder.org) once the credits stop (it's labeled "Ramp cloud card", I don't know what that is). I set up a separate 'mybinder-gke' project last funding-pressure round, and that's where I'd like to deploy the next GKE cluster, if there's ever funding for that, to more easily separate the federation member from the core infrastructure, and more easily bring cluster costs all the way to zero. There is nothing there now, though.

I'll have to keep an eye on costs over the weekend and into next week, but looking at what we're doing currently, I think around $300/month is within reach.

[choldgraf]

Ah perfect, OK a few responses:

• In this case, 2i2c's card will be auto-charged once the credits run out, so we are in good shape there.
• The current project is called binderhub. I suggest we rename to mybinder-core. This would be a nice complement to mybinder-gke.
• I'll make a docs PR to document that this is happening.

Any objections to either of those? I don't think that there are unintended consequences if we rename a project, since it still keeps its unique ID, but wanted to double check.

[minrk]

A blog post is a great idea, maybe aim for next week when some dust has settled?

core infrastructure project

• has 'prod' cluster serving mybinder.org (no binderhub, no image cache anymore) (~$200/mo)
• very cheap 'staging' cluster for testing deployments (~$25-50/mo)
• SQL for matomo analytics (~$65/mo)
• events archive (very cheap, but not sure exactly. Possibly actually zero)

Costs are unclear for now for logs, which I expect will go to zero without binderhub deployed (first 50GB is free and we used 400GB for $175 last month). I think we can do some log exclusions to make sure that happens (nginx ingress logs are pretty verbose and we don't need all of them)

[minrk]

Any objections to either of those? I don't think that there are unintended consequences if we rename a project, since it still keeps its unique ID, but wanted to double check.

Sounds good! Yes, the name is used exclusively for display purposes. You can't actually use it anywhere, it's always the immutable project ID that makes things work.