Development task: Tools for managing complex JupyterHub deployments

[lheagy]

• [ ] We will work on improving HubPloy and similar tooling to help continuous deployment be adoptable by a wide community of deployment maintainers, including those who maintain Pangeo deployments.
• [ ] We will also work to create a community of practice around managing complex JupyterHub deployments so that the number of resources available to administrators can continue to grow and best practices can be spread.
• [ ] These tools will be validated at some of the nation’s HPC facilities (e.g. Cheyenne, NESRC)

[lheagy]

I know @consideRatio has been making some progress here! Do you have relevant issues you could point to that are tracking a few more of the specifics?

[consideRatio]

Thanks for the nudge @lheagy! Here is an overview of some work done regarding tooling to manage JupyterHub deployments.

Work on tooling

I may have missed some projects,

hubploy

Hubploy is tailored for Helm chart users. It helps users manage secret configuration and a custom docker images for user environments and makes it a bit easier to setup GitOps for automated deployments.

My main contributions to hubploy stems from trying to deploy to AWS which led to fixing some pain points and refactoring some code to enable myself and others to contribute more easily in the future. Here are the PRs merged by me.

chartpress - 1.0 released!

Chartpress is tailored for Helm chart developers. It is used by z2jh, binderhub, and daskhub for example.

I have worked to make a 1.0.0 release and is happy about its current state with sufficient tests, docs, and some relevant features added. Here are PRs merged by me.

jupyterhub/action-k3s-helm - 1.0 released!

action-k3s-helm is a GitHub action, a building block to setup a k8s cluster in a GitHub CI environment. Its provides value by enabling a GitHub CI system to easily get a k8s cluster up and running. It can be a cheap version of having a dedicated staging cluster to validate changes.

@manics put in effort to making this and I'm very happy about it! We have now released 1.0 and done work to make it somewhat easy to maintain going onwards. Here are PRs by me.

full_namespace_report bash script - to become a GitHub action

With a k8s cluster part of a CI system, it's often very relevant to extract information of what goes wrong in case that happens. This script is dedicated to providing relevant information to debug such things currently part of the z2jh repo, but I look to extract it to something that makes us able to run this as a GitHub action instead.

This will be useful for any end user of a Helm chart with a CI system to deploy as well as any developers of Helm charts for the test suite, such as in z2jh, binderhub, and daskhub.

Example from a GitHub Job that failed and contain a full namespace report:

Z2JH vulnerability scans and patching - so end users don't have to

Some deployments with a lot of security focus wants to see known vulnerabilities in software patched quickly. I setup a system to make Z2JH automatically patch known and patchable knwon vulnerabilities in the Helm chart's associated images. This allows for end users to just update to the latest version if needed rather than building their own updated image with the vulnerability patched.

This could probably be extracted to a GitHub action, but I consider it a bit premature at this point.

• The GitHub Workfow doing the scans and submitting the PRs
• Example PR automatically submitted (with the GitHub Job executed to create it)

[choldgraf]

This is an amazing update, thanks @consideRatio

[lheagy]

Thanks so much @consideRatio!! This is an awesome update!