nebari [BUG] - Docker containers using Docker Schema 1

Describe the bug

Containerd and Docker Schema 1 will be end of life next year, and we won't be able to run Nebari on K8s 1.33 using docker schema 1 containers. We need to check our images and make sure that none of them are using Docker Schema 1.

Expected behavior

Nebari can run on K8s 1.33

OS and architecture in which you are running Nebari

GKE

How to Reproduce the problem?

We got notified by GCP of this pending issue

Command output

Versions and dependencies used.

No response

Compute environment

None

Integrations

No response

Anything else?

No response

Sep 18 '25 09:09 dcmcand

affiliated discussion #2873

Sep 18 '25 13:09 viniciusdc

Using the below script on a running GCP Nebari cluster I confirmed the only schema 1 image is the NFS one that is discussed in @viniciusdc 's linked comment. This issue is blocked by #2873

#!/bin/bash

# This script checks the manifest schema version for all running container images
# in a Kubernetes cluster across all namespaces.
#
# Dependencies: kubectl, skopeo, jq

# 1. Check for required command-line tools
for cmd in kubectl skopeo jq; do
  if ! command -v $cmd &> /dev/null; then
    echo "Error: Required command '$cmd' is not installed." >&2
    echo "Please install it and ensure it's in your PATH to continue." >&2
    exit 1
  fi
done

# 2. Get a unique list of all running container images
echo "🔍 Finding unique container images from all running pods..."
images=$(kubectl get pods --all-namespaces -o jsonpath='{range .items[*].spec.containers[*]}{.image}{"\n"}{end}' | sort -u)

if [ -z "$images" ]; then
  echo "No running container images found in the cluster."
  exit 0
fi

echo "✅ Found images. Inspecting schema version for each..."
echo "-------------------------------------------------------------"

# 3. Loop through each image and inspect its schema
for image in $images; do
  # Trim the SHA digest if it exists, as skopeo needs to inspect the tag.
  trimmed_image=${image%%@*}

  # Use the trimmed image name for inspection.
  # Errors are redirected to /dev/null to keep the output clean.
  # The '|| true' ensures the script doesn't exit if skopeo fails for one image.
  schema_version=$(skopeo inspect --raw "docker://$trimmed_image" 2>/dev/null | jq '.schemaVersion' || true)

  # The original full image name is printed for clarity in the report.
  if [[ -n "$schema_version" && "$schema_version" != "null" ]]; then
    # Check if the schema version is the legacy v1
    if [[ "$schema_version" == "1" ]]; then
      # Print in red for high visibility
      echo -e "Image: $image"
      echo -e "└── 🚨 \033[0;31mSchema Version: $schema_version (Legacy Schema 1 - NEEDS UPDATE)\033[0m"
    else
      # Print in green for modern schemas
      echo -e "Image: $image"
      echo -e "└── ✅ \033[0;32mSchema Version: $schema_version\033[0m"
    fi
  else
    # Handle cases where inspection fails (e.g., private registry, invalid image name)
    echo -e "Image: $image"
    echo -e "└── ⚠️ \033[0;33mCould not determine schema version. (Check registry access or image name)\033[0m"
  fi
done

echo "-------------------------------------------------------------"
echo "Scan complete."

And the output

🔍 Finding unique container images from all running pods...
✅ Found images. Inspecting schema version for each...
-------------------------------------------------------------
Image: docker.io/bitnami/redis:7.0.4-debian-11-r4
└── ✅ Schema Version: 2
Image: docker.io/bitnamilegacy/minio:2021.4.22
└── ✅ Schema Version: 2
Image: docker.io/bitnamilegacy/postgresql:11.14.0
└── ✅ Schema Version: 2
Image: docker.io/grafana/grafana:10.4.1
└── ✅ Schema Version: 2
Image: docker.io/grafana/loki-canary:2.9.4
└── ✅ Schema Version: 2
Image: docker.io/grafana/loki:2.9.4
└── ✅ Schema Version: 2
Image: docker.io/grafana/promtail:2.9.3
└── ✅ Schema Version: 2
Image: docker.io/nginxinc/nginx-unprivileged:1.24-alpine
└── ✅ Schema Version: 2
Image: gcr.io/google_containers/volume-nfs:0.8
└── 🚨 Schema Version: 1 (Legacy Schema 1 - NEEDS UPDATE)
Image: ghcr.io/dask/dask-gateway-server:2022.4.0
└── ✅ Schema Version: 2
Image: kiwigrid/k8s-sidecar:1.24.3
└── ✅ Schema Version: 2
Image: maxisme/traefik-forward-auth:sha-a98e568
└── ✅ Schema Version: 2
Image: quansight/conda-store-server:2025.2.2
└── ✅ Schema Version: 2
Image: quay.io/argoproj/argocli:v3.4.4
└── ✅ Schema Version: 2
Image: quay.io/argoproj/workflow-controller:v3.4.4
└── ✅ Schema Version: 2
Image: quay.io/jupyterhub-ssh/sftp:0.0.1-0.dev.git.142.h402a3d6
└── ✅ Schema Version: 2
Image: quay.io/jupyterhub-ssh/ssh:0.0.1-0.dev.git.149.he5107a4
└── ✅ Schema Version: 2
Image: quay.io/jupyterhub/configurable-http-proxy:4.6.3
└── ✅ Schema Version: 2
Image: quay.io/keycloak/keycloak:15.0.2
└── ✅ Schema Version: 2
Image: quay.io/kiwigrid/k8s-sidecar:1.26.1
└── ✅ Schema Version: 2
Image: quay.io/nebari/nebari-jupyterhub:2025.6.1
└── ✅ Schema Version: 2
Image: quay.io/nebari/nebari-workflow-controller:2025.6.1
└── ✅ Schema Version: 2
Image: quay.io/prometheus-operator/prometheus-config-reloader:v0.73.2
└── ✅ Schema Version: 2
Image: quay.io/prometheus-operator/prometheus-operator:v0.73.2
└── ✅ Schema Version: 2
Image: quay.io/prometheus/alertmanager:v0.27.0
└── ✅ Schema Version: 2
Image: quay.io/prometheus/node-exporter:v1.8.0
└── ✅ Schema Version: 2
Image: quay.io/prometheus/prometheus:v2.51.2
└── ✅ Schema Version: 2
Image: registry.k8s.io/kube-scheduler:v1.30.11
└── ✅ Schema Version: 2
Image: registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.12.0
└── ✅ Schema Version: 2
Image: traefik:2.9.1
└── ✅ Schema Version: 2
Image: us-central1-artifactregistry.gcr.io/gke-release/gke-release/cluster-proportional-autoscaler:v1.8.11-gke.38@sha256:4b90b5a890385a796c6b1f2f4e3a46477349a5fe620a5bfcf67fd5b3c9621ed4
└── ✅ Schema Version: 2
Image: us-central1-artifactregistry.gcr.io/gke-release/gke-release/cluster-proportional-autoscaler:v1.9.1-gke.17@sha256:3b9d333418723c07ac6ee488d517fa9a9afddd20a8c3b91ac73b5de2ba736f83
└── ✅ Schema Version: 2
Image: us-central1-artifactregistry.gcr.io/gke-release/gke-release/csi-node-driver-registrar:v2.9.4-gke.42@sha256:7394b701866d6f10296dd1652ca6f29ab690ea528d75327076d8a4ca5a9f5ca7
└── ✅ Schema Version: 2
Image: us-central1-artifactregistry.gcr.io/gke-release/gke-release/event-exporter:v0.5.0-gke.4@sha256:ce083125fdbe6a24abc7468b02c9cad0fc489a1a3684b0c91b5d0ce36a31eac2
└── ✅ Schema Version: 2
Image: us-central1-artifactregistry.gcr.io/gke-release/gke-release/fluent-bit-gke-exporter:v0.27.9-gke.13@sha256:6f776db546d57359f991d34a9eb19f0c7006e49e2bc2d636ffb228562d8d42b4
└── ✅ Schema Version: 2
Image: us-central1-artifactregistry.gcr.io/gke-release/gke-release/fluent-bit:v1.8.1200-gke.14@sha256:fe028dfcf00bdaded6770720de8df8f3d24e841f41a968138ae00d699003aa0f
└── ✅ Schema Version: 2
Image: us-central1-artifactregistry.gcr.io/gke-release/gke-release/gcp-compute-persistent-disk-csi-driver:v1.14.6-gke.3@sha256:daa009dd144d60055c6b192d08a36325ee15e7c2b247257d2b4c19be3b03fedc
└── ✅ Schema Version: 2
Image: us-central1-artifactregistry.gcr.io/gke-release/gke-release/gke-metrics-agent:1.13.0-gke.4@sha256:a258598d2c51769077b0c0b2e63aa558e8621b14d1c3b31aab3dc62f5df7eb00
└── ✅ Schema Version: 2
Image: us-central1-artifactregistry.gcr.io/gke-release/gke-release/gke-metrics-collector:20240501_2300_RC0@sha256:af727fbef6a16960bd3541d89b94e1a4938b57041e5869f148995d8c271a6334
└── ✅ Schema Version: 2
Image: us-central1-artifactregistry.gcr.io/gke-release/gke-release/gke-metrics-collector:20240717_2300_RC0@sha256:d460e6b5088332f62b990f8a1f7bf6d9eca7c3f41cb974e3db493d6b0fc4ad70
└── ✅ Schema Version: 2
Image: us-central1-artifactregistry.gcr.io/gke-release/gke-release/gke-metrics-collector:20240902_2300_RC0@sha256:52df59dabb65d3d315ee03768ca1e9d84da2821a799c54cab7539f5f5b19849e
└── ✅ Schema Version: 2
Image: us-central1-artifactregistry.gcr.io/gke-release/gke-release/gke-metrics-collector:20250724_2300_RC0@sha256:42875f6f4a2a79ad1f409fb5624eb66709738568e2d1da02f141493dd7ad216f
└── ✅ Schema Version: 2
Image: us-central1-artifactregistry.gcr.io/gke-release/gke-release/gke-metrics-collector:20250821_2300_RC0@sha256:44b242f7acad93034bf7c5d67009b5feec1eafcc53840546225b00cb03a53d37
└── ✅ Schema Version: 2
Image: us-central1-artifactregistry.gcr.io/gke-release/gke-release/ingress-gce-404-server-with-metrics:v1.32.4@sha256:0691a9c988af4fd4ae58f96fff1005cdf596eaab91feff88895bcb491052ec25
└── ✅ Schema Version: 2
Image: us-central1-artifactregistry.gcr.io/gke-release/gke-release/k8s-dns-dnsmasq-nanny:1.23.0-gke.21@sha256:0aa0dd4637f2766334f7f89617fd16b23439e20ddc7c9099e9f73cb6f5c449c4
└── ✅ Schema Version: 2
Image: us-central1-artifactregistry.gcr.io/gke-release/gke-release/k8s-dns-kube-dns:1.23.0-gke.21@sha256:3e98357ee0ef1ef6548ea5d1fca84b00d04fe6d6dcaa75a56af836c6daea6107
└── ✅ Schema Version: 2
Image: us-central1-artifactregistry.gcr.io/gke-release/gke-release/k8s-dns-sidecar:1.23.0-gke.21@sha256:65ab37f0efc2aae513ce5cfc9d98724f514adc7091ff9ca9b131f05f4197c279
└── ✅ Schema Version: 2
Image: us-central1-artifactregistry.gcr.io/gke-release/gke-release/kube-proxy-amd64:v1.31.12-gke.1220000
└── ✅ Schema Version: 2
Image: us-central1-artifactregistry.gcr.io/gke-release/gke-release/kube-state-metrics:v2.7.0-gke.82@sha256:ccb84ee0531fba295147c0095ce9496ee40d6c8e271e7aed750c9a09ddd7aafb
└── ✅ Schema Version: 2
Image: us-central1-artifactregistry.gcr.io/gke-release/gke-release/metrics-server:v0.7.1-gke.45@sha256:6d492cdefe6ca4b4582f37318b70fb2098cb35058f04128fb9f5a4cf9bd73243
└── ✅ Schema Version: 2
Image: us-central1-artifactregistry.gcr.io/gke-release/gke-release/prometheus-engine/config-reloader:v0.14.3-gke.0@sha256:3e0e786cc4159f0c65ba667986a9c4e013089b6c2e80e88d545e3b2b94bad7f1
└── ✅ Schema Version: 2
Image: us-central1-artifactregistry.gcr.io/gke-release/gke-release/prometheus-engine/operator:v0.14.3-gke.0@sha256:80cd81d20181c47c784b55fcc5da77b88eccd48bfd18ece45bf08afc1486f76a
└── ✅ Schema Version: 2
Image: us-central1-artifactregistry.gcr.io/gke-release/gke-release/prometheus-engine/prometheus:v2.45.3-gmp.11-gke.0@sha256:3ba777873a3267c008049297c4e2292a27e7dbeebdae292f99b7af261175c54f
└── ✅ Schema Version: 2
Image: us-central1-artifactregistry.gcr.io/gke-release/gke-release/prometheus-to-sd:v0.11.12-gke.51@sha256:798127b7368b1a3a2851a6a336776739f32b0ed741d5d6ee07b97d6ac2998fa3
└── ✅ Schema Version: 2
Image: us-central1-artifactregistry.gcr.io/gke-release/gke-release/prometheus-to-sd:v0.12.1-gke.24@sha256:4e76d5f407e7a072f1f26dd1d8b019950d1c7632ed96fdd19315f664413c9b82
└── ✅ Schema Version: 2
Image: us-central1-artifactregistry.gcr.io/gke-release/gke-release/proxy-agent:v0.31.3-gke.1@sha256:aa89e06b001dd7ef9e9082de3da7ef4b7995c6c857e905d3810eb9901664158f
└── ✅ Schema Version: 2

Sep 25 '25 20:09 typotts42

The NFS part has been addressed in the newer release, and I think moving dask to helm chart #3176 should help with the dask one there

Nov 13 '25 15:11 viniciusdc