Dragonfly and nydus are integrated via mirror

[gaius-qi]

Feature request:

Use case:

UI Example:

[imeoer]

@sctb512 @raoxiang1996 We can reference the issue once the PR about mirror support is ready on nydus side, some docs should be published.

[gaius-qi]

Test: https://hackmd.io/2g_NiyapQJOavVKPu4NNfw?view

@sctb512

[gaius-qi]

Install Test Environment:

Install Dragonfly

Create Kind Cluster

Create kind config in ./kind-config.yaml:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
networking:
  ipFamily: dual
nodes:
- role: control-plane
  image: kindest/node:v1.23.4@sha256:0e34f0d0fd448aa2f2819cfd74e99fe5793a6e4938b328f657c8e3f81ee0dfb9
- role: worker
  image: kindest/node:v1.23.4@sha256:0e34f0d0fd448aa2f2819cfd74e99fe5793a6e4938b328f657c8e3f81ee0dfb9
  extraPortMappings:
  - containerPort: 30950
    hostPort: 65001
- role: worker
  image: kindest/node:v1.23.4@sha256:0e34f0d0fd448aa2f2819cfd74e99fe5793a6e4938b328f657c8e3f81ee0dfb9
- role: worker
  image: kindest/node:v1.23.4@sha256:0e34f0d0fd448aa2f2819cfd74e99fe5793a6e4938b328f657c8e3f81ee0dfb9

Create kind cluster:

kind create cluster --config ./kind-config.yaml

Helm Install Dragonfly

Create helm charts config in ./charts-config.yaml:

scheduler:
  replicas: 1
  metrics:
    enable: true
  config:
    verbose: true
    pprofPort: 18066

seedPeer:
  replicas: 3
  metrics:
    enable: true
  config:
    verbose: true
    pprofPort: 18066
    download:
      prefetch: true

dfdaemon:
  hostNetwork: true
  config:
    verbose: true
    pprofPort: 18066
    metrics: 127.0.0.1:8000
    download:
      prefetch: true
    scheduler:
      disableAutoBackSource: true
      manager:
        refreshInterval: 10s
    proxy:
      # -- Filter for hash url
      # when defaultFilter: "Expires&Signature", for example:
      #  http://localhost/xyz?Expires=111&Signature=222 and http://localhost/xyz?Expires=333&Signature=999
      # is same task
      defaultFilter: "Expires&Signature&ns"
      # -- Proxy security option
      security:
        insecure: true
      tcpListen:
        # -- Namespace stands the linux net namespace, like /proc/1/ns/net
        # it's useful for running daemon in pod with ip allocated and listening the special port in host net namespace
        # Linux only
        namespace: /run/dragonfly/net
        # -- Listen address
        listen: 0.0.0.0
        # If you want to change port, please update hostPort in $.Values.dfdaemon.hostPort
        # port in configmap is generated from $.Values.dfdaemon.hostPort
        # port: 65001
      registryMirror:
        # -- When enabled, use value of "X-Dragonfly-Registry" in http header for remote instead of url host
        dynamic: true
        # -- URL for the registry mirror
        url: https://index.docker.io
      proxies:
        # -- Proxy all http image layer download requests with dfget
        - regx: blobs/sha256.*

manager:
  replicas: 1
  metrics:
    enable: true
  config:
    verbose: true
    pprofPort: 18066
jaeger:
  enable: true

Helm install dragonfly:

helm repo add dragonfly https://dragonflyoss.github.io/helm-charts/
helm install --wait --create-namespace --namespace dragonfly-system dragonfly  dragonfly/dragonfly  -f ./charts-config.yaml

Apply Dfdaemon Service

Create dfdaemon service config in ./svc.yaml:

apiVersion: v1
kind: Service
metadata:
  name: peer
  namespace: dragonfly-system
spec:
  type: NodePort
  ports:
  - name: http
    nodePort: 30950
    port: 65001
  selector:
    app: dragonfly
    component: dfdaemon
    release: dragonfly

Apply service config:

kubectl apply -f ./svc.yaml

Validation

curl -x 127.0.0.1:65001 https://github.com -o test-content

Install Nydus

Install Containerd

Refer to https://github.com/containerd/containerd/blob/main/docs/getting-started.md#step-1-installing-containerd

Install nerdctl

wget https://github.com/containerd/nerdctl/releases/download/v0.22.2/nerdctl-full-0.22.2-linux-amd64.tar.gz
sudo tar -zxvf nerdctl-full-0.22.2-linux-amd64.tar.gz -C /usr/local

Install nydus-snapshotter

git clone https://github.com/containerd/nydus-snapshotter.git
cd nydus-snapshotter
make 
make install

Configure Containerd

Change config /etc/containerd/config.toml:

[proxy_plugins]
  [proxy_plugins.nydus]
    type = "snapshot"
    address = "/run/containerd-nydus/containerd-nydus-grpc.sock"

[plugins.cri]
  [plugins.cri.containerd]
    snapshotter = "nydus"
    disable_snapshot_annotations = false

Restrat containerd:

systemctl restart containerd

Configure Nydus Snapshotter

Create config /etc/nydus/config.json:

{
  "device": {
    "backend": {
      "type": "registry",
      "config": {
        "host": "https://index.docker.io",
        "mirrors": [
          {
            "scheme": "http",
            "host": "127.0.0.1:65001",
            "headers": {
              "X-Dragonfly-Registry": "https://index.docker.io"
            }
          },
          {
            "scheme": "http",
            "host": "127.0.0.1:65001",
            "headers": {
              "X-Dragonfly-Registry": "https://index.docker.io"
            }
          }

        ],
        "scheme": "https",
        "timeout": 5,
        "connect_timeout": 5,
        "retry_limit": 2
      }
    },
    "cache": {
      "type": "blobcache",
      "config": {
        "work_dir": "/var/lib/nydus/cache/"
      }
    }
  },
  "mode": "direct",
  "digest_validate": false,
  "iostats_files": false,
  "enable_xattr": true,
  "fs_prefetch": {
    "enable": true,
    "threads_count": 4
  }
}

Start nydus-snapshotter:

git clone [email protected]:dragonflyoss/image-service.git
cd image-service

make fusedev-release
make install

systemctl enable nydus-snapshotter
systemctl start nydus-snapshotter

Build Nydus Image

Install nydusify:

wget https://github.com/dragonflyoss/image-service/releases/download/v2.1.0-rc.3.1/nydus-static-v2.1.0-rc.3.1-linux-amd64.tgz
tar zxvf nydus-static-v2.1.0-rc.3.1-linux-amd64.tgz
cd nydus-static
sudo cp nydusify /usr/local/bin

Build nydus image:

nydusify convert --nydus-image /usr/local/bin/nydus-image --source dragonflyoss/scheduler:latest --target your-repository/scheduler-nydus:latest

Download and run nydus image:

nerdctl --snapshotter nydus run --rm -it gaius qi/scheduler-nydus:latest

[gaius-qi]

Testing:

Start downloading blobs with range header => 2022-09-26 06:00:18.292 Finish downloading blobs with range header => 2022-09-26 06:00:20.098

The two most time-consuming parts:

1. First auth request to return 401 http code => 06:00:18.296 - 06:00:19.233
2. Downlod blobs with range header => 06:00:19.484 - 06:00:20.098

Tracing:

Download blob with range header will be back-to-source twice. The solution is to cache the auth token, and the blob download does not require multiple verifications.

cc @sctb512

[imeoer]

Maybe we need to fix auth header cache for mirror implementation in nydusd.

[sctb512]

Maybe we need to fix auth header cache for mirror implementation in nydusd.

It is under repair.