nri icon indicating copy to clipboard operation
nri copied to clipboard

hook-injector - example does not work - no hooks to inject, ignoring

Open lukasmrtvy opened this issue 1 month ago • 1 comments

I did a quick test with hook-injector with kind v0.30.0, but I had no luck with triggering OCI hooks. Any ideas? Thanks

Reproducer:

kind create cluster

cat << EOF > always.json
{
    "version": "1.0.0",
    "hook": {
        "path": "/usr/local/sbin/hook.sh"
    },
    "when": {
        "always": true
    },
    "stages": ["prestart", "poststop"]
}
EOF

cat << EOF >hook.sh
LOG=/tmp/hook.log
touch $LOG
echo "$(date)" >> $LOG
EOF

cat << EOF > patch-hook-injector.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: nri-plugin-hook-injector
spec:
  template:
    spec:
      initContainers:
        - name: inject-hooks
          image: busybox
          command:
            - sh
            - -c
            - |
              cp -v /config/hooks/* /etc/containers/oci/hooks.d 2>/dev/null || true
              cp -v /config/bins/* /usr/local/sbin 2>/dev/null || true
              chmod +x /usr/local/sbin/hook.sh
          securityContext:
            privileged: true
          volumeMounts:
            - name: hooks-config
              mountPath: /config/hooks
            - name: hooks-host
              mountPath: /etc/containers/oci/hooks.d
            - name: bins-config
              mountPath: /config/bins
            - name: bins-host
              mountPath: /usr/local/sbin
      volumes:
        - name: hooks-config
          configMap:
            name: hooks
        - name: bins-config
          configMap:
            name: bins
        - hostPath:
            path: /etc/containers/oci/hooks.d
            type: DirectoryOrCreate
          name: hooks-host
        - hostPath:
            path: /usr/local/sbin
            type: DirectoryOrCreate
          name: bins-host
EOF

cat << EOF > kustomization.yaml
resources:
- https://github.com/containerd/nri/contrib/kustomize/hook-injector
configMapGenerator:
- files:
  - always.json
  name: hooks
- files:
  - hook.sh
  name: bins
namespace: oci-hook-daemonset
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
patches:
- path: patch-hook-injector.yaml
EOF

kustomize build | kubectl apply -f- 

kubectl run --rm -it \
  --image=alpine \
  --restart=Never \
  test -- sh -c 'sleep 5; exit 0'

kubectl logs -l app.kubernetes.io/name=nri-plugin-hook-injector -n oci-hook-daemonset             Defaulted container "plugin" out of: plugin, inject-hooks (init)
time="2025-11-02T22:16:53Z" level=info msg="Created plugin 10-hook-injector (plugin, handles CreateContainer)"
time="2025-11-02T22:16:53Z" level=info msg="watching directories \"/usr/share/containers/oci/hooks.d /etc/containers/oci/hooks.d\" for new changes"
time="2025-11-02T22:16:53Z" level=info msg="Registering plugin 10-hook-injector..."
time="2025-11-02T22:16:53Z" level=info msg="Configuring plugin 10-hook-injector for runtime containerd/v2.1.3..."
time="2025-11-02T22:16:53Z" level=info msg="Started plugin 10-hook-injector..."
time="2025-11-02T22:25:20Z" level=info msg="test/test: no hooks to inject, ignoring"

lukasmrtvy avatar Nov 02 '25 22:11 lukasmrtvy

suggest just starting with the hooks in the right place for the root context of the container runtime hook injector.. vs trying to set the hooks up via pod initcontainer

divide and conquer..

mikebrow avatar Nov 03 '25 16:11 mikebrow

@lukasmrtvy I think your problem is that you do not inject the hook binary into the hook injector plugin's container. Therefore, it will not try to inject it into any container, as it thinks the hook is a nonexistent one. Here is a working example without any kustomize, just using a self-contained daemonset YAML and our published hook injector plugin image. In principle, you should be able to get your setup with kustomize working as well, if you can patch things up so that the plugin will see the to be injected hook binary.

Credit where credit is due, this example is a modified version of an RDT per-container monitoring example contributed (but not yet filed as a PR) by @marquiz. He has done all the heavy lifting hacking the self-contained example together. I merely stole it and retrofitted your test/example on top of it.

#
# This example demonstrates how to run the OCI Hook injector with a custom hook injected.
#
# It leverages the hook-injector NRI plugin to inject a test OCI hooks into every container
# the injector sees being created.
#
# 1. A custom location (/etc/containers/nri/test-hook) is used to store the OCI hook binary and configuration
# 2. An init container creates the OCI hook binary on the host
# 3. A second init container creates the OCI hook configuration on the host
# 4. The hook-injector NRI plugin is run in the main container. The hook injector is
#    configured to only watch for OCI hooks in the custom location.
#
apiVersion: apps/v1
kind: DaemonSet
metadata:
  labels:
    app: nri-plugin-hook-injector
  name: nri-plugin-hook-injector
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: nri-plugin-hook-injector
  template:
    metadata:
      labels:
        app: nri-plugin-hook-injector
    spec:
      initContainers:
      - name: deploy-hook-binary
        image: busybox:latest
        command:
          - sh
          - -c
          - |
            echo "Creating OCI hook binary"
            cat > "$HOOKS_DIR/hook.sh" <<'EOF'
            #!/bin/sh
            LOG=/tmp/hook.log
            touch $LOG
            echo "$(date)" >> $LOG
            EOF
            chmod +x "$HOOKS_DIR/hook.sh"
        env:
        - name: HOOKS_DIR
          value: /hooks
        volumeMounts:
        - name: hooks-dir
          mountPath: /hooks
     - name: deploy-hook-config
        image: busybox:latest
        command:
          - sh
          - -c
          - |
            echo "Creating OCI hook config"
            cat > "$HOOKS_DIR/hook.json" <<'EOF'
            {
              "version": "1.0.0",
              "hook": {
                  "path": "/etc/containers/nri/test-hook/hook.sh",
                  "args": ["", "auto"]
              },
              "when": {
                  "always": true
              },
              "stages": ["prestart", "poststop"]
            }
            EOF
        env:
        - name: HOOKS_DIR
          value: /hooks
        volumeMounts:
        - name: hooks-dir
          mountPath: /hooks
      containers:
      - args:
        - -idx
        - "10"
        image: ghcr.io/containerd/nri/plugins/hook-injector:unstable
        imagePullPolicy: Always
        name: plugin
        resources:
          requests:
            cpu: 2m
            memory: 5Mi
        volumeMounts:
        - name: nri-socket
          mountPath: /var/run/nri/nri.sock
        - name: hooks-dir
          mountPath: /etc/containers/oci/hooks.d
          readOnly: true
        - name: hooks-dir
          mountPath: /etc/containers/nri/test-hook
          readOnly: true
      volumes:
      - name: nri-socket
        hostPath:
          path: /var/run/nri/nri.sock
          type: Socket
      - name: hooks-dir
        hostPath:
          path: /etc/containers/nri/test-hook

klihub avatar Nov 05 '25 15:11 klihub

@lukasmrtvy Actually let me correct myself. @marquiz has indeed contributed the RDT example, as a kustomization, in a PR and it has been merged. It is here in the repo: https://github.com/containerd/nri/tree/main/contrib/kustomize/samples/rdt-monitoring. I tried it from our repo HEAD and it works out of the box. Maybe you can take that one as a starting point.

klihub avatar Nov 05 '25 16:11 klihub

And here is it hacked to become your example:

# cat initcontainers-patch.yaml 
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: nri-plugin-hook-injector
spec:
  template:
    spec:
      initContainers:
      - name: deploy-hook-binary
        image: busybox:latest
        command:
          - sh
          - -c
          - |
            echo "Creating OCI hook binary"
            cat > "$HOOKS_DIR/hook.sh" <<'EOF'
            #!/bin/sh
            LOG=/tmp/hook.log
            touch $LOG
            echo "$(date)" >> $LOG
            echo "  $*" >> $LOG
            EOF
            chmod +x "$HOOKS_DIR/hook.sh"
        env:
        - name: HOOKS_DIR
          value: /hooks
        volumeMounts:
        - name: etc-hooks-d
          mountPath: /hooks

      - name: deploy-hook-config
        image: busybox:latest
        command:
          - sh
          - -c
          - |
            echo "Creating OCI hook config"
            cat > "$HOOKS_DIR/hook.json" <<'EOF'
            {
              "version": "1.0.0",
              "hook": {
                  "path": "/etc/containers/nri/test-hook/hook.sh",
                  "args": ["foo", "bar", "xyzzy", "foobar" ]
              },
              "when": {
                  "always": true
              },
              "stages": ["createRuntime", "poststop"]
            }
            EOF
        env:
        - name: HOOKS_DIR
          value: /hooks
        volumeMounts:
        - name: etc-hooks-d
          mountPath: /hooks
# cat kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../../hook-injector/unstable
patches:
  - path: initcontainers-patch.yaml
  - path: volumes-patch.yaml
# cat volumes-patch.yaml 
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: nri-plugin-hook-injector
spec:
  template:
    spec:
      containers:
        - name: plugin
          volumeMounts:
            - name: etc-hooks-d
              mountPath: /etc/containers/nri/test-hook
            # Remove unwanted mounts
            - $patch: delete
              mountPath: /usr/share/containers/oci/hooks.d
            - $patch: delete
              mountPath: /usr/libexec/oci/hooks.d
      volumes:
        # Replace the default hooks directory with a custom one
        - name: etc-hooks-d
          hostPath:
            path: /etc/containers/nri/test-hook
            type: DirectoryOrCreate
        - $patch: delete
          name: usr-share-hooks-d
        - $patch: delete
          name: libexec-hooks-d

Now if I spin this up (although not in kind, but in a single-node cluster in VM), everthing works. I get the expected logs if I create pods/containers:

[root@n4c16-fedora-42-containerd hacked-test]# kustomize build | kubectl apply -f -
daemonset.apps/nri-plugin-hook-injector created
[root@n4c16-fedora-42-containerd hacked-test]# tail -f /tmp/hook.log 
Wed Nov  5 04:30:42 PM UTC 2025
  bar xyzzy foobar
Wed Nov  5 04:30:42 PM UTC 2025
  bar xyzzy foobar

klihub avatar Nov 05 '25 16:11 klihub

Thanks 👍 I made a mistake 🤦 ( also edited original description with the correct kustomization patch )

lukasmrtvy avatar Nov 05 '25 21:11 lukasmrtvy