[EKS][Fargate] [request]: Support SYS_PTRACE for EKS Fargate

[evq]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request What do you want us to build?

Support SYS_PTRACE for EKS Fargate ( already supported on ECS )

Which service(s) is this request for? Fargate on EKS

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard? Debugability and introspection of running workloads. May also allow for an alternative method for capturing process stdout to allow sidecar logging with EKS Fargate in a more transparent method through stdout redirection.

See https://github.com/aws/containers-roadmap/issues/409 for the same request on ECS.

Are you currently working around this issue?

Additional context

Attachments

[j-rat]

We are looking to have this capability to perform run time analysis of workloads.

[d0nwilliams]

I too echo this We are looking to have this capability to perform run time analysis of workloads.

[ollypom]

EKS Fargate doesn't explicity support ptrace workloads today, however it does not prevent you from running them either. This is because no seccomp policy is applied to Kubernetes Pods by default. This will change in the future (Maybe Kubernetes 1.25), and when it does you could write a custom seccomp policy to allow a pod to use ptrace. That being said EKS/Fargate should allow you to add the ptrace capability explicity (as it does on ECS/Fargate), so this issue should stay open. In the mean time, for those curious, you can run ptrace workloads on EKS/Fargate today.

---

A quick test using strace:

Dockerfile

FROM debian

RUN apt-get update && \
    apt-get install strace -y

CMD ["strace", "echo", "hello"]

Deploy 3 Pods to EKS Fargate:

1. A "default pod" with no seccomp policy.

apiVersion: v1
kind: Pod
metadata:
  name: defaultpod
spec:
  restartPolicy: Never
  containers:
  - name: strace
    image: 111222333444.dkr.ecr.eu-west-1.amazonaws.com/strace:latest

2. A "seccomp pod", this is a Pod with the runtime seccomp policy applied

apiVersion: v1
kind: Pod
metadata:
  name: seccomppod
spec:
  restartPolicy: Never
  securityContext:
    seccompProfile:
      type: RuntimeDefault
  containers:
  - name: strace
    image: 111222333444.dkr.ecr.eu-west-1.amazonaws.com/strace:latest

3. A "ptrace pod", this is a Pod with the runtime seccomp policy applied, but I am attempting to explicitly add ptrace to my Pod.

apiVersion: v1
kind: Pod
metadata:
  name: ptracepod
spec:
  restartPolicy: Never
  securityContext:
    seccompProfile:
      type: RuntimeDefault
  containers:
  - name: strace
    image: 111222333444.dkr.ecr.eu-west-1.amazonaws.com/strace:latest
    securityContext:
      capabilities:
        add: ["SYS_PTRACE"]

The Results:

$ kubectl get pods
NAME          READY   STATUS      RESTARTS   AGE
defaultpod    0/1     Completed   0          7m30s
seccomppod    0/1     Error       0          7m30s
ptracepod     0/1     Pending     0          7m30s

1. The default pod worked.

$ kubectl logs defaultpod
execve("/bin/echo", ["echo", "hello"], 0x7ffd1acc9888 /* 18 vars */) = 0
brk(NULL)                               = 0x559437d47000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=6562, ...}) = 0
mmap(NULL, 6562, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7ff492ad7000
close(3)

2. The seccomp pod ran but strace got blocked by the runtime seccomp policy.

$ kubectl logs -f seccomppod
strace: test_ptrace_get_syscall_info: PTRACE_TRACEME: Operation not permitted
strace: ptrace(PTRACE_TRACEME, ...): Operation not permitted
strace: PTRACE_SETOPTIONS: Operation not permitted
strace: detach: waitpid(9): No child processes
strace: Process 9 detached

3. The ptrace pod got blocked by the scheduler as EKS Fargate does not allow the ptrace Linux capability.

$ kubectl describe pod ptracepod
<snip>
Events:
  Type     Reason            Age    From               Message
  ----     ------            ----   ----               -------
  Warning  FailedScheduling  9m42s  fargate-scheduler  Pod not supported on Fargate: invalid SecurityContext fields: Capabilities added: SYS_PTRACE

[danisevas]

Any updates?

[sschamp]

Still no word on this?

I'd like to use this to build multi-arch unprivileged Docker images. This would use proot to be able to use qemu-user-static without binfmt_misc. proot needs SYS_PTRACE to intercept the calls.

This would allow for truly rootless / unprivileged multi-arch Dockerfile builds on EKS Fargate.

[sschamp]

I believe SYS_PTRACE just works on EKS Fargate?

I am using moby/buildkit:master-rootless to make multi-arch builds and it works.