Cannot run goreplay as sidecar in K8S

[javidaslan7s]

Hi,

We are trying to run goreplay as sidecar in AWS EKS in pod along with PHP container. However, we are running into the following error:

PCAP Activate device error: "Permission Denied", interface: "eth0"
PCAP Activate device error: "Permission Denied", interface: "lo"

Here are details:

1. goreplay release version: v1.3.3
2. Dockerfile:

FROM alpine:3.16 as builder

ENV RELEASE_VERSION=1.3.3

RUN apk add --no-cache ca-certificates openssl
RUN wget https://github.com/buger/goreplay/releases/download/${RELEASE_VERSION}/gor_${RELEASE_VERSION}_x64.tar.gz -O gor.tar.gz
RUN tar xzf gor.tar.gz

FROM scratch
COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
COPY --from=builder /gor /usr/local/bin/gor

ENTRYPOINT ["/usr/local/bin/gor", "--input-raw", ":8081", "--output-kafka-host", "<kafka_host>", "--output-kafka-topic", "<kafka_topic>"]

Google didn't really help. I tried with older version of goreplay too, but unfortunately no success.

I was wondering whether there could be any limitation from K8S on utilisation/interacting with certain ports.

Any ideas are appreciated, thanks in advance.

[wangfeng22]

Hi,

We are trying to run goreplay as sidecar in AWS EKS in pod along with PHP container. However, we are running into the following error:

PCAP Activate device error: "Permission Denied", interface: "eth0"
PCAP Activate device error: "Permission Denied", interface: "lo"

Here are details:

1. goreplay release version: v1.3.3
2. Dockerfile:

FROM alpine:3.16 as builder

ENV RELEASE_VERSION=1.3.3

RUN apk add --no-cache ca-certificates openssl
RUN wget https://github.com/buger/goreplay/releases/download/${RELEASE_VERSION}/gor_${RELEASE_VERSION}_x64.tar.gz -O gor.tar.gz
RUN tar xzf gor.tar.gz

FROM scratch
COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
COPY --from=builder /gor /usr/local/bin/gor

ENTRYPOINT ["/usr/local/bin/gor", "--input-raw", ":8081", "--output-kafka-host", "<kafka_host>", "--output-kafka-topic", "<kafka_topic>"]

Google didn't really help. I tried with older version of goreplay too, but unfortunately no success.

I was wondering whether there could be any limitation from K8S on utilisation/interacting with certain ports.

Any ideas are appreciated, thanks in advance.

Maybe you can try to use docker run -u root or docker run --privileged=true commands to run. It's most likely a Permissions issue.

[MrEcco]

For running it in sidecar you should enable these special capabilities for the container:

docker run --cap-add NET_RAW --cap-add NET_ADMIN your:image

You can specify container security context. See details/howto here: https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.22/#securitycontext-v1-core

Schematic example:

kind: Pod
spec:
  containers:
    - name: application
    - name: gor-sidecar
      securityContext:
        capabilities:
          drop:
            - ALL
          add:
            - NET_RAW
            - NET_ADMIN

Don't use privileged mode when you can avoid it :)

[txvu100]

Those special caps didn't work for me.

Error: pods "<pod_name>" is forbidden: PodSecurityPolicy: unable to admit pod: [spec.containers[1].securityContext.capabilities.add: Invalid value: "NET_ADMIN": capability may not be added spec.containers[1].securityContext.capabilities.add: Invalid value: "NET_RAW": capability may not be added:Deployment does not have minimum availability.

[jxperf]

@txvu I'm facing the same issue as you. Are you by any chance using GKE autopilot? because I think NET_RAW and NET_ADMIN are blocked in that environment

https://cloud.google.com/kubernetes-engine/docs/concepts/autopilot-security#built-in-security

[txvu100]

I got the error in my cluster in Rancher. I checked the PodSecurityPolicy object and it's possibly the culprit here.