What happened

The following error happens when running kube-hunter with the following definition on k8s v1.21.2:

containers:
            - name: kube-hunter
              image: aquasec/kube-hunter:0.5.2
              command: [ "kube-hunter" ]
              args: [ "--pod", "--log", "DEBUG" ]

Expected behavior

The table should be generated and printed to stdout

Thanks for reporting! This looks really weird. could you provide the full debug log data? @AljoschaP

Hi,

I started playing with kube-hunter today with my playground cluster on GKE and I am facing a similar issue when I create a dedicated NetworkPolicy for the Pod.

I will create a separate issue if @AljoschaP 's issue is not exactly the same as the one I am facing.

Current Setup

Versions

kube-hunter: v0.5.2 built from the Dockerfile in the repository for tag 0.5.2 kubernetes: v1.19.10-gke.1600 (Server) and v1.19.4 (Client)

Manifests to reproduce

The simplified manifests below are sufficient to reproduce the issue and non-cluster resources will be located in the default namespace.

Note that, for brievety, the NetworkPolicy applies to the whole namespace (podSelector: {})

NetworkPolicies are baked by Calico CNI plugin.

# kube-hunter.yaml
---
# SA
apiVersion: v1
kind: ServiceAccount
metadata:
  name: ci-kube-hunter-in-cluster
  namespace: default
---
# ClusterRole
# FIXME, kube-hunter project doesn't seem provide an exhaustive list of the required
# RBAC permissions, so we simply grant it read access (get, list) to all resources for now
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: ci-kube-hunter-in-cluster
rules:
  - apiGroups: ["*"]
    resources: ["*"]
    verbs: ["get", "list"]
---
# Bind SA to the ClusterRole.
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: ci-kube-hunter-in-cluster
subjects:
  - kind: ServiceAccount
    name: ci-kube-hunter-in-cluster
    namespace: default
    apiGroup: ""
roleRef:
  kind: ClusterRole
  name: ci-kube-hunter-in-cluster
  apiGroup: rbac.authorization.k8s.io
---
# NetworkPolicy
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: ci-kube-hunter-in-cluster
  namespace: default
spec:
  policyTypes:
    - Egress
    - Ingress
  podSelector: {} # simplified to match all pods for brievety
  egress:
    # Allow DNS
    - to:
        - podSelector:
            matchLabels:
              k8s-app: kube-dns
          namespaceSelector:
            matchLabels:
              networking/namespace: kube-system
      ports:
        - protocol: TCP
          port: 53
        - protocol: UDP
          port: 53
    # Allow Kubernetes networks
    # TODO, need to check in details what kube-hunter really needs to access
    - to:
        - ipBlock:
            # Master Network (kube api server)
            cidr: *redacted*
        - ipBlock:
            # Nodes Networks
            cidr: *redacted*
        - ipBlock:
            # Pods IPs
            cidr: *redacted*
        - ipBlock:
            # Service IPs
            cidr: *redacted*
---
apiVersion: batch/v1
kind: Job
metadata:
  name: ci-kube-hunter-in-cluster
  namespace: default
spec:
  template:
    spec:
      serviceAccountName: ci-kube-hunter-in-cluster
      containers:
        - name: kube-hunter-in-cluster
          # @see https://github.com/aquasecurity/kube-hunter/tree/main#container
          image: *redacted*/kube-hunter:0.5.2 # ${KUBE_HUNTER_VERSION}
          command: ["kube-hunter"]
          args: ["--pod", "--report", "plain", "--log", "debug"]
      restartPolicy: Never

k apply -f kube-hunter.yaml

Logs

The logs below were generated from a kube-hunter run on a freshly generated cluster with no additional workloads.

Note that I manually redacted the auth_token and client_cert info using a regexp

# The output is too long for GitHub comments

See output.txt

Note that when I remove the NetworkPolicy, or more precisely perform an allow all on egress (.spec.egress[0].to: {}), the error doesn't occur any more and the report is properly available in stdout.

Optionally, can you please clarify, if available, the exhaustive list of required RBAC permissions and network egress targets ?

Sorry for the late reply. I think the issue of @julien-sugg is the same as the one that occurs in our cluster.

Same issue is happening to me when running kube-hunter inside the cluster using the --pod flag

Kube-hunter deployment

apiVersion: v1
kind: Pod
metadata:
  name: kube-hunter-with-sa
  namespace: default
  labels:
    app: kube-hunter-with-sa
spec:
  serviceAccountName: kube-hunter
  containers:
    - name: kube-hunter-with-sa
      image: aquasec/kube-hunter:0.6.8
      command: ["kube-hunter"]
      args: ["--pod",  "--log", "debug"]
  restartPolicy: Never

Kubernetes version:

NAME                             STATUS   ROLES           AGE    VERSION
workshop-cluster-control-plane   Ready    control-plane   3d3h   v1.27.3

Full error log

2023-09-23 07:13:24,403 DEBUG kube_hunter.core.events.event_handler Event <class 'kube_hunter.core.events.types.HuntFinished'> got published to hunter - <class 'kube_hunter.modules.report. │
│ 2023-09-23 07:13:24,403 DEBUG kube_hunter.core.events.event_handler Executing <class 'kube_hunter.modules.report.collector.SendFullReport'> with {'previous': None, 'hunter': None}          │
│ 2023-09-23 07:13:24,406 DEBUG kube_hunter.core.events.event_handler '<' not supported between instances of 'str' and 'IPAddress'                                                             │
│ Traceback (most recent call last):                                                                                                                                                           │
│   File "/usr/local/lib/python3.8/site-packages/kube_hunter/core/events/event_handler.py", line 346, in worker                                                                                │
│     hook.execute()                                                                                                                                                                           │
│   File "/usr/local/lib/python3.8/site-packages/kube_hunter/modules/report/collector.py", line 56, in execute                                                                                 │
│     report = config.reporter.get_report(statistics=config.statistics, mapping=config.mapping)                                                                                                │
│   File "/usr/local/lib/python3.8/site-packages/kube_hunter/modules/report/plain.py", line 30, in get_report                                                                                  │
│     output += self.nodes_table()                                                                                                                                                             │
│   File "/usr/local/lib/python3.8/site-packages/kube_hunter/modules/report/plain.py", line 62, in nodes_table                                                                                 │
│     nodes_ret = f"\nNodes\n{nodes_table}\n"                                                                                                                                                  │
│   File "/usr/local/lib/python3.8/site-packages/prettytable/prettytable.py", line 327, in __str__                                                                                             │
│     return self.get_string()                                                                                                                                                                 │
│   File "/usr/local/lib/python3.8/site-packages/prettytable/prettytable.py", line 1700, in get_string                                                                                         │
│     rows = self._get_rows(options)                                                                                                                                                           │
│   File "/usr/local/lib/python3.8/site-packages/prettytable/prettytable.py", line 1618, in _get_rows                                                                                          │
│     rows.sort(reverse=options["reversesort"], key=options["sort_key"])                                                                                                                       │
│ TypeError: '<' not supported between instances of 'str' and 'IPAddress'                                                                                                                      │
│ 2023-09-23 07:13:24,412 DEBUG kube_hunter.__main__ Cleaned Queue