k8s-bigip-ctlr icon indicating copy to clipboard operation
k8s-bigip-ctlr copied to clipboard

Published 20 hours ago • verified publisher icon F5Networks

CIS encountered a crash because of misconfigured（Configure service exposure while configuring external endpoints）

Open kkfinkkfin opened this issue 3 years ago • 6 comments

Setup Details

CIS Version : 2.6.0
Build: f5networks/k8s-bigip-ctlr:latest
BIGIP Version: BIG-IP 15.1.4 Build 0.0.47 Final
AS3 Version: none
Agent Mode: CCCL
Orchestration: K8S
Orchestration Version: kubernetes v1.21.5
Pool Mode: Cluster
Additional Setup details: <Platform/CNI Plugins/ cluster nodes/ etc> - Platform : CentOS Linux release 8.4.2105 - Kernel: 4.18.0-305.19.1.el8_4.x86_64 - CNI Plugins: flannel

Description

CIS encountered a crash because the application misconfigured（Configure service exposure while configuring external endpoints）. CIS needs to consider the service configuration with external endpoints, when the service is exposed, it should outputs the error logs instead of directly crashing and affecting the refresh of the entire cluster.

Steps To Reproduce

  1. The CIS yaml
kind: Deployment
apiVersion: apps/v1
metadata:
  name: cc-k8s-to-bigip1
  namespace: kube-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: k8s-bigip-ctlr
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: k8s-bigip-ctlr
    spec:
      containers:
        - name: k8s-bigip-ctlr
          image: 'f5networks/k8s-bigip-ctlr:2.6.0'
          command:
            - /app/bin/k8s-bigip-ctlr
          args:
            - '--bigip-username=admin'
            - '--bigip-password=admin.F5demo.com'
            - '--bigip-url=10.1.20.232'
            - '--bigip-partition=p1'
            - '--pool-member-type=cluster'
            - '--flannel-name=/Common/flannel_vxlan'
            - '--insecure=true'
            - '--log-level=INFO'
            - '--agent=cccl'
            - '--disable-teems=true'
            - '--verify-interval=5'
            - '--node-poll-interval=5'
            - '--http-listen-address=0.0.0.0:9113'
          resources: {}
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          imagePullPolicy: IfNotPresent
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      dnsPolicy: ClusterFirst
      serviceAccountName: bigip-ctlr
      serviceAccount: bigip-ctlr
      nodeName: 172.16.240.6
      securityContext: {}
      schedulerName: default-scheduler
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 25%
      maxSurge: 25%
  revisionHistoryLimit: 10
  progressDeadlineSeconds: 600
  1. Declare a service of externalIP and declare the external endpoint.
apiVersion: v1
kind: Service
metadata:
  labels:
    app: db-master-1
  name: postgresql
  namespace: emp-yd01
spec:
  selector:
    app: db-master-1
  externalIPs:
  - 80.11.12.10
  ports:
  - port: 5432
    protocol: TCP
    targetPort: 5432
  sessionAffinity: None
  type: ClusterIP
---
apiVersion: v1
kind: Endpoints
metadata:
  labels:
    app: db-master-1
  name: postgresql
  namespace: emp-yd01
subsets:
- addresses:
  - ip: 192.168.69.240
  ports:
  - port: 5432
    protocol: TCP
---
apiVersion: v1
data:
  data: |-
    {
      "virtualServer": {
        "backend": {
          "serviceName": "postgresql",
          "servicePort": 5432
        },
        "frontend": {
          "partition": "p1",
          "iapp": "/Common/iapp_tcp",
          "iappOptions": {
            "description": "iapp-tcp"
          },
          "iappVariables": {
            "pool__pool_to_use": "/#create_new#",
            "pool__addr": "10.1.10.140",
            "pool__port": "60000",
            "vs__SNATConfig": "automap",
            "vs__ProfileDefaultPersist": "none",
            "pool__irules": "none",
            "vs__ProfileTCP": "tcp",
            "monitor__Monitors": "tcp_default",
            "pool_lb": "least-connections-member"
          },
          "iappPoolMemberTable": {
            "name": "pool__members",
            "columns": [
              {
                "name": "addr",
                "kind": "IPAddress"
              },
              {
                "name": "port",
                "kind": "Port"
              },
              {
                "name": "connection_limit",
                "value": "0"
              }
            ]
          }
        }
      }
    }
  schema: f5schemadb://bigip-virtual-server_v0.1.7.json
kind: ConfigMap
metadata:
  labels:
    cattle.io/creator: norman
    f5type: virtual-server
  name: gp-postsql-postgresql-5432-ve-configmap
  namespace: emp-yd01
  1. Then CIS crashed

Expected Result

Outputs the error logs（Include enough information as much as possible, such as namespace name, VS name, etc.）instead of directly crashing. Need to enhance cis to be more robust .

Actual Result

The CIS crashed. Here is the CIS log

2021/10/27 09:11:15 [INFO] [INIT] Starting: Container Ingress Services - Version: 2.6.0, BuildInfo: azure-1038-b16efbb771d65d2cfc30dd16f50a8398207f89c8
2021/10/27 09:11:15 [INFO] ConfigWriter started: 0xc000457800
2021/10/27 09:11:15 [INFO] Started config driver sub-process at pid: 17
2021/10/27 09:11:15 [INFO] [INIT] Creating Agent for cccl
2021/10/27 09:11:15 [INFO] [CCCL] Initializing CCCL Agent
2021/10/27 09:11:15 [INFO] [CCCL] Removing Partition p1_AS3 

2021/10/27 09:11:15 [INFO] [CORE] NodePoller (0xc0005201b0) registering new listener: 0x17957a0
2021/10/27 09:11:15 [INFO] [CORE] NodePoller (0xc0005201b0) registering new listener: 0x1733980
2021/10/27 09:11:15 [INFO] [CORE] NodePoller started: (0xc0005201b0)
2021/10/27 09:11:15 [INFO] [CORE] Watching Ingress resources.
2021/10/27 09:11:15 [INFO] [CORE] Watching ConfigMap resources.
2021/10/27 09:11:15 [INFO] [CORE] Handling ConfigMap resource events.
2021/10/27 09:11:15 [INFO] [CORE] Handling Ingress resource events.
2021/10/27 09:11:15 [INFO] [CORE] Registered BigIP Metrics
E1027 09:11:15.928085       1 runtime.go:78] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
goroutine 164 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic(0x19460e0, 0x28b9c10)
	/go/src/github.com/F5Networks/k8s-bigip-ctlr/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:74 +0x95
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
	/go/src/github.com/F5Networks/k8s-bigip-ctlr/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:48 +0x89
panic(0x19460e0, 0x28b9c10)
	/usr/local/go/src/runtime/panic.go:969 +0x1b9
github.com/F5Networks/k8s-bigip-ctlr/pkg/appmanager.(*Manager).getEndpointsForCluster(0xc000594b00, 0x0, 0x0, 0xc0007d25f0, 0xc0003a2bf0, 0xb, 0x0, 0x0, 0x13)
	/go/src/github.com/F5Networks/k8s-bigip-ctlr/pkg/appmanager/appManager.go:2922 +0x464
github.com/F5Networks/k8s-bigip-ctlr/pkg/appmanager.(*Manager).updatePoolMembersForCluster(0xc000594b00, 0xc00075fad0, 0xc000217210, 0xa, 0x1538, 0xc000049f58, 0x8, 0xc000599180, 0xc000278000, 0x0, ...)
	/go/src/github.com/F5Networks/k8s-bigip-ctlr/pkg/appmanager/appManager.go:2580 +0x3ba
github.com/F5Networks/k8s-bigip-ctlr/pkg/appmanager.(*Manager).handleConfigForType(0xc000594b00, 0xc000599180, 0xc000049f58, 0x8, 0xc0003a29f0, 0xa, 0x1b6bfd9, 0x8, 0xc0003a29f0, 0xa, ...)
	/go/src/github.com/F5Networks/k8s-bigip-ctlr/pkg/appmanager/appManager.go:2484 +0xba5
github.com/F5Networks/k8s-bigip-ctlr/pkg/appmanager.(*Manager).syncConfigMaps(0xc000594b00, 0xc000997908, 0xc000049f58, 0x8, 0xc0003a29f0, 0xa, 0x1b6bfd9, 0x8, 0xc0003a29f0, 0xa, ...)
	/go/src/github.com/F5Networks/k8s-bigip-ctlr/pkg/appmanager/appManager.go:1420 +0xfbf
github.com/F5Networks/k8s-bigip-ctlr/pkg/appmanager.(*Manager).syncVirtualServer(0xc000594b00, 0xc000049f58, 0x8, 0xc0003a29f0, 0xa, 0x1b6bfd9, 0x8, 0xc0003a29f0, 0xa, 0x1b6a73e, ...)
	/go/src/github.com/F5Networks/k8s-bigip-ctlr/pkg/appmanager/appManager.go:1226 +0xac5
github.com/F5Networks/k8s-bigip-ctlr/pkg/appmanager.(*Manager).processNextVirtualServer(0xc000594b00, 0x203000)
	/go/src/github.com/F5Networks/k8s-bigip-ctlr/pkg/appmanager/appManager.go:1083 +0x1b8
github.com/F5Networks/k8s-bigip-ctlr/pkg/appmanager.(*Manager).virtualServerWorker(0xc000594b00)
	/go/src/github.com/F5Networks/k8s-bigip-ctlr/pkg/appmanager/appManager.go:1020 +0x2b
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc000637540)
	/go/src/github.com/F5Networks/k8s-bigip-ctlr/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x5f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000637540, 0x1d5ad20, 0xc0003009c0, 0x1, 0xc0002b3500)
	/go/src/github.com/F5Networks/k8s-bigip-ctlr/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xad
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000637540, 0x3b9aca00, 0x0, 0x1, 0xc0002b3500)
	/go/src/github.com/F5Networks/k8s-bigip-ctlr/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x98
k8s.io/apimachinery/pkg/util/wait.Until(0xc000637540, 0x3b9aca00, 0xc0002b3500)
	/go/src/github.com/F5Networks/k8s-bigip-ctlr/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90 +0x4d
created by github.com/F5Networks/k8s-bigip-ctlr/pkg/appmanager.(*Manager).runImpl
	/go/src/github.com/F5Networks/k8s-bigip-ctlr/pkg/appmanager/appManager.go:964 +0x130
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x16b7da4]

goroutine 164 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
	/go/src/github.com/F5Networks/k8s-bigip-ctlr/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:55 +0x10c
panic(0x19460e0, 0x28b9c10)
	/usr/local/go/src/runtime/panic.go:969 +0x1b9
github.com/F5Networks/k8s-bigip-ctlr/pkg/appmanager.(*Manager).getEndpointsForCluster(0xc000594b00, 0x0, 0x0, 0xc0007d25f0, 0xc0003a2bf0, 0xb, 0x0, 0x0, 0x13)
	/go/src/github.com/F5Networks/k8s-bigip-ctlr/pkg/appmanager/appManager.go:2922 +0x464
github.com/F5Networks/k8s-bigip-ctlr/pkg/appmanager.(*Manager).updatePoolMembersForCluster(0xc000594b00, 0xc00075fad0, 0xc000217210, 0xa, 0x1538, 0xc000049f58, 0x8, 0xc000599180, 0xc000278000, 0x0, ...)
	/go/src/github.com/F5Networks/k8s-bigip-ctlr/pkg/appmanager/appManager.go:2580 +0x3ba
github.com/F5Networks/k8s-bigip-ctlr/pkg/appmanager.(*Manager).handleConfigForType(0xc000594b00, 0xc000599180, 0xc000049f58, 0x8, 0xc0003a29f0, 0xa, 0x1b6bfd9, 0x8, 0xc0003a29f0, 0xa, ...)
	/go/src/github.com/F5Networks/k8s-bigip-ctlr/pkg/appmanager/appManager.go:2484 +0xba5
github.com/F5Networks/k8s-bigip-ctlr/pkg/appmanager.(*Manager).syncConfigMaps(0xc000594b00, 0xc000997908, 0xc000049f58, 0x8, 0xc0003a29f0, 0xa, 0x1b6bfd9, 0x8, 0xc0003a29f0, 0xa, ...)
	/go/src/github.com/F5Networks/k8s-bigip-ctlr/pkg/appmanager/appManager.go:1420 +0xfbf
github.com/F5Networks/k8s-bigip-ctlr/pkg/appmanager.(*Manager).syncVirtualServer(0xc000594b00, 0xc000049f58, 0x8, 0xc0003a29f0, 0xa, 0x1b6bfd9, 0x8, 0xc0003a29f0, 0xa, 0x1b6a73e, ...)
	/go/src/github.com/F5Networks/k8s-bigip-ctlr/pkg/appmanager/appManager.go:1226 +0xac5
github.com/F5Networks/k8s-bigip-ctlr/pkg/appmanager.(*Manager).processNextVirtualServer(0xc000594b00, 0x203000)
	/go/src/github.com/F5Networks/k8s-bigip-ctlr/pkg/appmanager/appManager.go:1083 +0x1b8
github.com/F5Networks/k8s-bigip-ctlr/pkg/appmanager.(*Manager).virtualServerWorker(0xc000594b00)
	/go/src/github.com/F5Networks/k8s-bigip-ctlr/pkg/appmanager/appManager.go:1020 +0x2b
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc000637540)
	/go/src/github.com/F5Networks/k8s-bigip-ctlr/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x5f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000637540, 0x1d5ad20, 0xc0003009c0, 0x1, 0xc0002b3500)
	/go/src/github.com/F5Networks/k8s-bigip-ctlr/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xad
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000637540, 0x3b9aca00, 0x0, 0x1, 0xc0002b3500)
	/go/src/github.com/F5Networks/k8s-bigip-ctlr/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x98
k8s.io/apimachinery/pkg/util/wait.Until(0xc000637540, 0x3b9aca00, 0xc0002b3500)
	/go/src/github.com/F5Networks/k8s-bigip-ctlr/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90 +0x4d
created by github.com/F5Networks/k8s-bigip-ctlr/pkg/appmanager.(*Manager).runImpl
	/go/src/github.com/F5Networks/k8s-bigip-ctlr/pkg/appmanager/appManager.go:964 +0x130

Diagnostic Information

Observations (if any)

I found that the main cause of the CIS crash was that it found an external Endpoint. 
If the externalIP service is declared but there are no Endpoint associated, the CIS will not detect the service, will not output any logs, and the VS will not be updated on the VE. I am not sure if this is a normal situation.

kkfinkkfin avatar Oct 27 '21 09:10 kkfinkkfin

@kkfinkkfin Unable to reproduce this issue, no exact environment. is this issue from 2.6? or 1.x versions ?

trinaths avatar Oct 28 '21 08:10 trinaths

@kkfinkkfin Unable to reproduce this issue, no exact environment. is this issue from 2.6? or 1.x versions ?

@trinaths This problem can be reproduced in k8s version 1.18+ and centos 7+. This problem also exists in the CIS 2.4 version, and I also reproduced it in the CIS 2.6 version. The CIS version used in this issue report is "f5networks/k8s-bigip-ctlr:2.6.0".

kkfinkkfin avatar Oct 29 '21 02:10 kkfinkkfin

@kkfinkkfin is this issue reproducible with 1.14 an 2.0 ?

trinaths avatar Nov 09 '21 19:11 trinaths

@kkfinkkfin is this issue reproducible with 1.14 an 2.0 ?

I just test the CIS version over 2.4.0 and I have not test it on 1.14 and 2.0

kkfinkkfin avatar Nov 10 '21 06:11 kkfinkkfin

Created [CONTCNTR-3239] for internal tracking.

trinaths avatar Mar 30 '22 11:03 trinaths

Here, CIS needs to support external EP. So moving this as RFE rather than a bug.

trinaths avatar Mar 30 '22 11:03 trinaths

CIS doesn't support Service Type "External". Closing this issue.

trinaths avatar Feb 02 '23 18:02 trinaths