Improve Update tests

[smalleni]

The current upgrade tests at https://github.com/cloud-bulldozer/e2e-benchmarking/blob/master/workloads/upgrade-perf/run_upgrade_fromgit.sh seem to be too simplistic. I would like to see this improve based on the recent upgrade testing we had done for a customer. It is important to load up the cluster and here are some suggested improvements to the script.

1. Use kube-burner to load up the cluster with 100 projects using the cluster-density test
2. For each MachineConfig Pool that is not master, calculate the total allocatable CPUs for all the nodes in that MCP
3. Per MCP create a a new project and launch a sample HTTP node.js app through a deployment with enough replicas such that it consumes 50% of allocatable through requests/limits cores in the MCP
4. Expose each app through a service
5. Expose the service as a route
6. Enforce network policy per app
7. Hit the routes for the duration for the upgrade test using mb

Some yamls for sample

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sampleapp
spec:
  replicas: 300
  selector:
    matchLabels:
      app: sample
  template:
    metadata:
      labels:
        app: sample
    spec:
      containers:
      - name: app
        image: quay.io/smalleni/sampleapp:latest
        readinessProbe:
          httpGet:
            path: /
            port: 8080
          initialDelaySeconds: 3
        ports:
        - containerPort: 8080
          protocol: TCP
        resources:
          requests:
            cpu: "1"
          limits:
            cpu: "1"
      nodeSelector:
        app: "true"

apiVersion: v1
kind: Service
metadata:
  name: samplesvc
spec:
  selector:
    app: sample
  ports:
  - port: 80
    targetPort: 8080

kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: except
spec:
  podSelector:
    matchLabels:
      app: sample
  ingress:
  - from:
    - ipBlock:
        cidr: 10.128.0.0/14
        except:
        - "10.130.36.0/23"
        - "10.130.12.0/23"
        - "10.128.18.0/23"
        - "10.131.10.0/23"
        - "10.131.22.0/23"
        - "10.128.24.0/23"
        - "10.128.14.0/23"

[mukrishn]

@smalleni Assuming it is something to build an environment like VZW, load it up to 50% then upgrade the cluster while it is busy. The requirement is already clear in the description but one question about the MCP, I remember we create multiple MCP to roll out the upgrade per MCP so do we need to follow the same approach here as well by creating multiple MCP(may be 10 worker per MCP) and then follow step 1-7 ? or we can keep all workers in same MCP and upgrade it at once ?

[smalleni]

I say the number of MCPs should be configurable in the test