K8SSAND-1585 ⁃ k8ss operator does not produce any logs/events in an event of the reaper deployment failure

[andrey-dubnik]

Hi, just got into the issue while enabling Reaper due to the reaper deployment not having the resources defined due to our K8S cluster having OPA gatekeeper policy enforcing the requirement.

Behaviour is following:

1. Operator is stuck in Deploying reaper resource forever without throwing any errors
2. Once limit enforcement is suspended on the gatekeeper Reaper is able to proceed

Reaper Object/Operator should have some sort of log/event/status displayed in the event of the the faulty deployments of the Reaper resource

This is the reaper got stuck for hours state k get reapers.reaper.k8ssandra.io -o yaml

apiVersion: v1
items:
- apiVersion: reaper.k8ssandra.io/v1alpha1
  kind: Reaper
  metadata:
    annotations:
      k8ssandra.io/resource-hash: KYe14Itq609cgc+O66a43OKVRVq088Ev5fGgqOIYtCU=
    creationTimestamp: "2022-06-13T14:35:48Z"
    generation: 2
    labels:
      app.kubernetes.io/component: reaper
      app.kubernetes.io/created-by: k8ssandracluster-controller
      app.kubernetes.io/managed-by: k8ssandra-operator
      app.kubernetes.io/name: k8ssandra-operator
      app.kubernetes.io/part-of: k8ssandra
      k8ssandra.io/cluster-name: dev-westeurope-01
      k8ssandra.io/cluster-namespace: temporal-state
    name: dev-westeurope-01-primary-reaper
    namespace: temporal-state
    resourceVersion: "210319458"
    uid: 6125bead-67b9-47ca-ace5-d6a1462bec5b
  spec:
    ServiceAccountName: default
    autoScheduling:
      enabled: true
      initialDelayPeriod: PT15S
# Please edit the object below. Lines beginning with a '#' will be ignored,
      percentUnrepairedThreshold: 10
      periodBetweenPolls: PT10M
      repairType: AUTO
      scheduleSpreadPeriod: PT6H
      timeBeforeFirstSchedule: PT5M
    cassandraUserSecretRef:
      name: cassandra-reaper-cql
    containerImage:
      name: cassandra-reaper
      registry: docker.io
      repository: thelastpickle
      tag: 3.1.1
    datacenterAvailability: ALL
    datacenterRef:
      name: primary
      namespace: temporal-state
    heapSize: 2Gi
    initContainerImage:
      name: cassandra-reaper
      registry: docker.io
      repository: thelastpickle
      tag: 3.1.1
    jmxUserSecretRef:
      name: cassandra-reaper-jmx
    keyspace: reaper_db
    skipSchemaMigration: false
    uiUserSecretRef:
      name: cassandra-reaper-ui
  status:
    conditions:
    - lastTransitionTime: "2022-06-17T09:01:18Z"
      status: "False"
      type: Ready
    progress: Deploying
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

┆Issue is synchronized with this Jira Story by Unito