500 errors when publishing events

[Carles-Figuerola]

Long story short

kopf is throwing 500 errors in a queuein event after the handler has finished running

Description

I have created an operator with two handlers, one for iam roles and one for iam profiles. For some reason, the iam profile event handler returns a 500 error when trying to post an event.

[2020-01-09 20:05:27,497] kopf.objects         [INFO    ] [default/101-example-policy] Handler 'on_policy_create' succeeded.
[2020-01-09 20:05:27,498] kopf.objects         [INFO    ] [default/101-example-policy] All handlers succeeded for creation.
[2020-01-09 20:05:28,592] kopf.reactor.queuein [ERROR   ] functools.partial(<function resource_handler at 0x7f6cc3ba99e0>, lifecycle=<function asap at 0x7f6cc3ba4950>,
registry=<kopf.toolkits.legacy_registries.SmartGlobalRegistry object at 0x7f6cc3476e10>, memories=<kopf.structs.containers.ResourceMemories object at 0x7f6cc3944a90>, resource=Resource(group='mydomain.com', version='v1', plural='iampolicies'), event_queue=<Queue at 0x7f6cc38b8990 maxsize=0 _getters[1] tasks=18>, freeze=<asyncio.locks.Event object at 0x7f6cc34a9710 [unset]>) failed with an exception. Ignoring the event.
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/kopf/reactor/queueing.py", line 173, in worker
    await handler(event=event, replenished=replenished)
  File "/usr/local/lib/python3.7/site-packages/kopf/reactor/handling.py", line 229, in resource_handler
    await patching.patch_obj(resource=resource, patch=patch, body=body)
  File "/usr/local/lib/python3.7/site-packages/kopf/clients/auth.py", line 46, in wrapper
    return await fn(*args, **kwargs, session=session)
  File "/usr/local/lib/python3.7/site-packages/kopf/clients/patching.py", line 54, in patch_obj
    raise_for_status=True,
  File "/usr/local/lib/python3.7/site-packages/aiohttp/client.py", line 588, in _request
    resp.raise_for_status()
  File "/usr/local/lib/python3.7/site-packages/aiohttp/client_reqrep.py", line 946, in raise_for_status
    headers=self.headers)
aiohttp.client_exceptions.ClientResponseError: 500, message='Internal Server Error', url=URL('https://172.20.0.1:443/apis/mydomain.com/v1/namespaces/default/iampolicies/101-example-policy')

The operator has a cluster role with these permissions:

rules:
- apiGroups:
  - mydomain.com
  resources:
  - iamroles
  - iampolicies
  verbs:
  - '*'
- apiGroups:
  - ''
  resources:
  - events
  verbs:
  - '*'

Is there some tips on what I might be doing wrong? The code / CRD are all basically the same with the name changed so it should work the same way.

Thanks!

[nolar]

Which version of Kubernetes do you use?

As a quick-guess, try using explicit verbs instead of a star.

  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]

The RBAC documentation shows no examples with a star, so I am no sure this will work at all.

---

Generally, try doing the same from CLI:

kubectl patch iampolicies -n default 101-example-policy --type merge -p '{"status": {"xyz": "abc"}}'

It is Kopf-unrelated, and it should succeed.

PS: From inside of the operator's pod, not under your personal (probably admin) account.