Root task 'watcher of kopfexamples.zalando.org' is failed: 403, message='Forbidden' - EKS

[yahavb]

Question

I am trying to run the first example on EKS. The first example fails with Forbidden error when accessing the API server. I am not sure where kopf takes its creds from.

I followed https://kopf.readthedocs.io/en/latest/install/ after cloning the kopf repo

[kopf]$kopf run examples/01-minimal/example.py 
[2020-07-08 19:33:25,754] kopf.reactor.activit [INFO    ] Initial authentication has been initiated.
[2020-07-08 19:33:25,783] kopf.activities.auth [INFO    ] Activity 'login_via_pykube' succeeded.
[2020-07-08 19:33:25,784] kopf.reactor.activit [INFO    ] Initial authentication has finished.
[2020-07-08 19:33:25,913] kopf.engines.peering [WARNING ] Default peering object not found, falling back to the standalone mode.
[2020-07-08 19:33:25,934] kopf.reactor.running [ERROR   ] Root task 'watcher of kopfexamples.zalando.org' is failed: 403, message='Forbidden', url=URL('https://d69c740c8494333e5dcd7b23b0af0ee0.gr7.us-west-2.eks.amazonaws.com/apis/zalando.org/v1/kopfexamples')
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/kopf/reactor/running.py", line 453, in _root_task_checker
    await coro
  File "/usr/local/lib/python3.7/site-packages/kopf/reactor/queueing.py", line 109, in watcher
    async for raw_event in stream:
  File "/usr/local/lib/python3.7/site-packages/kopf/clients/watching.py", line 75, in infinite_watch
    async for raw_event in stream:
  File "/usr/local/lib/python3.7/site-packages/kopf/clients/watching.py", line 112, in streaming_watch
    async for raw_event in stream:
  File "/usr/local/lib/python3.7/site-packages/kopf/clients/watching.py", line 130, in continuous_watch
    items, resource_version = await fetching.list_objs_rv(resource=resource, namespace=namespace)
  File "/usr/local/lib/python3.7/site-packages/kopf/clients/auth.py", line 45, in wrapper
    return await fn(*args, **kwargs, context=context)
  File "/usr/local/lib/python3.7/site-packages/kopf/clients/fetching.py", line 101, in list_objs_rv
    response.raise_for_status()
  File "/Users/birayaha/Library/Python/3.7/lib/python/site-packages/aiohttp/client_reqrep.py", line 946, in raise_for_status
    headers=self.headers)
aiohttp.client_exceptions.ClientResponseError: 403, message='Forbidden', url=URL('https://d69c740c8494333e5dcd7b23b0af0ee0.gr7.us-west-2.eks.amazonaws.com/apis/zalando.org/v1/kopfexamples')
[2020-07-08 19:33:25,942] kopf.reactor.running [WARNING ] Root task 'daemon killer' is finished unexpectedly.
Traceback (most recent call last):
  File "/usr/local/bin/kopf", line 8, in <module>
    sys.exit(main())
  File "/Users/birayaha/Library/Python/3.7/lib/python/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/Users/birayaha/Library/Python/3.7/lib/python/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/Users/birayaha/Library/Python/3.7/lib/python/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/birayaha/Library/Python/3.7/lib/python/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/birayaha/Library/Python/3.7/lib/python/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/kopf/cli.py", line 36, in wrapper
    return fn(*args, **kwargs)
  File "/Users/birayaha/Library/Python/3.7/lib/python/site-packages/click/decorators.py", line 73, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "/Users/birayaha/Library/Python/3.7/lib/python/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/kopf/cli.py", line 87, in run
    vault=__controls.vault,
  File "/usr/local/lib/python3.7/site-packages/kopf/reactor/running.py", line 117, in run
    vault=vault,
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/base_events.py", line 587, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.7/site-packages/kopf/reactor/running.py", line 161, in operator
    await run_tasks(operator_tasks, ignored=existing_tasks)
  File "/usr/local/lib/python3.7/site-packages/kopf/reactor/running.py", line 370, in run_tasks
    await _reraise(root_done | root_cancelled | hung_done | hung_cancelled)
  File "/usr/local/lib/python3.7/site-packages/kopf/reactor/running.py", line 437, in _reraise
    task.result()  # can raise the regular (non-cancellation) exceptions.
  File "/usr/local/lib/python3.7/site-packages/kopf/reactor/running.py", line 453, in _root_task_checker
    await coro
  File "/usr/local/lib/python3.7/site-packages/kopf/reactor/queueing.py", line 109, in watcher
    async for raw_event in stream:
  File "/usr/local/lib/python3.7/site-packages/kopf/clients/watching.py", line 75, in infinite_watch
    async for raw_event in stream:
  File "/usr/local/lib/python3.7/site-packages/kopf/clients/watching.py", line 112, in streaming_watch
    async for raw_event in stream:
  File "/usr/local/lib/python3.7/site-packages/kopf/clients/watching.py", line 130, in continuous_watch
    items, resource_version = await fetching.list_objs_rv(resource=resource, namespace=namespace)
  File "/usr/local/lib/python3.7/site-packages/kopf/clients/auth.py", line 45, in wrapper
    return await fn(*args, **kwargs, context=context)
  File "/usr/local/lib/python3.7/site-packages/kopf/clients/fetching.py", line 101, in list_objs_rv
    response.raise_for_status()
  File "/Users/birayaha/Library/Python/3.7/lib/python/site-packages/aiohttp/client_reqrep.py", line 946, in raise_for_status
    headers=self.headers)
aiohttp.client_exceptions.ClientResponseError: 403, message='Forbidden', url=URL('https://d69c740c8494333e5dcd7b23b0af0ee0.gr7.us-west-2.eks.amazonaws.com/apis/zalando.org/v1/kopfexamples')

Checklist

• [x ] I have read the documentation and searched there for the problem
• [ x] I have searched in the GitHub Issues for similar questions

Keywords

[yahavb]

It is not clear if kopf run should be executed from my client (MacOS) or deployed as a container so the permissions are taken from the pod service account or the node IAM role. Where kops is taking its permissions to run?

[magebeans]

It takes its permissions from the service account associated with it; those creds won't be on your local client, only on the container in the pod that you deploy it to (assuming the pod is set up with a service account that has the right role bindings to a role with sufficient permissions). You have to deploy it, running locally won't work on anything but minikube.