Running the Tutorial against an EKS cluster

[jicowan]

Expected Behavior

I would like to run the kopf tutorial on an EKS cluster instead of minikube.

Actual Behavior

When I run kopf run <file>.py --verbose I get an unauthorized error: pykube.exceptions.HTTPError: customresourcedefinitions.apiextensions.k8s.io "kopfpeerings.zalando.org" is forbidden: User "system:anonymous" cannot get resource "customresourcedefinitions" in API group "apiextensions.k8s.io" at the cluster scope

I see that kopf is using the pykube library which doesn't natively support IAM authentication. Is there a way to make this work without using a bearer token for authentication?

Specifications

• Platform: EKS
• Kubernetes version: (use kubectl version) 1.13
• Python version: (use python --version) 3.7.4
• Python packages installed: (use pip freeze --all) aiohttp==3.6.0 aiojobs==0.2.2 async-timeout==3.0.1 attrs==19.1.0 cachetools==3.1.1 certifi==2019.9.11 chardet==3.0.4 Click==7.0 google-auth==1.6.3 idna==2.8 iso8601==0.1.12 kopf==0.21 kubernetes==10.0.1 multidict==4.5.2 oauthlib==3.1.0 pip==19.0.3 pyasn1==0.4.7 pyasn1-modules==0.2.6 pykube-ng==19.9.2 python-dateutil==2.8.0 PyYAML==5.1.2 requests==2.22.0 requests-oauthlib==1.2.0 rsa==4.0 setuptools==40.8.0 six==1.12.0 urllib3==1.25.3 websocket-client==0.56.0 yarl==1.3.0

[nolar]

Kopf uses pykube-ng (which is a fork of pykube). Beside token-bearer, it also supports client certificate, username-password, and some other auth methods. At least, it is in the code.

I will try to reproduce the error locally for EKS. Maybe @hjacobs can help here too.

---

Anyway, for the record, the mid-term plans for Kopf are to make @kopf.on.login(...) as a user-provided handler/hook, and let the users implement their own authentication methods — with kubernetes client and pykube-ng pre-implemented as examples in the implicit "piggybacking" mode (extracting the auth info from them). The "piggybacking" approach is partially implemented, but not yet carefully tested, in #176 (no handlers yet).

At the moment, there are no other ways to override the built-in authentication, except as by monkey-patching kopf.clients.auth.get_pykube_cfg()/get_pykube_api() and returning your own object as it would be in kubectl-compatible ~/.kube/config (see examples). The best place is at the operator's module level together with the handlers — this is where we do this for our own complicated setup of one of the operators).

import kopf

def my_auth_cfg():
    return pykube.KubeConfig({
        ...  # as in ~/.kube/config
        "clusters": [...],
        "users": [...],
        "contexts": [...],
        "current-context": "...",
    })

kopf.clients.auth.get_pykube_cfg = my_auth_cfg

@kopf.on.event('', 'v1', 'pods')
def pod_fn(**_):
    pass

BEWARE: This is not a documented and official way: one day, this function will just go away without warning.

[hjacobs]

There is a simple workaround: just use kubectl proxy locally and point Kopf/Pykube to the localhost proxy port, see https://kube-web-view.readthedocs.io/en/latest/setup.html#local-usage as an example.

[jdamata]

@hjacobs I tried this approach but can't seem to get get it working:

dockerfile:

FROM python:3.7-alpine AS build-env
COPY Pipfile .
COPY Pipfile.lock .
RUN pip install pipenv && pipenv install --system --deploy
ENTRYPOINT ["kopf", "run", "sop-operator.py"]

code

# K8s authentication
if 'KUBERNETES_PORT' in os.environ:
    config.load_incluster_config()
else:
    config.load_kube_config()
    kopf.clients.auth.login_pykube = pykube.KubeConfig.from_url('http://localhost:8001'))

commands

kubectl proxy --accept-hosts '.*' --port=8001 &
docker build . --tag sop-operator-local
docker run sop-operator-local

[jicowan]

Thanks @hjacobs. I added a section to my code so I could test it locally instead of running in the cluster.

[nolar]

A little informational update:

A normal non-hacky solution is coming soon. The work-in-progress code is already functional in my own fork, the examples are runnable, though there is no test coverage yet, and it is still with some WIP garbage.

You can preview it in my branch in this doc: https://github.com/nolar/kopf/blob/authentication/docs/authentication.rst, and these two login implementations: https://github.com/nolar/kopf/blob/authentication/kopf/utilities/piggybacking.py#L67.

It will take some time to go through a chain of 3-5 PRs and reviews, but the normal solution with @kopf.on.login() handlers is on its way. After that, any arbitrary authentication methods can be implemented as needed (GKE, EKS, etc).

[jicowan]

That's great news. Thanks for the update!

[Dmitry1987]

@nolar it will be merged into this official kopf repo? good job and thanks for working on this 👍 I'm struggling with local development because of authentication issues.

[nolar]

@jicowan @Dmitry1987 The custom authentication methods were merged and released as kopf==0.23rc1 (see the release notes — there are many other changes). If the live testing goes well, it will be released as 0.23 in few days.

The documentation for custom authentication methods is here: https://kopf.readthedocs.io/en/latest/authentication/

An example: https://github.com/zalando-incubator/kopf/blob/master/examples/13-hooks/example.py (see login_fn).

Please, let me know if this solution helps to resolve the original issue with IAM/EKS, and if the new release candidate works well in that environment.