Ephemeral container not added to pod

[ErikEngerd]

What happened (please include outputs or screenshots):

Trying to add an ephemeral container to a pod. The code executes successfully but no container is added. 'kubectl debug' on the other hand works fine.

What you expected to happen:

ephemeral container added to the pod

How to reproduce it (as minimally and precisely as possible):

from kubernetes import client, config

config.load_kube_config()
v1 = client.CoreV1Api()

NAMESPACE = 'exposure'
PODPATTERN = 'httpd-wamblee-org'

# find a pod
ret = v1.list_namespaced_pod(NAMESPACE)
container = next(p for p in ret.items if p.metadata.name.startswith(PODPATTERN))
print(f"Found: {container.metadata.name} {container.metadata.namespace}")

# add a debug container to it
body = client.models.V1EphemeralContainer(
    name="debugger",
    image = "centos:7",
    stdin=True,
    tty=True)
response = v1.patch_namespaced_pod_ephemeralcontainers(
    container.metadata.name,
    container.metadata.namespace,
    body)```

**Anything else we need to know?**:

The command `kubectl debug -it --image centos:7 httpd-wamblee-org-64d65d4696-j4xmn` works. 

**Environment**:
- Kubernetes version (`kubectl version`): 1.24.3
- OS (e.g., MacOS 10.13.6): Centos 8
- Python version (`python --version`): 3.8.0
- Python client version (`pip list | grep kubernetes`): 24.2.0

[ErikEngerd]

I have done some extensive debugging in pycharm and it looks like everything is ok (including the most important part, which is the request body). However, when running 'kubectl debug', I see that kubectl is not using this patch request but generates a strategic merge path instead.

This is the command which I ran:

kubectl debug -it --image=centos:7 httpd-wamblee-org-64d65d4696-2lpn6 -v=8

Now it uses the following URL:

I0718 22:52:07.673282    2432 round_trippers.go:463] PATCH https://192.168.178.89:6443/api/v1/namespaces/exposure/pods/httpd-wamblee-org-64d65d4696-2lpn6/ephemeralcontainers

which is the same as the python code

https://192.168.178.89:6443/api/v1/namespaces/exposure/pods/httpd-wamblee-org-64d65d4696-2lpn6/ephemeralcontainers

However, the body differs since kubectl is creating a strategic merge patch yaml:

I0718 22:52:07.673082    2432 debug.go:415] generated strategic merge patch for debug container: {"spec":{"ephemeralContainers":[{"image":"centos:7","name":"debugger-tqh4v","resources":{},"stdin":true,"terminationMessagePolicy":"File","tty":true}]}}
I0718 22:52:07.673190    2432 request.go:1073] Request Body: {"spec":{"ephemeralContainers":[{"image":"centos:7","name":"debugger-tqh4v","resources":{},"stdin":true,"terminationMessagePolicy":"File","tty":true}]}}

It then uses the strategic merge patch but the python code is only sending the .spec.ephemeralCOntainers part of the patch.

So this is definitely the difference between the two. This makes me wonder whether it would not be simpler to use a regular patch on the pod to add the ephemeral container to it.

[ErikEngerd]

I just tried with a kubectl patch command and this does not work since updates to the ephemeralContainers are not allowed through a patch. So it seems that indeed the .../ephemeralcontainers endpoint must be used.

[ErikEngerd]

By the way, the version of kubernetes that I am running in the cluster is 1.23.7 so I assume I would need the 23.6.0 version of the kubernetes API. Trying that however I get the same issue.

What I see is when I request the swagger.json for my kubernetes using

kubectl proxy --port=8080
# other terminal
curl localhost:8080/openapi/v2| jq . -  > swaggernew.json

that I get a difference for the PATCH request for /api/v1/namespaces/{namespace}/pods/{name}/ephemeralcontainers that is needed here. The 23.6.0 version of the python kubernetes library has

"patch": {
        "consumes": [
          "application/json-patch+json",
          "application/merge-patch+json",
          "application/strategic-merge-patch+json",
          "application/apply-patch+yaml"
        ],
        "description": "partially update ephemeralcontainers of the specified Pod",
        "operationId": "patchNamespacedPodEphemeralcontainers",
        "parameters": [
          {
            "in": "body",
            "name": "body",
            "required": true,
            "schema": {
              "description": "Patch is provided to give a concrete name and type to the Kubernetes PATCH request body.",
              "type": "object"
            }
          },

whereas my kubernetes cluster gives me

"patch": {
        "description": "partially update ephemeralcontainers of the specified Pod",
        "consumes": [
          "application/json-patch+json",
          "application/merge-patch+json",
          "application/strategic-merge-patch+json",
          "application/apply-patch+yaml"
        ],
        "produces": [
          "application/json",
          "application/yaml",
          "application/vnd.kubernetes.protobuf"
        ],
        "schemes": [
          "https"
        ],
        "tags": [
          "core_v1"
        ],
        "operationId": "patchCoreV1NamespacedPodEphemeralcontainers",
        "parameters": [
          {
            "name": "body",
            "in": "body",
            "required": true,
            "schema": {
              "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.Patch"
            }
          },

in other words, the schema in the kubernetes python library mentions schema type 'object' which probably serializes the object using the fields, but my kubernetes cluster refers to the Patch schema. The latter is also what I see kubectl doing when it invokes the request.

[ErikEngerd]

I am stuck now trying to create a new version of the library with the swagger.json from my kubernetes cluster. It is not clear what I need to do in order to build. First I am running into the fact that it uses a raw github URL to fetch the swagger.json. I can work around that, but after that it fails with a SyntaxError saying

--- Downloading and pre-processing OpenAPI spec
  File "//preprocess_spec.py", line 15
    from __future__ import print_function
    ^
SyntaxError: from __future__ imports must occur at the beginning of the file

Is there an instruction somewhere on the complete set of requirements and procedures to build the package?

[jooppey]

I am getting the same issue using patch_namespaced_pod_ephemeralcontainers

with Python client v22.6.0 or v23.6.0 to a v1.20.4 cluster API -> OK

• ephemeralContainer is created,
• patch_namespaced_pod_ephemeralcontainers returns an object with kind 'EphemeralContainers'

  'api_version': 'v1',
  'kind': 'EphemeralContainers',
  'metadata': {...},
  'spec': None,
  'status': None}
  

with Python client v22.6.0 or v23.6.0 to a v1.23.5 cluster API -> fails

• no ephemeralContainer created,
• patch_namespaced_pod_ephemeralcontainers returns an object with kind 'Pod'

  'api_version': 'v1',
  'kind': 'Pod',
  'metadata': {...},
  ...
  }
  

• kubectl debug works

[ErikEngerd]

I have the feeling that the upstream swagger.json from kubernetes is incorrect. When I look at the definition of #/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.Patch, then I also see type = object, which is the same as in the swagger.json that the python client is using.

I am still having problems building the python library though. The key step is to fork the kubernetes repository in github, and then to execute update-clients from the scripts directory. There I have to specify my github username in lower case, like this:

USERNAME=erikengerd ./update-client.sh

Then, however, building and installing the package fails:

> python3 setup.py  install
running install
/home/erik/miniconda3/envs/kpython/lib/python3.8/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
/home/erik/miniconda3/envs/kpython/lib/python3.8/site-packages/setuptools/command/easy_install.py:144: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
running bdist_egg
running egg_info
writing kubernetes.egg-info/PKG-INFO
writing dependency_links to kubernetes.egg-info/dependency_links.txt
writing requirements to kubernetes.egg-info/requires.txt
writing top-level names to kubernetes.egg-info/top_level.txt
error: package directory 'kubernetes/client/apis' does not exist

Building the library is essential to test changes to the swagger. First step is to test the current kubernetes swagger.json and if it fails to modify the swagger.json.

Am I following the correct steps for building the python client?

[ErikEngerd]

Looking at the output of the update-client.sh I see that this also fails at the very last statement trying to patch rest.py

>>> patching client...
fatal: corrupt patch at line 18

[ErikEngerd]

Just tried to load the swagger.json in SwaggerUI for the 1.23, 1.24, and master branches of the kubernetes project on github. It seems that the request schema is not included in the openapi spec.

[ErikEngerd]

After digging around a bit in the source code I have found a workaround.

What you do is simply create the requested body by wrapping the body object and passing that to the API call instead of the original body object.

body = client.models.V1EphemeralContainer(image="centos:7", name="debugger")
body = {
    "spec": {
        "ephemeralContainers": [
            body.to_dict()
        ]
    }
}
res = corev1.patch_namespaced_pod_ephemeralcontainers(pod.metadata.name, pod.metadata.namespace, body, _preload_content=False)

The magic is in the line where the body is modified and in the body.to_json() since the sanitizer, which creates the data structure that is used as the body, does not understand objects but does understand dictionaries.

[ErikEngerd]

Most likely the fix should be in the V1EphemeralContainer class but since this is generated from the swagger.json, the change should probably be in the swagger.json then. I would like some advice from one of the maintainers on how to proceed with this. Should I write a PR on the kubernetes project?

[pbarker]

The magic is in the line where the body is modified and in the body.to_json() since the sanitizer, which creates the data structure that is used as the body, does not understand objects but does understand dictionaries.

This works for me except not with the volume mounts for some reason. Has anyone gotten any further on this?

[ErikEngerd]

This works for me except not with the volume mounts for some reason. Has anyone gotten any further on this?

Do you have a code sample?

[pbarker]

Hey @ErikEngerd heres the code sample:

        spec = V1EphemeralContainer(
            name=f"snapshot-{int(time.time())}",
            image="gcr.io/kaniko-project/executor:latest",
            args=[
                f"--context={REPO_ROOT}",
                f"--destination={uri}",
            ],
            volume_mounts=[
                V1VolumeMount(name="dockercfg", mount_path="/kaniko/.docker/"),
                V1VolumeMount(name="build", mount_path=REPO_ROOT),
            ],
        )

        body = {"spec": {"ephemeralContainers": [spec.to_dict()]}}

        ret = core_v1_api.patch_namespaced_pod_ephemeralcontainers(
            self.pod_name,
            self.pod_namespace,
            body,
            _preload_content=False,
        )

This succeeds and the ephemeral container is added but without any volume mounts or messages as to why

[pbarker]

ah okay this is because its sending the dictionary as snake_case and k8s is expecting camelCase, solved this by just making a raw dictionary with the proper casing

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.