apiserver fails json patches with 422 when patch body does not match default json.Marshal encoding

[shadjiiski]

Issue description We are using a modified version of the python's official client for kubernetes. I was investigating a bug where JSON patches to custom resource with "test" operation would fail with 422 Unprocessable Entity.

From my investigation, this appears to be caused by a combination of these:

• To apply the json patch, the api-server serializes current contents as json with json.Marshal() which escapes <,>,& and does not escape non-ascii unicodes
• The client library uses json.dumps() which does not escape <,>,& and escapes non-ascii characters
• Api-server then uses json-patch library to execute the patch. That library appears to compare patch body and current contents byte to byte, ignoring the possibility to have the same content encoded in 2 different ways - see https://github.com/evanphx/json-patch/issues/172#issuecomment-1814307350

JSON patches work fine from kubectl as it uses same json.Marshal to prepare the patch body. However, issue can be easily hit with curl or any other non-golang client.

Kubernetes version (outdate but code in repo appears to be the same)

$ kubectl version Client Version: version.Info{Major:"1", Minor:"20+", GitVersion:"v1.20.11-1+1f8a47eae6d024-dirty", GitCommit:"1f8a47eae6d0246ddca52cf925d15e5f88ec876a", GitTreeState:"dirty", BuildDate:"2023-10-23T12:01:36Z", GoVersion:"go1.21.3 X:boringcrypto", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"20+", GitVersion:"v1.20.11-1+1f8a47eae6d024-dirty", GitCommit:"1f8a47eae6d0246ddca52cf925d15e5f88ec876a", GitTreeState:"dirty", BuildDate:"2023-10-23T12:01:36Z", GoVersion:"go1.21.3 X:boringcrypto", Compiler:"gc", Platform:"linux/amd64"}

Repro steps

# create foo crd
kubectl apply -f - <<EOF
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: foos.bar.example.com
spec:
  group: bar.example.com
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                baz:
                  type: string
  scope: Namespaced
  names:
    plural: foos
    singular: foo
    kind: Foo
EOF

# create myfoo foo
kubectl apply -f - <<EOF
---
apiVersion: bar.example.com/v1
kind: Foo
metadata:
  name: myfoo
spec:
  baz: "<&>β"
EOF

# This will work:
kubectl patch foo myfoo --type json -p '[{"op":"test", "path":"/spec/baz", "value":"<&>\u03b2"}]'

# This will fail with 422
curl -kv -X PATCH \
  -d '[{"op":"test", "path":"/spec/baz", "value":"<&>\u03b2"}]' \
  -H "Accept: application/json" \
  -H "Content-Type: application/json-patch+json" \
  'http://localhost:8483/apis/bar.example.com/v1/namespaces/default/foos/myfoo'

# Workaround: this will work because it matches byte to byte the result of json.Marshal
curl -kv -X PATCH \
  -d '[{"op":"test", "path":"/spec/baz", "value":"\u003c\u0026\u003eβ"}]' \
  -H "Accept: application/json" \
  -H "Content-Type: application/json-patch+json" \
  'http://localhost:8483/apis/bar.example.com/v1/namespaces/default/foos/myfoo'

Workaround Find a way to force your k8s client to encode the patch body in the exact same manner that golang's json.Marshal would do

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.