Support User Namespaces in pods

[derekwaynecarr]

trafficstars

Enhancement Description

• One-line enhancement description (can be used as a release note): Add support for user namespaces in pods
• Kubernetes Enhancement Proposal: https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/127-user-namespaces/README.md
• Discussion Link: This was discussed in several sig-node meetings, most notable Nov 2 2021 and in this PR, that feedback was incorporated already by this KEP is taking: https://github.com/kubernetes/enhancements/pull/2101.
• Primary contact (assignee): @rata @giuseppe
• Responsible SIGs: sig-node
• Enhancement target (which target equals to which milestone):
  • Alpha release target (x.y): 1.25
  • Beta1 release target (x.y): 1.30
  • Beta2 release target (x.y): 1.33
  • Stable release target (x.y):
• [x] Alpha
  • [x] KEP (k/enhancements) update PR(s):
    • [x] v1.24 https://github.com/kubernetes/enhancements/pull/3065
    • [x] v1.25 https://github.com/kubernetes/enhancements/pull/3275
    • [x] v1.27 https://github.com/kubernetes/enhancements/pull/3811
    • [x] v1.28 https://github.com/kubernetes/enhancements/pull/4044
    • [x] v1.28 https://github.com/kubernetes/enhancements/pull/4084
    • [x] v1.28 https://github.com/kubernetes/enhancements/pull/4107
    • [x] v1.28 https://github.com/kubernetes/enhancements/pull/4147
  • [x] Code (k/k) update PR(s):
    • [x] v1.27 https://github.com/kubernetes/kubernetes/pull/116377
    • [x] v1.28 https://github.com/kubernetes/kubernetes/pull/118691
    • [x] v1.29 https://github.com/kubernetes/kubernetes/pull/118760
  • [x] Docs (k/website) update PR(s):
    • [x] v1.27 https://github.com/kubernetes/website/pull/39860
    • [x] v1.28 https://github.com/kubernetes/website/pull/41908 (v1.28)
    • [x] v1.29 https://github.com/kubernetes/website/pull/43803 (v1.29)
• [x] Beta 1 （default off）
  • [x] KEP (k/enhancements) update PR(s):
    • [x] https://github.com/kubernetes/enhancements/pull/4439
  • [x] Code (k/k) update PR(s):
    • [x] https://github.com/kubernetes/kubernetes/pull/123216
    • [x] https://github.com/kubernetes/kubernetes/pull/123593
    • [x] https://github.com/kubernetes/kubernetes/pull/121606
  • [x] Docs (k/website) update(s):
    • [x] https://github.com/kubernetes/website/pull/45354
    • [x] https://github.com/kubernetes/website/pull/45178
• [x] Beta 2 （default on）
  • [x] KEP (k/enhancements) update PR(s):
    • [X] https://github.com/kubernetes/enhancements/pull/5118
    • [x] https://github.com/kubernetes/enhancements/pull/5020
    • [x] https://github.com/kubernetes/enhancements/pull/5141
    • [ ] https://github.com/kubernetes/enhancements/pull/5413
  • [x] Code PRs for 1.33:
    • [x] https://github.com/kubernetes/kubernetes/pull/130138
    • [x] https://github.com/kubernetes/kubernetes/pull/130028
    • [x] https://github.com/kubernetes/kubernetes/pull/130243
    • [x] https://github.com/kubernetes/kubernetes/pull/130726
    • [x] https://github.com/kubernetes/kubernetes/pull/130800
    • [x] https://github.com/kubernetes/kubernetes/pull/130820.
  • [x] Doc PR for 1.33:
    • [x] https://github.com/kubernetes/website/pull/49749

Please keep this description up to date. This will help the Enhancement Team to track the evolution of the enhancement efficiently.

[derekwaynecarr]

This work is being done by @pweil- and is reviewed by @derekwaynecarr, it is sponsored by @kubernetes/sig-node

[mdshuai]

@derekwaynecarr Could you help create a user story card for this feature?

[idvoretskyi]

@derekwaynecarr can you confirm that this feature targets alpha for 1.5?

[pweil-]

@derekwaynecarr can you confirm that this feature targets alpha for 1.5?

Yes, this feature is experimental only so it would be considered alpha.

[idvoretskyi]

@derekwaynecarr @pweil- can you confirm that this item targets beta in 1.6?

[adelton]

@derekwaynecarr, the proposal https://github.com/kubernetes/kubernetes/pull/34569 was closed by bot due to inactivity.

@pweil-, in https://github.com/kubernetes/kubernetes/pull/34569#issuecomment-273531120 you've proposed the approach https://github.com/pweil-/kubernetes/commit/16f29ebb076dfa3c44c7c4669d55fe21c206e149 which changes the group of /var/lib/kubelet/pods to the remapped root group. Do I understand it correctly that this is currently not tracked in any pull request?

[adelton]

@pweil-, I also wonder if similar to docker's /var/lib/docker/<uid>.<gid> approach when --userns-remap is used, it might make sense to use /var/lib/kubelet/pods-<uid>.<gid> and just chown/chgroup everything in those subdirectories to the remapped <uid>.<gid>. Why did you opt for just the chgrp and not the full chown?

[pweil-]

@adelton in the end, I think having this be transparent to Kubernetes is the right approach. Whether that be something like shiftfs or implementation in the CRI (https://github.com/moby/moby/issues/28593). You are correct that my existing proposal is not currently tracked in an open PR anymore.

The reasoning behind using the chgrp was to follow our fsgroup strategy where we just ensure group access instead of uid access.

[adelton]

Thanks @pweil-.

When you say transparent, you mean that nothing should be needed to be added to code or to configuration on Kubernetes' side to allow running under docker with userns-remap?

As for the fsgroup strategy, do you mean https://kubernetes.io/docs/concepts/policy/pod-security-policy/#fsgroup or some generic methodology within Kubernetes?

I have now filed https://github.com/kubernetes/kubernetes/pull/55707 as an alternative approach where I make the remapped uid/gid an explicit option, and use those values to chown/chgrp the necessary directories.

[pweil-]

When you say transparent, you mean that nothing should be needed to be added to code or to configuration on Kubernetes' side to allow running under docker with userns-remap?

that would be ideal. Whether that is feasible (or more likely, feasible in an acceptable time frame) is another question :smile:

As for the fsgroup strategy, do you mean https://kubernetes.io/docs/concepts/policy/pod-security-policy/#fsgroup or some generic methodology within Kubernetes?

Yes

I have now filed kubernetes/kubernetes#55707 as an alternative approach where I make the remapped uid/gid an explicit option, and use those values to chown/chgrp the necessary directories.

:+1: subscribed

[adelton]

When you say transparent, you mean that nothing should be needed to be added to code or to configuration on Kubernetes' side to allow running under docker with userns-remap?

that would be ideal. Whether that is feasible (or more likely, feasible in an acceptable time frame) is another question

Ideally, the pod would specify how many distinct uids/gids it would require / list of uids it wants to see inside of the containers, and docker or different container runtime would setup the user namespace accordingly. But unless docker also changes ownership of the volumes mounted to the containers, Kubernetes will have to do that as part of the setup.

[adelton]

@pwel-, what is the best way to get some review and comments on https://github.com/kubernetes/kubernetes/pull/55707, to get it closer to mergeable state?

[0xmichalis]

@pweil- ^

[pweil-]

@adelton I would try to engage the sig-node folks either at their Tuesday meeting or on slack: https://github.com/kubernetes/community/tree/master/sig-node

[adelton]

@derekwaynecarr, could you please bring https://github.com/kubernetes/kubernetes/pull/55707 to sig-node's radar?

[idvoretskyi]

@pweil- @derekwaynecarr any progress on this feature is expected?

[derekwaynecarr]

i will raise this topic in k8s 1.11 planning for sig-node.

[Quentin-M]

Just leaving a note here..

Because most Kubernetes deployments have kube-system services that require root privileges on the host (e.g. overlay networks), we will need to support --userns=host, and extend PodSecurityPolicies with a a permission around using its usage.

Edit: Although I assume that allowing pid=host would include allowing userns=host too. Edit: Alright, that is actually exactly: https://github.com/kubernetes/kubernetes/pull/31169.

[php-coder]

we will need to support --userns=host, and extend PodSecurityPolicies with a a permission around using its usage

PSP already has such support: see HostNetwork

[justaugustus]

@pweil- @derekwaynecarr Any plans for this in 1.11?

If so, can you please ensure the feature is up-to-date with the appropriate:

• Description
• Milestone
• Assignee(s)
• Labels:
  • stage/{alpha,beta,stable}
  • sig/*
  • kind/feature

cc @idvoretskyi

[pweil-]

@justaugustus I am not actively working this so I'll defer to @adelton. @adelton can you comment on roadmap here? Thanks!

[adelton]

We are working for KEP for sig-node.

[derekwaynecarr]

@justaugustus

I have done the following:

1. updated description.
2. assigned the 1.11 milestone per sig-node planning
3. updated assignees from sig-node community
4. the feature is planned for stage/alpha in 1.11
5. linked to the proposal under discussion in 4/24 sig-node meeting.

[justaugustus]

@derekwaynecarr thanks for the update!

[adelton]

@derekwaynecarr, the https://github.com/kubernetes/community/pull/2042 was superseded by https://github.com/kubernetes/community/pull/2067.

[justaugustus]

@derekwaynecarr, the kubernetes/community#2042 was superseded by kubernetes/community#2067.

Thanks @adelton, I've updated the description to reflect that.

[mdlinville]

@derekwaynecarr please fill out the appropriate line item of the 1.11 feature tracking spreadsheet and open a placeholder docs PR against the release-1.11 branch by 5/25/2018 (tomorrow as I write this) if new docs or docs changes are needed and a relevant PR has not yet been opened.

[justaugustus]

@derekwaynecarr -- What's the current status of this feature? As we haven't heard from you with regards to some items, this feature has been moved to the Milestone risks sheet within the 1.11 Features tracking spreadsheet.

Please update the line item for this feature on the Milestone risks sheet ASAP AND ping myself and @idvoretskyi, so we can assess the feature status or we will need to officially remove it from the milestone.

[sjenning]

@justaugustus @mistyhacks the PR is almost merged https://github.com/kubernetes/kubernetes/pull/64005

Just needs API approval which we should have before freeze.

[justaugustus]

@sjenning what's the Docs status? That's the reason the feature is currently listed as At Risk.