enhancements
enhancements copied to clipboard
kubernetes
contextual logging
Enhancement Description
-
One-line enhancement description (can be used as a release note): Contextual logging enables the caller of a function to control all aspects of logging (output formatting, verbosity, additional values and names).
-
Kubernetes Enhancement Proposal: https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/3077-contextual-logging
-
Discussion Link: WG Structured Logging
-
Primary contact (assignee): @pohly
-
Responsible SIGs: SIG Instrumentation, WG Structured Logging
-
Enhancement target (which target equals to which milestone):
- Alpha release target (x.y): 1.24
- Beta release target (x.y): 1.28
- Stable release target (x.y): 1.30
-
[x] Alpha
- [X] KEP (
k/enhancements) update PR(s):- https://github.com/kubernetes/enhancements/pull/3078
- https://github.com/kubernetes/enhancements/pull/3222
- [x] Code (
k/k) update PR(s):- https://github.com/kubernetes/klog/pull/296
- https://github.com/kubernetes/klog/pull/297
- https://github.com/kubernetes/kubernetes/pull/108159
- https://github.com/kubernetes/kubernetes/pull/108725
- https://github.com/kubernetes/kubernetes/pull/108995
- [x] Docs (
k/website) update PR(s):- https://github.com/kubernetes/website/pull/32452
- https://github.com/kubernetes/community/pull/6560
- [X] KEP (
-
[ ] Beta
- [x] KEP (
k/enhancements) update PR(s):- https://github.com/kubernetes/enhancements/pull/4219
- https://github.com/kubernetes/enhancements/pull/4519
- [x] Code (
k/k) update PR(s):- [ ] optional, can be added later: https://github.com/kubernetes/kubernetes/pull/120696
- [x] feature promotion: https://github.com/kubernetes/kubernetes/pull/122589
- [ ] Docs (
k/website) update(s): https://github.com/kubernetes/website/pull/45288
- [x] KEP (
Current configuration
https://github.com/kubernetes/kubernetes/blob/master/hack/logcheck.conf
Status
The following table counts log calls that need to be converted. The numbers for contextual logging include those for structured logging.
At this point, controllers could get converted to contextual logging or one of the components that was already converted to structured logging. If you want to pick one, ping @pohly on the #wg-structured-logging Slack channel. See structured and contextual logging migration instructions for guidance.
Besides migrating log calls, we also might have to migrate from APIs which don't support contextual logging to APIs which do:
- [ ] k8s.io/client-go/tools/[events|record]: add WithContext
- https://github.com/kubernetes/kubernetes/issues/115574
- [ ] HandlePanic/HandleError -> HandlePanicWithContext, HandleErrorWithContext
From 2022-10-27 ~= Kubernetes 1.26
The focus was on converting kube-controller-manager. Of 1944 unstructured and/or non-contexual logging calls in pkg/controller and cmd/kube-controller-manager, 82% were converted to structured, contextual logging in Kubernetes 1.27.
| Component | Non-Structured Logging | Non-Contextual Logging | Owner |
|---|---|---|---|
| pkg/controller/bootstrap | 15 | 28 | @mengjiao-liu, https://github.com/kubernetes/kubernetes/pull/113464 |
| pkg/controller/certificates | 22 | 31 | @mengjiao-liu, https://github.com/kubernetes/kubernetes/pull/113994 |
| pkg/controller/clusterroleaggregation | 2 | 2 | @mengjiao-liu, https://github.com/kubernetes/kubernetes/pull/113910 |
| pkg/controller/cronjob | 1 | 44 | @mengjiao-liu, https://github.com/kubernetes/kubernetes/pull/113428 |
| pkg/controller/daemon | 45 | 85 | @249043822, https://github.com/kubernetes/kubernetes/pull/113622 |
| pkg/controller/deployment | 23 | 79 | @249043822, https://github.com/kubernetes/kubernetes/pull/113525 |
| pkg/controller/disruption | 29 | 56 | @Namanl2001, https://github.com/kubernetes/kubernetes/pull/116021 |
| pkg/controller/endpoint | 12 | 24 | lunhuijie (Slack) |
| pkg/controller/endpointslice | 22 | 36 | @Namanl2001, https://github.com/kubernetes/kubernetes/pull/115295 |
| pkg/controller/endpointslicemirroring | 18 | 28 | @Namanl2001, https://github.com/kubernetes/kubernetes/pull/114982 |
| pkg/controller/garbagecollector | 55 | 105 | @ncdc, https://github.com/kubernetes/kubernetes/pull/113471 |
| pkg/controller/job | 12 | 36 | was: @sanwishe, https://github.com/kubernetes/kubernetes/pull/113576, now: @mengjiao-liu |
| pkg/controller/namespace | 30 | 55 | @yangjunmyfm192085, https://github.com/kubernetes/kubernetes/pull/113443 |
| pkg/controller/nodeipam | 135 | 210 | @yangjunmyfm192085, https://github.com/kubernetes/kubernetes/pull/112670 |
| pkg/controller/nodelifecycle | 60 | 106 | @yangjunmyfm192085, https://github.com/kubernetes/kubernetes/pull/112670 |
| pkg/controller/podautoscaler | 9 | 13 | @freddie400, https://github.com/kubernetes/kubernetes/pull/114687 |
| pkg/controller/podgc | 10 | 24 | @pravarag, https://github.com/kubernetes/kubernetes/pull/114689 |
| pkg/controller/replicaset | 20 | 49 | @Namanl2001, https://github.com/kubernetes/kubernetes/pull/114871 |
| pkg/controller/resourcequota | 24 | 37 | @ncdc, https://github.com/kubernetes/kubernetes/pull/113315 |
| pkg/controller/serviceaccount | 22 | 31 | @Namanl2001, https://github.com/kubernetes/kubernetes/pull/114918 |
| pkg/controller/statefulset | 19 | 59 | @249043822, https://github.com/kubernetes/kubernetes/pull/113840 |
| pkg/controller/storageversiongc | 4 | 6 | @songxiao-wang87, https://github.com/kubernetes/kubernetes/pull/113986 |
| pkg/controller/testutil | 9 | 9 | @Octopusjust, https://github.com/kubernetes/kubernetes/pull/114061 |
| pkg/controller/ttl | 4 | 8 | wxs (Slack) = @songxiao-wang87, https://github.com/kubernetes/kubernetes/pull/113916 |
| pkg/controller/ttlafterfinished | 9 | 15 | @obaranov1, https://github.com/kubernetes/kubernetes/pull/115332 |
| pkg/controller/util | 0 | 19 | @fatsheep9146, https://github.com/kubernetes/kubernetes/pull/115049 |
| pkg/controller/volume | 351 | 673 | @yangjunmyfm192085, https://github.com/kubernetes/kubernetes/pull/113584 |
| pkg/kubelet | 1 | 1805 | @fmuyassarov |
| pkg/scheduler | 0 | 348 | @knelasevero, https://github.com/kubernetes/kubernetes/pull/111155 |
| staging/src/k8s.io/apiextensions-apiserver | 57 | 81 | |
| staging/src/k8s.io/apimachinery | 73 | 114 | @yanjing1104 |
| staging/src/k8s.io/apiserver | 262 | 543 | |
| staging/src/k8s.io/client-go | 161 | 267 | |
| staging/src/k8s.io/cloud-provider | 108 | 146 | |
| staging/src/k8s.io/cluster-bootstrap | 2 | 4 | |
| staging/src/k8s.io/code-generator | 108 | 168 | |
| staging/src/k8s.io/component-base | 32 | 63 | |
| staging/src/k8s.io/component-helpers | 7 | 8 | |
| staging/src/k8s.io/controller-manager | 10 | 10 | |
| staging/src/k8s.io/csi-translation-lib | 3 | 4 | |
| staging/src/k8s.io/kube-aggregator | 52 | 76 | |
| staging/src/k8s.io/kube-controller-manager | 0 | 0 | |
| staging/src/k8s.io/kubectl | 89 | 147 | @yanjing1104 |
| staging/src/k8s.io/legacy-cloud-providers | 1445 | 2238 | |
| staging/src/k8s.io/mount-utils | 54 | 92 | |
| staging/src/k8s.io/pod-security-admission | 1 | 34 | @Namanl2001, https://github.com/kubernetes/kubernetes/pull/114471 |
| staging/src/k8s.io/sample-controller | 16 | 22 | @pchan, https://github.com/kubernetes/kubernetes/pull/113879 |
From 2023-03-17 = Kubernetes v1.27.0-beta.0
All of kube-controller-manager got converted.
Tables created with:
go install sigs.k8s.io/logtools/logcheck@latest
echo "Component | Non-Structured Logging | Non-Contextual Logging | Owner " && \
echo "------ | ------- | ------" && \
for i in $(find pkg/controller/* pkg/scheduler pkg/kubelet pkg/apis pkg/api cmd/kube-* cmd/kubelet -maxdepth 0 -type d | sort); do \
echo "$i | $(cd $i; ${GOPATH}/bin/logcheck -check-structured -check-deprecations=false 2>&1 ./... | wc -l ) | $(cd $i; ${GOPATH}/bin/logcheck -check-structured -check-deprecations=false -check-contextual ./... 2>&1 | wc -l ) |" | grep -v '| 0 | 0 |'; \
done
| Component | Non-Structured Logging | Non-Contextual Logging | Owner |
|---|---|---|---|
| cmd/kube-apiserver | 7 | 8 | on hold |
| cmd/kubelet | 0 | 47 | @fmuyassarov |
| cmd/kube-proxy | 0 | 46 | on hold |
| pkg/controller/certificates | 22 | 31 | @mengjiao-liu, https://github.com/kubernetes/kubernetes/pull/113994 |
| pkg/controller/deployment | 2 | 5 | @fatsheep9146, https://github.com/kubernetes/kubernetes/pull/116930 |
| pkg/controller/disruption | 29 | 54 | ~~@obaranov1, https://github.com/kubernetes/kubernetes/pull/116021~~, @mengjiao-liu, https://github.com/kubernetes/kubernetes/pull/119147 |
| pkg/controller/endpoint | 12 | 24 | @my-git9, https://github.com/kubernetes/kubernetes/pull/116755 |
| pkg/controller/endpointslice | 20 | 35 | @Namanl2001, https://github.com/kubernetes/kubernetes/pull/115295 |
| pkg/controller/endpointslicemirroring | 18 | 28 | @Namanl2001, https://github.com/kubernetes/kubernetes/pull/114982 |
| pkg/controller/garbagecollector | 3 | 3 | @fatsheep9146, https://github.com/kubernetes/kubernetes/pull/116930 |
| pkg/controller/job | 12 | 35 | @sanwishe, https://github.com/kubernetes/kubernetes/pull/113576 (needs new owner?) |
| pkg/controller/nodeipam | 8 | 13 | @fatsheep9146, https://github.com/kubernetes/kubernetes/pull/116930 |
| pkg/controller/podgc | 10 | 24 | ~~@pravarag, https://github.com/kubernetes/kubernetes/pull/114689~~, @pohly, https://github.com/kubernetes/kubernetes/pull/119250 |
| pkg/controller/replicaset | 9 | 18 | @fatsheep9146, https://github.com/kubernetes/kubernetes/pull/116930 |
| pkg/controller/statefulset | 3 | 5 | @kerthcet, https://github.com/kubernetes/kubernetes/pull/118071 |
| pkg/controller/testutil | 9 | 9 | @Octopusjust, https://github.com/kubernetes/kubernetes/pull/114061 |
| pkg/controller/util | 0 | 4 | @fatsheep9146, https://github.com/kubernetes/kubernetes/pull/116930 |
| pkg/controller/volume | 5 | 20 | @fatsheep9146, https://github.com/kubernetes/kubernetes/pull/116930 |
| pkg/kubelet | 2 | 1923 | @fmuyassarov, https://github.com/kubernetes/kubernetes/pull/114352 |
| pkg/scheduler | 2 | 349 | @mengjiao-liu, https://github.com/kubernetes/kubernetes/issues/91633 |
From 2023-09-18 =~ Kubernetes v1.28
| Component | Non-Structured Logging | Non-Contextual Logging | Owner |
|---|---|---|---|
| cmd/kube-apiserver | 6 | 7 | on hold |
| cmd/kubelet | 0 | 52 | @fmuyassarov (?), https://github.com/kubernetes/kubernetes/pull/114352 |
| cmd/kube-proxy | 0 | 41 | on hold |
| pkg/kubelet | 2 | 1942 | @fmuyassarov (?) |
| pkg/scheduler | 1 | 137 | @mengjiao-liu, https://github.com/kubernetes/kubernetes/pulls/mengjiao-liu |
| staging/src/k8s.io/apiserver | ? | ? | @tallclair, https://github.com/kubernetes/kubernetes/pull/114198 |
| staging/src/k8s.io/client-go/discovery | 11 | 21 | on hold |
| staging/src/k8s.io/client-go/examples | 14 | 14 | on hold |
| staging/src/k8s.io/client-go/metadata | 2 | 4 | on hold |
| staging/src/k8s.io/client-go/plugin | 5 | 8 | on hold |
| staging/src/k8s.io/client-go/rest | 16 | 37 | on hold |
| staging/src/k8s.io/client-go/restmapper | 3 | 6 | on hold |
| staging/src/k8s.io/client-go/tools | 104 | 171 | @pohly, https://github.com/kubernetes/kubernetes/pull/120729 |
| staging/src/k8s.io/client-go/transport | 17 | 31 | on hold |
| staging/src/k8s.io/client-go/util | 12 | 19 | on hold |
Table created manually and with:
go install sigs.k8s.io/logtools/logcheck@latest
echo "Component | Non-Structured Logging | Non-Contextual Logging | Owner " && \
echo "------ | ------- | ------- | ------" && \
for i in $(find pkg/scheduler pkg/kubelet pkg/apis pkg/api cmd/kube-* cmd/kubelet staging/src/k8s.io/client-go/* -maxdepth 0 -type d | sort); do \
echo "$i | $(cd $i; ${GOPATH}/bin/logcheck -check-structured -check-deprecations=false 2>&1 ./... | wc -l ) | $(cd $i; ${GOPATH}/bin/logcheck -check-structured -check-deprecations=false -check-contextual ./... 2>&1 | wc -l ) |" | grep -v '| 0 | 0 |'; \
done
From 2023-11-20 =~ Kubernetes v1.29
| Component | Non-Structured Logging | Non-Contextual Logging | Owner |
|---|---|---|---|
| cmd/kube-apiserver | 6 | 7 | @tallclair |
| cmd/kubelet | 0 | 52 | @fmuyassarov (?), https://github.com/kubernetes/kubernetes/pull/114352 |
| pkg/kubelet | 2 | 1983 | @fmuyassarov |
| cmd/kube-proxy | 0 | 42 | @ fatsheep9146, https://github.com/kubernetes/kubernetes/pull/122197 |
| pkg/proxy | 0 | 360 | @ fatsheep9146, see above |
| staging/src/k8s.io/apiserver | 285 | 655 | @tallclair, https://github.com/kubernetes/kubernetes/pull/114198 |
| staging/src/k8s.io/client-go/discovery | 11 | 21 | |
| staging/src/k8s.io/client-go/examples | 14 | 14 | |
| staging/src/k8s.io/client-go/metadata | 2 | 4 | |
| staging/src/k8s.io/client-go/plugin | 5 | 8 | |
| staging/src/k8s.io/client-go/rest | 16 | 37 | |
| staging/src/k8s.io/client-go/restmapper | 3 | 6 | |
| staging/src/k8s.io/client-go/tools | 83 | 143 | @pohly |
| staging/src/k8s.io/client-go/transport | 17 | 31 | |
| staging/src/k8s.io/client-go/util | 12 | 19 |
Table created with:
go install sigs.k8s.io/logtools/logcheck@latest
echo "Component | Non-Structured Logging | Non-Contextual Logging | Owner " && echo "------ | ------- | ------- | ------" && for i in $(find pkg/scheduler pkg/kubelet pkg/apis pkg/api cmd/kube-* cmd/kubelet staging/src/k8s.io/client-go/* staging/src/k8s.io/apiserver -maxdepth 0 -type d | sort); do echo "$i | $(cd $i; ${GOPATH}/bin/logcheck -check-structured -check-deprecations=false 2>&1 ./... | wc -l ) | $(cd $i; ${GOPATH}/bin/logcheck -check-structured -check-deprecations=false -check-contextual ./... 2>&1 | wc -l ) |" | grep -v '| 0 | 0 |'; done
Hello @pohly
v1.24 Enhancements team here.
Just checking in as we approach enhancements freeze on 18:00pm PT on Thursday Feb 3rd, 2022. This enhancement is targeting alpha for v1.24,
Here’s where this enhancement currently stands:
- [X] Updated KEP file using the latest template has been merged into the k/enhancements repo.
- [X] KEP status is marked as implementable for this release
- [X] KEP has a test plan section filled out.
- [X] KEP has up to date graduation criteria.
- [X] KEP has a production readiness review that has been completed and merged into k/enhancements.
The status of this enhancement is marked as tracked. Please keep the issue description and the targeted stage up-to-date for release v1.24.
Thanks!
@encodeflush : the KEP PR was merged, all criteria for alpha in 1.24 should be met now.
Hi @pohly 👋 1.24 Docs shadow here.
This enhancement is marked as 'Needs Docs' for the 1.24 release.
Please follow the steps detailed in the documentation to open a PR against the dev-1.24 branch in the k/website repo. This PR can be just a placeholder at this time and must be created before Thu March 31, 11:59 PM PDT.
Also, if needed take a look at Documenting for a release to familiarize yourself with the docs requirement for the release.
Thanks!
Hi @pohly :wave: 1.24 Release Comms team here.
We have an opt-in process for the feature blog delivery. If you would like to publish a feature blog for this issue in this cycle, then please opt in on this tracking sheet.
The deadline for submissions and the feature blog freeze is scheduled for 01:00 UTC Wednesday 23rd March 2022 / 18:00 PDT Tuesday 22nd March 2022. Other important dates for delivery and review are listed here: https://github.com/kubernetes/sig-release/tree/master/releases/release-1.24#timeline.
For reference, here is the blog for 1.23.
Please feel free to reach out any time to me or on the #release-comms channel with questions or comments.
Thanks!
Hello @pohly
I'm just checking in once more as we approach the 1.24 Code Freeze on 18:00 PDT, Tuesday, March 29th 2022
Please ensure the following items are completed:
- [X] All PRs to the Kubernetes repo that are related to your enhancement are merged by the code freeze deadline.
- [X] Have a documentation placeholder PR open by 18:00 PDT, Thursday, March 31, 2022.
- https://github.com/kubernetes/website/pull/32452
- https://github.com/kubernetes/community/pull/6560
For note, the status of this enhancement is currently marked as tracked.
Thank you!
We still need to move this feature through beta to GA, so it has to stay open.
No work is planned for 1.25. I suggest we try to go for beta in 1.26.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.
This bot triages issues and PRs according to the following rules:
- After 90d of inactivity,
lifecycle/staleis applied - After 30d of inactivity since
lifecycle/stalewas applied,lifecycle/rottenis applied - After 30d of inactivity since
lifecycle/rottenwas applied, the issue is closed
You can:
- Mark this issue or PR as fresh with
/remove-lifecycle stale - Mark this issue or PR as rotten with
/lifecycle rotten - Close this issue or PR with
/close - Offer to help out with Issue Triage
Please send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale
The instrumentation of kube-scheduler with contextual logging is still in progress. That would have been the first real confirmation of the concept.
So no, let's delay at least until 1.27.
KCM / CCM controller aliases should help with consistent component names (eventually once we do the wiring)
https://github.com/kubernetes/kubernetes/pull/115813
I would really like to work on the component pkg/controller/deployment, can someone assign this to me?
Can you also do pkg/controller/endpoint, pkg/controller/garbagecollector, pkg/controller/nodeipam, pkg/controller/replicaset, pkg/controller/statefulset, pkg/controller/util, pkg/controller/volume?
That can be a single cleanup PR because those components were already converted earlier.
Can you also do pkg/controller/endpoint, pkg/controller/garbagecollector, pkg/controller/nodeipam, pkg/controller/replicaset, pkg/controller/statefulset, pkg/controller/util, pkg/controller/volume?
That can be a single cleanup PR because those components were already converted earlier.
@pohly, will be happy to work on those as well :)
@gurpreet-legend: I added you to the table.
It looks like pkg/controller/endpoint hasn't been touched at all yet. Then do that in one PR and the cleanup changes in another.
I am really interested in working on cmd/kube-proxy component. Can someone please assign this to me?
I would like to work on this component: cmd/kube-apiserver .Can someone assign this to me.
I would really like to work on this component cmd/kube-apiserver.Can someone please assign this to me
@gurpreet-legend: I added you to the table.
It looks like pkg/controller/endpoint hasn't been touched at all yet. Then do that in one PR and the cleanup changes in another.
Sure, will work on that first and then will work on the other components :)
Hello @pohly
Can I convert staging/src/k8s.io/client-go/metadata?
Perhaps I can do a couple more afterwards
Thank you
@ricardoapl: that seems like an easy package to get started with. Please go ahead.
Hi @pohly,
Can I work on migrating staging/src/k8s.io/client-go/transport & staging/src/k8s.io/client-go/util?
Thanks.
