dex
dex copied to clipboard
Tokens don't refresh in personal configs
Preflight Checklist
- [X] I agree to follow the Code of Conduct that this project adheres to.
- [X] I have searched the issue tracker for an issue that matches the one I want to file, without success.
- [X] I am not looking for support or already pursued the available support channels without success.
Version
2.30.2
Storage Type
Kubernetes
Installation Type
Official Helm chart
Expected Behavior
When id-token expires - automatically refresh via kubectl When refresh-token expires - automatically refresh via kubectl
Actual Behavior
Case 1:
When some users run:
kubectl get pods
Kubectl return error (500 internal server error):
❯ kubectl get pods -v7
I1208 17:24:34.056782 42200 loader.go:372] Config loaded from file: /Users/nixon89/.kube/config
I1208 17:24:34.057745 42200 round_trippers.go:432] GET https://k8s-api.example.ru:30000/api?timeout=32s
I1208 17:24:34.057757 42200 round_trippers.go:438] Request Headers:
I1208 17:24:34.057762 42200 round_trippers.go:442] Accept: application/json, */*
I1208 17:24:34.057766 42200 round_trippers.go:442] User-Agent: kubectl/v1.21.5 (darwin/amd64) kubernetes/aea7bba
I1208 17:24:34.057806 42200 round_trippers.go:432] GET https://k8s-login.example.ru/.well-known/openid-configuration
I1208 17:24:34.057812 42200 round_trippers.go:438] Request Headers:
I1208 17:24:34.372320 42200 round_trippers.go:457] Response Status: 200 OK in 314 milliseconds
I1208 17:24:34.372760 42200 round_trippers.go:432] POST https://k8s-login.example.ru/token
I1208 17:24:34.372797 42200 round_trippers.go:438] Request Headers:
I1208 17:24:34.372819 42200 round_trippers.go:442] Content-Type: application/x-www-form-urlencoded
I1208 17:24:34.372843 42200 round_trippers.go:442] Authorization: Basic <masked>
I1208 17:24:34.505009 42200 round_trippers.go:457] Response Status: 500 Internal Server Error in 132 milliseconds
I1208 17:24:34.505195 42200 round_trippers.go:432] POST https://k8s-login.example.ru/token
I1208 17:24:34.505215 42200 round_trippers.go:438] Request Headers:
I1208 17:24:34.505231 42200 round_trippers.go:442] Content-Type: application/x-www-form-urlencoded
I1208 17:24:34.629779 42200 round_trippers.go:457] Response Status: 500 Internal Server Error in 124 milliseconds
I1208 17:24:34.629857 42200 round_trippers.go:457] Response Status: in 572 milliseconds
I1208 17:24:34.630094 42200 cached_discovery.go:121] skipped caching discovery info due to Get "https://k8s-api.example.ru:30000/api?timeout=32s": failed to refresh token: oauth2: cannot fetch token: 500 Internal Server Error
Response: {"error":"invalid_request"}
I1208 17:24:34.630884 42200 round_trippers.go:432] GET https://k8s-api.example.ru:30000/api?timeout=32s
I1208 17:24:34.630921 42200 round_trippers.go:438] Request Headers:
I1208 17:24:34.630943 42200 round_trippers.go:442] Accept: application/json, */*
I1208 17:24:34.630961 42200 round_trippers.go:442] User-Agent: kubectl/v1.21.5 (darwin/amd64) kubernetes/aea7bba
I1208 17:24:34.631098 42200 round_trippers.go:432] GET https://k8s-login.example.ru/.well-known/openid-configuration
I1208 17:24:34.631137 42200 round_trippers.go:438] Request Headers:
I1208 17:24:34.672148 42200 round_trippers.go:457] Response Status: 200 OK in 40 milliseconds
I1208 17:24:34.672409 42200 round_trippers.go:432] POST https://k8s-login.example.ru/token
I1208 17:24:34.672423 42200 round_trippers.go:438] Request Headers:
I1208 17:24:34.672429 42200 round_trippers.go:442] Content-Type: application/x-www-form-urlencoded
I1208 17:24:34.672436 42200 round_trippers.go:442] Authorization: Basic <masked>
I1208 17:24:34.788667 42200 round_trippers.go:457] Response Status: 500 Internal Server Error in 116 milliseconds
I1208 17:24:34.788787 42200 round_trippers.go:432] POST https://k8s-login.example.ru/token
I1208 17:24:34.788807 42200 round_trippers.go:438] Request Headers:
I1208 17:24:34.788818 42200 round_trippers.go:442] Content-Type: application/x-www-form-urlencoded
I1208 17:24:34.912881 42200 round_trippers.go:457] Response Status: 500 Internal Server Error in 124 milliseconds
I1208 17:24:34.912995 42200 round_trippers.go:457] Response Status: in 282 milliseconds
I1208 17:24:34.913075 42200 cached_discovery.go:121] skipped caching discovery info due to Get "https://k8s-api.example.ru:30000/api?timeout=32s": failed to refresh token: oauth2: cannot fetch token: 500 Internal Server Error
Response: {"error":"invalid_request"}
I1208 17:24:34.913276 42200 shortcut.go:89] Error loading discovery information: Get "https://k8s-api.example.ru:30000/api?timeout=32s": failed to refresh token: oauth2: cannot fetch token: 500 Internal Server Error
Response: {"error":"invalid_request"}
I1208 17:24:34.913589 42200 round_trippers.go:432] GET https://k8s-api.example.ru:30000/api?timeout=32s
I1208 17:24:34.913616 42200 round_trippers.go:438] Request Headers:
I1208 17:24:34.913631 42200 round_trippers.go:442] Accept: application/json, */*
I1208 17:24:34.913648 42200 round_trippers.go:442] User-Agent: kubectl/v1.21.5 (darwin/amd64) kubernetes/aea7bba
I1208 17:24:34.913714 42200 round_trippers.go:432] GET https://k8s-login.example.ru/.well-known/openid-configuration
I1208 17:24:34.913727 42200 round_trippers.go:438] Request Headers:
I1208 17:24:34.954696 42200 round_trippers.go:457] Response Status: 200 OK in 40 milliseconds
I1208 17:24:34.954865 42200 round_trippers.go:432] POST https://k8s-login.example.ru/token
I1208 17:24:34.954875 42200 round_trippers.go:438] Request Headers:
I1208 17:24:34.954881 42200 round_trippers.go:442] Content-Type: application/x-www-form-urlencoded
I1208 17:24:34.954887 42200 round_trippers.go:442] Authorization: Basic <masked>
I1208 17:24:35.071954 42200 round_trippers.go:457] Response Status: 500 Internal Server Error in 117 milliseconds
I1208 17:24:35.072144 42200 round_trippers.go:432] POST https://k8s-login.example.ru/token
I1208 17:24:35.072181 42200 round_trippers.go:438] Request Headers:
I1208 17:24:35.072199 42200 round_trippers.go:442] Content-Type: application/x-www-form-urlencoded
I1208 17:24:35.196954 42200 round_trippers.go:457] Response Status: 500 Internal Server Error in 124 milliseconds
I1208 17:24:35.197027 42200 round_trippers.go:457] Response Status: in 283 milliseconds
I1208 17:24:35.197129 42200 cached_discovery.go:121] skipped caching discovery info due to Get "https://k8s-api.example.ru:30000/api?timeout=32s": failed to refresh token: oauth2: cannot fetch token: 500 Internal Server Error
Response: {"error":"invalid_request"}
I1208 17:24:35.197449 42200 round_trippers.go:432] GET https://k8s-api.example.ru:30000/api?timeout=32s
I1208 17:24:35.197486 42200 round_trippers.go:438] Request Headers:
I1208 17:24:35.197508 42200 round_trippers.go:442] Accept: application/json, */*
I1208 17:24:35.197526 42200 round_trippers.go:442] User-Agent: kubectl/v1.21.5 (darwin/amd64) kubernetes/aea7bba
I1208 17:24:35.197597 42200 round_trippers.go:432] GET https://k8s-login.example.ru/.well-known/openid-configuration
I1208 17:24:35.197611 42200 round_trippers.go:438] Request Headers:
I1208 17:24:35.238414 42200 round_trippers.go:457] Response Status: 200 OK in 40 milliseconds
I1208 17:24:35.238569 42200 round_trippers.go:432] POST https://k8s-login.example.ru/token
I1208 17:24:35.238586 42200 round_trippers.go:438] Request Headers:
I1208 17:24:35.238594 42200 round_trippers.go:442] Content-Type: application/x-www-form-urlencoded
I1208 17:24:35.238601 42200 round_trippers.go:442] Authorization: Basic <masked>
I1208 17:24:35.367725 42200 round_trippers.go:457] Response Status: 500 Internal Server Error in 129 milliseconds
I1208 17:24:35.367837 42200 round_trippers.go:432] POST https://k8s-login.example.ru/token
I1208 17:24:35.367993 42200 round_trippers.go:438] Request Headers:
I1208 17:24:35.368061 42200 round_trippers.go:442] Content-Type: application/x-www-form-urlencoded
I1208 17:24:35.482251 42200 round_trippers.go:457] Response Status: 500 Internal Server Error in 114 milliseconds
I1208 17:24:35.482374 42200 round_trippers.go:457] Response Status: in 284 milliseconds
I1208 17:24:35.482488 42200 cached_discovery.go:121] skipped caching discovery info due to Get "https://k8s-api.example.ru:30000/api?timeout=32s": failed to refresh token: oauth2: cannot fetch token: 500 Internal Server Error
Response: {"error":"invalid_request"}
I1208 17:24:35.482891 42200 round_trippers.go:432] GET https://k8s-api.example.ru:30000/api?timeout=32s
I1208 17:24:35.482934 42200 round_trippers.go:438] Request Headers:
I1208 17:24:35.482955 42200 round_trippers.go:442] Accept: application/json, */*
I1208 17:24:35.482971 42200 round_trippers.go:442] User-Agent: kubectl/v1.21.5 (darwin/amd64) kubernetes/aea7bba
I1208 17:24:35.483044 42200 round_trippers.go:432] GET https://k8s-login.example.ru/.well-known/openid-configuration
I1208 17:24:35.483063 42200 round_trippers.go:438] Request Headers:
I1208 17:24:35.523516 42200 round_trippers.go:457] Response Status: 200 OK in 40 milliseconds
I1208 17:24:35.523652 42200 round_trippers.go:432] POST https://k8s-login.example.ru/token
I1208 17:24:35.523666 42200 round_trippers.go:438] Request Headers:
I1208 17:24:35.523675 42200 round_trippers.go:442] Content-Type: application/x-www-form-urlencoded
I1208 17:24:35.523687 42200 round_trippers.go:442] Authorization: Basic <masked>
I1208 17:24:35.634540 42200 round_trippers.go:457] Response Status: 500 Internal Server Error in 110 milliseconds
I1208 17:24:35.634638 42200 round_trippers.go:432] POST https://k8s-login.example.ru/token
I1208 17:24:35.634696 42200 round_trippers.go:438] Request Headers:
I1208 17:24:35.634714 42200 round_trippers.go:442] Content-Type: application/x-www-form-urlencoded
I1208 17:24:35.752720 42200 round_trippers.go:457] Response Status: 500 Internal Server Error in 117 milliseconds
I1208 17:24:35.752808 42200 round_trippers.go:457] Response Status: in 269 milliseconds
I1208 17:24:35.752905 42200 cached_discovery.go:121] skipped caching discovery info due to Get "https://k8s-api.example.ru:30000/api?timeout=32s": failed to refresh token: oauth2: cannot fetch token: 500 Internal Server Error
Response: {"error":"invalid_request"}
I1208 17:24:35.753242 42200 helpers.go:234] Connection error: Get https://k8s-api.example.ru:30000/api?timeout=32s: failed to refresh token: oauth2: cannot fetch token: 500 Internal Server Error
Response: {"error":"invalid_request"}
F1208 17:24:35.754424 42200 helpers.go:115] Unable to connect to the server: failed to refresh token: oauth2: cannot fetch token: 500 Internal Server Error
Response: {"error":"invalid_request"}
goroutine 1 [running]:
k8s.io/kubernetes/vendor/k8s.io/klog/v2.stacks(0xc00018e001, 0xc0007b4f00, 0xc4, 0x13d)
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:1021 +0xb9
k8s.io/kubernetes/vendor/k8s.io/klog/v2.(*loggingT).output(0x3c90720, 0xc000000003, 0x0, 0x0, 0xc0002870a0, 0x321ed28, 0xa, 0x73, 0x100e100)
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:970 +0x191
k8s.io/kubernetes/vendor/k8s.io/klog/v2.(*loggingT).printDepth(0x3c90720, 0xc000000003, 0x0, 0x0, 0x0, 0x0, 0x2, 0xc0000a9820, 0x1, 0x1)
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:733 +0x16f
k8s.io/kubernetes/vendor/k8s.io/klog/v2.FatalDepth(...)
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:1495
k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/util.fatal(0xc000278c80, 0x95, 0x1)
In Dex logs error (400 bad request):
level=error msg="failed to refresh identity: oidc: failed to get refresh token: oauth2: cannot fetch token: 400 Bad Request\nResponse: {\"error\":\"invalid_grant\"}"
Case 2:
When some users run:
kubectl get pods -v 7
Kubectl return error (400):
❯ kubectl get pods -v 7
I1122 12:08:01.509592 57599 loader.go:372] Config loaded from file: /Users/nixon89/.kube/config
I1122 12:08:01.510903 57599 round_trippers.go:432] GET https://k8s-api.example.ru:30000/api?timeout=32s
I1122 12:08:01.510940 57599 round_trippers.go:438] Request Headers:
I1122 12:08:01.510952 57599 round_trippers.go:442] User-Agent: kubectl/v1.21.5 (darwin/amd64) kubernetes/aea7bba
I1122 12:08:01.510960 57599 round_trippers.go:442] Accept: application/json, */*
I1122 12:08:01.511026 57599 round_trippers.go:432] GET https://k8s-login.example.ru/.well-known/openid-configuration
I1122 12:08:01.511035 57599 round_trippers.go:438] Request Headers:
I1122 12:08:01.898010 57599 round_trippers.go:457] Response Status: 200 OK in 386 milliseconds
I1122 12:08:01.898320 57599 round_trippers.go:432] POST https://k8s-login.example.ru/token
I1122 12:08:01.898342 57599 round_trippers.go:438] Request Headers:
I1122 12:08:01.898355 57599 round_trippers.go:442] Content-Type: application/x-www-form-urlencoded
I1122 12:08:01.898370 57599 round_trippers.go:442] Authorization: Basic <masked>
I1122 12:08:01.945645 57599 round_trippers.go:457] Response Status: 400 Bad Request in 47 milliseconds
I1122 12:08:01.945789 57599 round_trippers.go:432] POST https://k8s-login.example.ru/token
I1122 12:08:01.945799 57599 round_trippers.go:438] Request Headers:
I1122 12:08:01.945806 57599 round_trippers.go:442] Content-Type: application/x-www-form-urlencoded
I1122 12:08:01.994794 57599 round_trippers.go:457] Response Status: 400 Bad Request in 48 milliseconds
I1122 12:08:01.994898 57599 round_trippers.go:457] Response Status: in 483 milliseconds
I1122 12:08:01.994977 57599 cached_discovery.go:121] skipped caching discovery info due to Get "https://k8s-api.example.ru:30000/api?timeout=32s": failed to refresh token: oauth2: cannot fetch token: 400 Bad Request
Response: {"error":"invalid_request","error_description":"Refresh token is invalid or has already been claimed by another client."}
I1122 12:08:01.995549 57599 round_trippers.go:432] GET https://k8s-api.example.ru:30000/api?timeout=32s
I1122 12:08:01.995583 57599 round_trippers.go:438] Request Headers:
I1122 12:08:01.995593 57599 round_trippers.go:442] Accept: application/json, */*
I1122 12:08:01.995600 57599 round_trippers.go:442] User-Agent: kubectl/v1.21.5 (darwin/amd64) kubernetes/aea7bba
I1122 12:08:01.995650 57599 round_trippers.go:432] GET https://k8s-login.example.ru/.well-known/openid-configuration
I1122 12:08:01.995659 57599 round_trippers.go:438] Request Headers:
I1122 12:08:02.040974 57599 round_trippers.go:457] Response Status: 200 OK in 45 milliseconds
I1122 12:08:02.041220 57599 round_trippers.go:432] POST https://k8s-login.example.ru/token
I1122 12:08:02.041236 57599 round_trippers.go:438] Request Headers:
I1122 12:08:02.041243 57599 round_trippers.go:442] Content-Type: application/x-www-form-urlencoded
I1122 12:08:02.041252 57599 round_trippers.go:442] Authorization: Basic <masked>
I1122 12:08:02.089723 57599 round_trippers.go:457] Response Status: 400 Bad Request in 48 milliseconds
I1122 12:08:02.089845 57599 round_trippers.go:432] POST https://k8s-login.example.ru/token
I1122 12:08:02.089870 57599 round_trippers.go:438] Request Headers:
I1122 12:08:02.089892 57599 round_trippers.go:442] Content-Type: application/x-www-form-urlencoded
I1122 12:08:02.141262 57599 round_trippers.go:457] Response Status: 400 Bad Request in 51 milliseconds
I1122 12:08:02.141326 57599 round_trippers.go:457] Response Status: in 145 milliseconds
I1122 12:08:02.141403 57599 cached_discovery.go:121] skipped caching discovery info due to Get "https://k8s-api.example.ru:30000/api?timeout=32s": failed to refresh token: oauth2: cannot fetch token: 400 Bad Request
Response: {"error":"invalid_request","error_description":"Refresh token is invalid or has already been claimed by another client."}
I1122 12:08:02.141452 57599 shortcut.go:89] Error loading discovery information: Get "https://k8s-api.example.ru:30000/api?timeout=32s": failed to refresh token: oauth2: cannot fetch token: 400 Bad Request
Response: {"error":"invalid_request","error_description":"Refresh token is invalid or has already been claimed by another client."}
I1122 12:08:02.141677 57599 round_trippers.go:432] GET https://k8s-api.example.ru:30000/api?timeout=32s
I1122 12:08:02.141695 57599 round_trippers.go:438] Request Headers:
I1122 12:08:02.141705 57599 round_trippers.go:442] Accept: application/json, */*
I1122 12:08:02.141713 57599 round_trippers.go:442] User-Agent: kubectl/v1.21.5 (darwin/amd64) kubernetes/aea7bba
I1122 12:08:02.141762 57599 round_trippers.go:432] GET https://k8s-login.example.ru/.well-known/openid-configuration
I1122 12:08:02.141771 57599 round_trippers.go:438] Request Headers:
I1122 12:08:02.183973 57599 round_trippers.go:457] Response Status: 200 OK in 42 milliseconds
I1122 12:08:02.184268 57599 round_trippers.go:432] POST https://k8s-login.example.ru/token
I1122 12:08:02.184297 57599 round_trippers.go:438] Request Headers:
I1122 12:08:02.184308 57599 round_trippers.go:442] Content-Type: application/x-www-form-urlencoded
I1122 12:08:02.184323 57599 round_trippers.go:442] Authorization: Basic <masked>
I1122 12:08:02.248091 57599 round_trippers.go:457] Response Status: 400 Bad Request in 63 milliseconds
I1122 12:08:02.248255 57599 round_trippers.go:432] POST https://k8s-login.example.ru/token
I1122 12:08:02.248291 57599 round_trippers.go:438] Request Headers:
I1122 12:08:02.248308 57599 round_trippers.go:442] Content-Type: application/x-www-form-urlencoded
I1122 12:08:02.293144 57599 round_trippers.go:457] Response Status: 400 Bad Request in 44 milliseconds
I1122 12:08:02.293205 57599 round_trippers.go:457] Response Status: in 151 milliseconds
I1122 12:08:02.293260 57599 cached_discovery.go:121] skipped caching discovery info due to Get "https://k8s-api.example.ru:30000/api?timeout=32s": failed to refresh token: oauth2: cannot fetch token: 400 Bad Request
Response: {"error":"invalid_request","error_description":"Refresh token is invalid or has already been claimed by another client."}
I1122 12:08:02.293491 57599 round_trippers.go:432] GET https://k8s-api.example.ru:30000/api?timeout=32s
I1122 12:08:02.293508 57599 round_trippers.go:438] Request Headers:
I1122 12:08:02.293515 57599 round_trippers.go:442] Accept: application/json, */*
I1122 12:08:02.293522 57599 round_trippers.go:442] User-Agent: kubectl/v1.21.5 (darwin/amd64) kubernetes/aea7bba
I1122 12:08:02.293560 57599 round_trippers.go:432] GET https://k8s-login.example.ru/.well-known/openid-configuration
I1122 12:08:02.293565 57599 round_trippers.go:438] Request Headers:
I1122 12:08:02.333084 57599 round_trippers.go:457] Response Status: 200 OK in 39 milliseconds
I1122 12:08:02.333250 57599 round_trippers.go:432] POST https://k8s-login.example.ru/token
I1122 12:08:02.333258 57599 round_trippers.go:438] Request Headers:
I1122 12:08:02.333264 57599 round_trippers.go:442] Content-Type: application/x-www-form-urlencoded
I1122 12:08:02.333269 57599 round_trippers.go:442] Authorization: Basic <masked>
I1122 12:08:02.380786 57599 round_trippers.go:457] Response Status: 400 Bad Request in 47 milliseconds
I1122 12:08:02.380845 57599 round_trippers.go:432] POST https://k8s-login.example.ru/token
I1122 12:08:02.380855 57599 round_trippers.go:438] Request Headers:
I1122 12:08:02.380866 57599 round_trippers.go:442] Content-Type: application/x-www-form-urlencoded
I1122 12:08:02.428589 57599 round_trippers.go:457] Response Status: 400 Bad Request in 47 milliseconds
I1122 12:08:02.428625 57599 round_trippers.go:457] Response Status: in 135 milliseconds
I1122 12:08:02.428659 57599 cached_discovery.go:121] skipped caching discovery info due to Get "https://k8s-api.example.ru:30000/api?timeout=32s": failed to refresh token: oauth2: cannot fetch token: 400 Bad Request
Response: {"error":"invalid_request","error_description":"Refresh token is invalid or has already been claimed by another client."}
I1122 12:08:02.428786 57599 round_trippers.go:432] GET https://k8s-api.example.ru:30000/api?timeout=32s
I1122 12:08:02.428795 57599 round_trippers.go:438] Request Headers:
I1122 12:08:02.428799 57599 round_trippers.go:442] Accept: application/json, */*
I1122 12:08:02.428803 57599 round_trippers.go:442] User-Agent: kubectl/v1.21.5 (darwin/amd64) kubernetes/aea7bba
I1122 12:08:02.428829 57599 round_trippers.go:432] GET https://k8s-login.example.ru/.well-known/openid-configuration
I1122 12:08:02.428833 57599 round_trippers.go:438] Request Headers:
I1122 12:08:02.468816 57599 round_trippers.go:457] Response Status: 200 OK in 39 milliseconds
I1122 12:08:02.468931 57599 round_trippers.go:432] POST https://k8s-login.example.ru/token
I1122 12:08:02.468942 57599 round_trippers.go:438] Request Headers:
I1122 12:08:02.468948 57599 round_trippers.go:442] Content-Type: application/x-www-form-urlencoded
I1122 12:08:02.468953 57599 round_trippers.go:442] Authorization: Basic <masked>
I1122 12:08:02.512975 57599 round_trippers.go:457] Response Status: 400 Bad Request in 44 milliseconds
I1122 12:08:02.513027 57599 round_trippers.go:432] POST https://k8s-login.example.ru/token
I1122 12:08:02.513034 57599 round_trippers.go:438] Request Headers:
I1122 12:08:02.513040 57599 round_trippers.go:442] Content-Type: application/x-www-form-urlencoded
I1122 12:08:02.562111 57599 round_trippers.go:457] Response Status: 400 Bad Request in 49 milliseconds
I1122 12:08:02.562166 57599 round_trippers.go:457] Response Status: in 133 milliseconds
I1122 12:08:02.562213 57599 cached_discovery.go:121] skipped caching discovery info due to Get "https://k8s-api.example.ru:30000/api?timeout=32s": failed to refresh token: oauth2: cannot fetch token: 400 Bad Request
Response: {"error":"invalid_request","error_description":"Refresh token is invalid or has already been claimed by another client."}
I1122 12:08:02.562275 57599 helpers.go:234] Connection error: Get https://k8s-api.example.ru:30000/api?timeout=32s: failed to refresh token: oauth2: cannot fetch token: 400 Bad Request
Response: {"error":"invalid_request","error_description":"Refresh token is invalid or has already been claimed by another client."}
F1122 12:08:02.562304 57599 helpers.go:115] Unable to connect to the server: failed to refresh token: oauth2: cannot fetch token: 400 Bad Request
Response: {"error":"invalid_request","error_description":"Refresh token is invalid or has already been claimed by another client."}
goroutine 1 [running]:
k8s.io/kubernetes/vendor/k8s.io/klog/v2.stacks(0xc000138001, 0xc000746000, 0x118, 0x169)
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:1021 +0xb9
k8s.io/kubernetes/vendor/k8s.io/klog/v2.(*loggingT).output(0x3c90720, 0xc000000003, 0x0, 0x0, 0xc0001e42a0, 0x321ed28, 0xa, 0x73, 0x100e100)
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:970 +0x191
k8s.io/kubernetes/vendor/k8s.io/klog/v2.(*loggingT).printDepth(0x3c90720, 0xc000000003, 0x0, 0x0, 0x0, 0x0, 0x2, 0xc000554c00, 0x1, 0x1)
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:733 +0x16f
k8s.io/kubernetes/vendor/k8s.io/klog/v2.FatalDepth(...)
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:1495
k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/util.fatal(0xc000198000, 0xe9, 0x1)
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/util/helpers.go:93 +0x288
k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/util.checkErr(0x2c9ae60, 0xc000a2ba10, 0x2b244d0)
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/util/helpers.go:188 +0x935
k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/util.CheckErr(...)
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/util/helpers.go:115
k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/get.NewCmdGet.func1(0xc0005b0840, 0xc000081b30, 0x1, 0x3)
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/get/get.go:167 +0x159
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).execute(0xc0005b0840, 0xc000081b00, 0x3, 0x3, 0xc0005b0840, 0xc000081b00)
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:854 +0x2c2
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc00042f080, 0xc00013a120, 0xc00013e000, 0x5)
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:958 +0x375
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).Execute(...)
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:895
main.main()
_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubectl/kubectl.go:49 +0x21d
goroutine 18 [chan receive]:
k8s.io/kubernetes/vendor/k8s.io/klog/v2.(*loggingT).flushDaemon(0x3c90720)
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:1164 +0x8b
created by k8s.io/kubernetes/vendor/k8s.io/klog/v2.init.0
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:418 +0xdf
goroutine 8 [select]:
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x2b243f0, 0x2c993a0, 0xc000226000, 0x1, 0xc00010cb40)
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:167 +0x118
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x2b243f0, 0x12a05f200, 0x0, 0x1, 0xc00010cb40)
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x98
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.Until(0x2b243f0, 0x12a05f200, 0xc00010cb40)
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90 +0x4d
created by k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/util/logs.InitLogs
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/util/logs/logs.go:51 +0x96
goroutine 22 [IO wait]:
internal/poll.runtime_pollWait(0xc71f410, 0x72, 0xffffffffffffffff)
/usr/local/go/src/runtime/netpoll.go:222 +0x55
internal/poll.(*pollDesc).wait(0xc0007b6198, 0x72, 0x1200, 0x128e, 0xffffffffffffffff)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc0007b6180, 0xc0009f5300, 0x128e, 0x128e, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:166 +0x1d5
net.(*netFD).Read(0xc0007b6180, 0xc0009f5300, 0x128e, 0x128e, 0x1281, 0xc0009c6fa0, 0xd)
/usr/local/go/src/net/fd_posix.go:55 +0x4f
net.(*conn).Read(0xc00000e9c0, 0xc0009f5300, 0x128e, 0x128e, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:183 +0x91
crypto/tls.(*atLeastReader).Read(0xc000b20960, 0xc0009f5300, 0x128e, 0x128e, 0xc000354a80, 0xc00006a900, 0x0)
/usr/local/go/src/crypto/tls/conn.go:776 +0x63
bytes.(*Buffer).ReadFrom(0xc0009c7078, 0x2c979a0, 0xc000b20960, 0x100b605, 0x27e5b40, 0x29a6c40)
/usr/local/go/src/bytes/buffer.go:204 +0xbe
crypto/tls.(*Conn).readFromUntil(0xc0009c6e00, 0xc71f5e0, 0xc00000e9c0, 0x5, 0xc00000e9c0, 0x400)
/usr/local/go/src/crypto/tls/conn.go:798 +0xf3
crypto/tls.(*Conn).readRecordOrCCS(0xc0009c6e00, 0x0, 0x0, 0x2)
/usr/local/go/src/crypto/tls/conn.go:605 +0x115
crypto/tls.(*Conn).readRecord(...)
/usr/local/go/src/crypto/tls/conn.go:573
crypto/tls.(*Conn).Read(0xc0009c6e00, 0xc0001db000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/crypto/tls/conn.go:1276 +0x165
bufio.(*Reader).Read(0xc0003bfc20, 0xc000b42f18, 0x9, 0x9, 0x13bf2ab, 0xc000a4dc78, 0x1006c65)
/usr/local/go/src/bufio/bufio.go:227 +0x222
io.ReadAtLeast(0x2c977c0, 0xc0003bfc20, 0xc000b42f18, 0x9, 0x9, 0x9, 0xc000554aa0, 0x6cb0f689261800, 0xc000554aa0)
/usr/local/go/src/io/io.go:328 +0x87
io.ReadFull(...)
/usr/local/go/src/io/io.go:347
net/http.http2readFrameHeader(0xc000b42f18, 0x9, 0x9, 0x2c977c0, 0xc0003bfc20, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/http/h2_bundle.go:1477 +0x89
net/http.(*http2Framer).ReadFrame(0xc000b42ee0, 0xc000a2b9e0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/http/h2_bundle.go:1735 +0xa5
net/http.(*http2clientConnReadLoop).run(0xc000a4dfa8, 0x0, 0x0)
/usr/local/go/src/net/http/h2_bundle.go:8331 +0xd8
net/http.(*http2ClientConn).readLoop(0xc000582a80)
/usr/local/go/src/net/http/h2_bundle.go:8253 +0x6f
created by net/http.(*http2Transport).newClientConn
/usr/local/go/src/net/http/h2_bundle.go:7217 +0x6c5
In Dex logs error:
Response: {"error":"invalid_request","error_description":"Refresh token is invalid or has already been claimed by another client."}
Steps To Reproduce
I don't know how reproduce this cases. Users only use kubectl/k9s. kubectl/k9s via oidc may brake after 1,2,4 days, but may brakes after 45 days.
k8s APIServer options:
--oidc-issuer-url=https://k8s-login.example.ru
--oidc-client-id=gangway
--oidc-groups-claim=groups
--oidc-username-claim=email
our setup: k8s-apiserver + dex (in oidc mode) + oidc.example.com (our custom oidc server) + gangway (for present kubeconfigs to users) + oauth2-proxy/k8s-dashboard. We hasn't problem in tandem: dex + oauth2-proxy/k8s-dashboard. Problems only with personal kubeconfigs, that uses in kubectl/k9s.
Additional Information
Kubernetes v1.19.7 Dex 2.30.2 heptio/gangway 3.3.0 kubectl 1.19..1.22
Configuration
# cat /etc/dex/config.yaml
connectors:
- config:
basicAuthUnsupported: true
claimMapping: null
clientID: kubernetes
clientSecret: REDACTED
insecureSkipEmailVerified: true
issuer: https://oidc.example.ru/
redirectURI: https://k8s-login.example.ru/callback
scopes:
- profile
- email
- offline_access
id: example-oidc
name: example-oidc
type: oidc
expiry:
idTokens: 1440m
refreshTokens:
absoluteLifetime: 876000h
reuseInterval: 3s
validIfNotUsedFor: 4392h
issuer: https://k8s-login.example.ru
logger:
level: debug
oauth2:
AlwaysShowLoginScreen: false
skipApprovalScreen: true
staticClients:
- id: gangway
name: gangway
redirectURIs:
- https://k8s.example.ru/callback
- https://k8s-oauth2-proxy.example.ru/oauth2/callback
secret: REDACTED
storage:
config:
inCluster: true
type: kubernetes
web:
http: 0.0.0.0:5556/
Logs
No response
Hello, Nikolay!
Some thoughts:
-
This is a problem on the side of the
oidc.example.ru
. Provided response{"error":"invalid_grant"}
is what oid.example.com responded to Dex. Could you check its logs to find the actual reason for a problem, because theinvalid_grant
error has a lot of meanings in the OIDC world? -
Dex rotates refresh token on every refresh request in terms of security, thus no one can steal the refresh token and use it forever. It seems that kubectl has problems persisting credentials back to the kubeconfig file sometimes.
You can try to disable refresh tokens rotation and see if it helps:
expiry: idTokens: 1440m refreshTokens: disableRotation: true absoluteLifetime: 876000h validIfNotUsedFor: 4392h
Hello, @nabokihms
We lived 2 months with the disableRotation: true
setting enabled in dex
Subjectively, there are fewer problems with non-working configs, but they have not completely disappeared.
I did the following experiment:
- got a personal config for test user
- looked at what happened in CRD
refreshtokens.dex.coreos.com
in the cluster. TheconnectorData
field was filled in correctly - the testuser's refresh token from found the provider's oidc in base64 - made a simple call to the cluster via kubectl (everything was successful), without errors
- waited until 14 days have passed since the release of the personal config, and don't use this config for test user
- made a simple call to the cluster via kubectl (everything was successful), there were no errors
- waited another 18 days (because the value of validIfNotUsedFor for our oidc provider is 15 days, but in dex we still have the value validIfNotUsedFor: 4392h set), and don't use this config for test user
- again performed a simple call to the cluster through kubectl (got an error):
7.1 kubectl -
Internal Server Error Response: {"error":"invalid_request"}
7.2 dex -level=error msg="failed to refresh identity: oidc: failed to get refresh token: oauth2: cannot fetch token: 400 Bad Request\nResponse: {\"error\":\"invalid_grant\"}"
- looked again at CRD
refreshtokens.dex.coreos.com
- and for some reason there was not even aconnectorData
field there for test user.
Is this the correct work of dex? I mean even with disableRotation:true
in dex, the refresh token from the oidc provider was removed from the connectorData
field)
I expected that dex itself received a new refresh token from an external oidc provider and replace old token to new token in CRD refreshtokens.dex.coreos.com
in connectorData
field for test user
It seems that there is a problem with the concurrent call of the Refresh
method of the connector. I will provide a detailed description of the problem and possible solutions.
Hello everyone, I am also facing similar kind of issue with different error. I am generating id-token and refresh-token using dex for kube-oidc-proxy check to Kubernetes GKE server. On generating new tokens, everything works fine and I am able to perform kubectl commands which first get authenticated at kube-oidc-proxy server and then get forwarded to kubernetes API server on GKE.
But once id-token gets expired, refresh-token should work and generate new id-token but running kubectl commands with expired token gives
Unable to connect to the server: Get "https://dex.xyz:32000/.well-known/openid-configuration": x509: certificate signed by unknown authority
which is strange to me as new tokens always work and no error comes regarding unknown authority, but expired token give this where refresh-token should have worked... I have already entered root-ca config in oidc-proxy configurations and in dex-authenticator(in place of gangway) configurations
I am using google connector and kubernetes storage.
This is my configuration file: `issuer: https://dex.xyz:32000 storage: type: kubernetes config: inCluster: true web: https: 0.0.0.0:5556 tlsCert: /etc/dex/tls.crt tlsKey: /etc/dex/tls.key connectors:
- type: google id: google name: Google config: clientID: <> clientSecret: <> redirectURI: https://dex.xyz:32000/callback org: kubernetes serviceAccountFilePath: /etc/dex/dex-googleauth/googleauth.json adminEmail: <> expiry: idTokens: 2m refreshTokens: absoluteLifetime: 876000h reuseInterval: 3s validIfNotUsedFor: 4392h oauth2: skipApprovalScreen: false
staticClients:
- id: dex-auth
redirectURIs:
- 'http://login.abc:5555/callback' name: 'Dex-Authenticator' secret: <> `
It seems that there is a problem with the concurrent call of the
Refresh
method of the connector. I will provide a detailed description of the problem and possible solutions.
@nabokihms can you share your solution please?
Hello, @ansh-lehri. I think your problem differs from the one @nixon89 has. Kubectl fails to request Dex because of the TLS certificate.
The first step is to check auth flags in your kubeconfig
users:
- name: username
user:
auth-provider:
config:
...
idp-issuer-url: https://dex.example.com/
idp-certificate-authority-data: ...
Which certificate will you get if you try to access the Dex URL? Is it returns a valid TLS certificate? Is the idp-certificate-authority
/idp-certificate-authority-data
provided/required?
@nabokihms it looks like we are facing the same problem as described by @nixon89, it is reproduced from time to time. Do you have any news about possible solutions?
@sgremyachikh I tried to describe the problem in detail in the linked issue #2547. I hope when we come up with a solution for calling refresh multiple times, your situation will also be fixed.