ServiceAccounts for several different apps intermittently rejected with "invalid bearer token" error by kube-apiserver

[v1nsai]

Bug Report

Description

I have observed several apps getting 401 errors intermittently from kube-apiserver. I'm very confused that the problem is intermittent, I would expect either 100% failure or no issues from invalid tokens.

For example, after installing homepage with default settings outlined by the docs and enabling metrics-server as per the Talos docs, the logs for homepage frequently show Bearer token issues like the following:

[2024-08-11T15:38:03.165Z] [31merror[39m: <kubernetes-widget> Error getting metrics, ensure you have metrics-server installed: s {"response":{"statusCode":401,"body":"{\"kind\":\"Status\",\"apiVersion\":\"v1\",\"metadata\":{},\"status\":\"Failure\",\"message\":\"Unauthorized\",\"reason\":\"Unauthorized\",\"code\":401}\n","headers":{"audit-id":"1ebd468d-bcaf-4932-82ae-2984e514b3f3","cache-control":"no-cache, private","content-type":"application/json","date":"Sun, 11 Aug 2024 15:38:03 GMT","content-length":"129","connection":"close"},"request":{"uri":{"protocol":"https:","slashes":true,"auth":null,"host":"10.96.0.1:443","port":"443","hostname":"10.96.0.1","hash":null,"search":null,"query":null,"pathname":"/apis/metrics.k8s.io/v1beta1/nodes","path":"/apis/metrics.k8s.io/v1beta1/nodes","href":"https://10.96.0.1:443/apis/metrics.k8s.io/v1beta1/nodes"},"method":"GET","headers":{"Authorization":"Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IjI0cVZxYkI1Rlo0YVVSaGVvbWlQaFJTUmVfTVdBX1Z2LV9WdTFkTFAtcTgifQ.eyJhdWQiOlsiaHR0cHM6Ly8xOTIuMTY4LjEuMTcwOjY0NDMiXSwiZXhwIjoxNzU0OTI2NjQxLCJpYXQiOjE3MjMzOTA2NDEsImlzcyI6Imh0dHBzOi8vMTkyLjE2OC4xLjE3MDo2NDQzIiwianRpIjoiOTA3YmMyNmEtZTZiZi00OTMzLWEwMWYtNTE1M2FiNjhkNzZlIiwia3ViZXJuZXRlcy5pbyI6eyJuYW1lc3BhY2UiOiJob21lcGFnZSIsIm5vZGUiOnsibmFtZSI6ImJpZ3JpZyIsInVpZCI6IjFiYmY2NzVlLWNlODEtNDc2Ny1iOGI1LTlkOGM4Y2IwMWJhMyJ9LCJwb2QiOnsibmFtZSI6ImhvbWVwYWdlLTdiYjY3NmNiNWQtN3d0aGMiLCJ1aWQiOiJmNzI5ODQ0OS0yYzg5LTQ2ZTMtYmMxNy1kMGU3YmMxYzMyNjQifSwic2VydmljZWFjY291bnQiOnsibmFtZSI6ImhvbWVwYWdlIiwidWlkIjoiZGRlMzExMzAtMmFiNi00MTE0LTgwYjgtMjVkOThmYzgxMmQ1In0sIndhcm5hZnRlciI6MTcyMzM5NDI0OH0sIm5iZiI6MTcyMzM5MDY0MSwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50OmhvbWVwYWdlOmhvbWVwYWdlIn0.nUVIw2zItBYjOkmQr2S7ImGPNFjauzSZ4NDNFuNo9Z5i59x2f2T-260FoswFW5nsjWQl89NllTfqF0m6PhXD0iCJgnI5MwZWxZUnh6Sv_OD1k-axHB490jb7Zba6249EaB1dHhMoaCAaDt2Q2XTrR83ePLx8wknSISl_ntryvRZ4IgONWFLIeONNlw5g1fVBabff5eDdMRQGYszF3yTL645Sp3gtM3rOAdRbLeowFqwYk0-PSXE4wG5uFRY2CMejRBfXBOyyACQoTjMBEqbeKXWnQAenpL6CjAg3Qfh7psiBS0t8BQNC103ptyQhsn0LwNFL2z1bK-An_P_iYkc-kOcaX7EOdkdsEL8rLxM2gQBcNsqN45Uj1xvrnHmk1FfiuDcfAGVCs5YBE6HZgFn2h68i8Ih_RODzgAYfhQwcdTMEn8pInieBYeh_daUSkWi15eylbarkwRlm38V5NY7re-RX7MlHiXcT20iqi6vOHKiGnbpW9XQtSjxKDzwHgK5J-nZJzp0YHer6BTFh6UVdIIMJU9NB6LOQka1zyHqi__K7POW68u4Uzfzl1xTpyVa0g9gtMb5wfJFjcF37uAaLqEWOlUVtqeIBIcg8wv1H9xS5OKLxryhvBbmq428QDgLVWngzM-aediTbC4BpgsNaO4wQQ2umlFJbQNUf2HHCuR0"}}},"body":{"apiVersion":"v1","code":401,"kind":"Status","message":"Unauthorized","metadata":{},"reason":"Unauthorized","status":"Failure"},"statusCode":401,"name":"HttpError"}
[2024-08-11T15:38:09.256Z] [31merror[39m: <kubernetes-widget> Error getting ingresses: 401 {
  kind: 'Status',
  apiVersion: 'v1',
  metadata: {},
  status: 'Failure',
  message: 'Unauthorized',
  reason: 'Unauthorized',
  code: 401
} IncomingMessage {
  _readableState: [ReadableState],
  _events: [Object: null prototype],
  _eventsCount: 4,
  _maxListeners: undefined,
  socket: [TLSSocket],
  httpVersionMajor: 1,
  httpVersionMinor: 1,
  httpVersion: '1.1',
  complete: true,
  rawHeaders: [Array],
  rawTrailers: [],
  joinDuplicateHeaders: undefined,
  aborted: false,
  upgrade: false,
  url: '',
  method: null,
  statusCode: 401,
  statusMessage: 'Unauthorized',
  client: [TLSSocket],
  _consuming: false,
  _dumped: false,
  req: [ClientRequest],
  request: [Request],
  toJSON: [Function: responseToJSON],
  caseless: [Caseless],
  body: [Object],
  [Symbol(kCapture)]: false,
  [Symbol(kHeaders)]: [Object],
  [Symbol(kHeadersCount)]: 12,
  [Symbol(kTrailers)]: null,
  [Symbol(kTrailersCount)]: 0
}

I have confirmed that the jwt token does not have an expiration, and the token works at least some of the time, so the token seems to be valid and should not be getting a 401.

I observed this behavior on Talos 1.7.6 before downgrading to 1.7.5 to try to solve it, still seeing the issue though.

Logs

The following is repeated about every 20 seconds in the kube-apiserver logs

E0811 15:49:19.574078       1 authentication.go:73] "Unable to authenticate the request" err="invalid bearer token"

Environment

• Talos version: [talosctl version --nodes <problematic nodes>] Client: Tag: v1.7.5 SHA: 47731624 Built: Go version: go1.22.4 OS/Arch: darwin/arm64 Server: NODE: 192.168.1.162 Tag: v1.7.6 SHA: ae67123a Built: Go version: go1.22.5 OS/Arch: linux/amd64 Enabled: RBAC NODE: 192.168.1.170 Tag: v1.7.5 SHA: 47731624 Built: Go version: go1.22.4 OS/Arch: linux/amd64 Enabled: RBAC NODE: 192.168.1.155 Tag: v1.7.5 SHA: 47731624 Built: Go version: go1.22.4 OS/Arch: linux/amd64 Enabled: RBAC

• Kubernetes version: [kubectl version --short] Client Version: v1.30.1 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.30.1

• Platform: