dedicated mode: control plane auth fails after token expires

[stevenctl]

RE: https://github.com/istio/ztunnel/issues/440

With istio-agent, the token is intended for bootstrap and after it's fetched certs once we rely on mTLS auth.

Currently, we're tied just to the token for xDS auth

logs

2023-03-29T16:53:18.392212Z  INFO ztunnel::xds::client: sending initial request resources=0 type_url="type.googleapis.com/istio.security.Authorization"
2023-03-29T16:53:18.402299Z  WARN xds{id=5310}: ztunnel::xds::client: XDS client connection error: gRPC connection error (The request does not have valid authentication credentials): authentication failure, retrying in 15s
2023-03-29T16:53:33.415420Z  INFO ztunnel::xds::client: sending initial request resources=11 type_url="type.googleapis.com/istio.workload.Workload"
2023-03-29T16:53:33.415465Z  INFO ztunnel::xds::client: sending initial request resources=0 type_url="type.googleapis.com/istio.security.Authorization"
2023-03-29T16:53:33.424895Z  WARN xds{id=5311}: ztunnel::xds::client: XDS client connection error: gRPC connection error (The request does not have valid authentication credentials): authentication failure, retrying in 15s

Immediately after manually refreshing the token

2023-03-29T16:53:48.441406Z  INFO ztunnel::xds::client: sending initial request resources=11 type_url="type.googleapis.com/istio.workload.Workload"
2023-03-29T16:53:48.441454Z  INFO ztunnel::xds::client: sending initial request resources=0 type_url="type.googleapis.com/istio.security.Authorization"
2023-03-29T16:53:48.475730Z  INFO xds{id=5312}: ztunnel::xds::client: Stream established
2023-03-29T16:53:48.475805Z  INFO xds{id=5312}: ztunnel::xds::client: received response type_url="type.googleapis.com/istio.workload.Workload" size=11
2023-03-29T16:53:48.475938Z  INFO xds{id=5312}: ztunnel::xds::client: received response type_url="type.googleapis.com/istio.security.Authorization" size=0
2023-03-29T16:53:48.576655Z  INFO xds{id=5312}: ztunnel::xds::client: received response type_url="type.googleapis.com/istio.workload.Workload" size=1

[costinm]

My suggestion is to add an option (TOKEN_URL) to allow customization of the address to get new tokens for VMs and similar managed environments. A per VM agent acting as MDS could be used for non-cloud VMs - most cloud VMs have a MDS that can give them tokens.

It is a far simpler and direct mechanism then attempting to use mTLS and have the agent refresh the cert.

We can also support mTLS - if Spire or some other external entity takes care of refreshing them, but I would prioritize JWT-based to keep things consistent and simple.

Note that getting JWT tokens from a MDS will also help for ztunnel to talk with external hbone servers.