Ambient telemetry: implement "waypoint single" approach

[howardjohn]

https://github.com/istio/istio/issues/42320#issuecomment-1977262417

Currently one call from shell --> waypoint --> echo looks like:

istio_requests_total{connection_security_policy="mutual_tls"
destination_app="unknown"
destination_canonical_revision="latest"
destination_canonical_service="echo"
destination_cluster="Kubernetes"
destination_principal="spiffe://cluster.local/ns/default/sa/namespace-istio-waypoint"
destination_service="echo.default.svc.cluster.local"
destination_service_name="echo"
destination_service_namespace="default"
destination_version="unknown"
destination_workload="echo"
destination_workload_namespace="default"
gateway_istio_io_managed="istio.io-mesh-controller"
gateway_networking_k8s_io_gateway_name="namespace"
instance="10.244.1.11:15020"
istio_io_gateway_name="namespace"
job="kubernetes-pods"
namespace="default"
node="kind-worker"
pod="namespace-istio-waypoint-7c6566694f-8drf8"
pod_template_hash="7c6566694f"
reporter="destination"
request_protocol="http"
response_code="200"
response_flags="-"
service_istio_io_canonical_name="namespace-istio-waypoint"
service_istio_io_canonical_revision="latest"
sidecar_istio_io_inject="false"
source_app="shell"
source_canonical_revision="latest"
source_canonical_service="shell"
source_cluster="Kubernetes"
source_principal="spiffe://cluster.local/ns/default/sa/default"
source_version="latest"
source_workload="shell"
source_workload_namespace="default"}

removing the noise, and commentary inline on changes needed:

istio_requests_total{
# Why are these unknown?
destination_app="unknown"
destination_version="unknown"
destination_canonical_revision="latest"
destination_canonical_service="echo"
destination_cluster="Kubernetes"
# This is the wrong principal, should be 'echo'
destination_principal="spiffe://cluster.local/ns/default/sa/namespace-istio-waypoint"
destination_service="echo.default.svc.cluster.local"
destination_service_name="echo"
destination_service_namespace="default"
destination_workload="echo"
destination_workload_namespace="default"

# This needs to be "waypoint"
reporter="destination"

# All of this looks correct
source_app="shell"
source_version="latest"
source_canonical_revision="latest"
source_canonical_service="shell"
source_cluster="Kubernetes"
source_principal="spiffe://cluster.local/ns/default/sa/default"
source_workload="shell"
source_workload_namespace="default"
}

cc @whitneygriffith

[whitneygriffith]

@howardjohn in your call shell --> waypoint --> echo is echo in the default namespace?

And therefore only the principal value is wrong here:

destination_principal="spiffe://cluster.local/ns/default/sa/namespace-istio-waypoint"
destination_service="echo.default.svc.cluster.local"
destination_service_name="echo"
destination_service_namespace="default"
destination_workload="echo"
destination_workload_namespace="default"

Where it should be destination_principal="spiffe://cluster.local/ns/default/sa/default"

[howardjohn]

3 things are wrong:

1. should be destination_principal="spiffe://cluster.local/ns/default/sa/default"
2. The unknown values should not be unknown. maybe I didn't have version label, but i am sure I have app
3. reporter should be "waypoint"

[whitneygriffith]

envoy/#33857

[keithmattix]

I think this is mostly complete after istio/proxy#5472 and the accompanying cherry-pick. The only thing missing is destination principal