SELF_HOSTED_SETUP: Document prometheus metrics port

[bjethwan]

there's no /metrics endpoint or port given in the documentation for amazon-eks-pod-identity-webhook Since this sits in the path for pod creation for performance metrics are required. Documentation for auto-scaling the controller is required. Some visibility into usage of role to deploy mapping for grafana dashboard is required

[danopia]

I'm able to scrape http request count/latency (so, # of mutations performed) as well as runtime metrics from http://%%host%%:9999/metrics

I find that availablility is more of a concern than performance, given that the hook basically just be fail-open and thus won't block pod creation (instead your pods will come up without the mutation). So you'll end up running 2 or 3 to keep reliability up. Auto scaling up from there shouldn't be necessary unless you're talking crazy scale (like, hundreds of pods created every second)

[micahhausler]

@bjethwan I renamed this issue to add docs for the metrics port. As @danopia mentioned, we do emit prometheus metrics on port 9999