Support Tracing via OpenTelemetry

[mpstefan]

trafficstars

As a user of NGF I want NGF to have the ability to generate and send traces So that I can easily get tracing for all of my traffic to help debug my environment.

Background

We evaluated adding tracing capabilities before 1.0, but the nginx 1.0 otel module had not yet been released yet. Now that is has, and that @ciarams87 has completed a POC, it is time to add the capability.

Acceptance Criteria

• When enabled, the NGINX otel module is utilized to generate traces.
• Generated traces can be exported to a configurable endpoint.
• All fields from otel_exporter directive are configurable by the user.
• Documentation is created to explain this feature.

For Discussion

• Does the above acceptance criteria make sense?

### Tasks
- [ ] https://github.com/nginxinc/nginx-gateway-fabric/issues/1660
- [ ] https://github.com/nginxinc/nginx-gateway-fabric/issues/1662
- [ ] https://github.com/nginxinc/nginx-gateway-fabric/issues/1166
- [ ] https://github.com/nginxinc/nginx-gateway-fabric/issues/1800
- [ ] https://github.com/nginxinc/nginx-gateway-fabric/issues/1826
- [ ] https://github.com/nginxinc/nginx-gateway-fabric/issues/1827
- [ ] https://github.com/nginxinc/nginx-gateway-fabric/issues/1828
- [ ] https://github.com/nginxinc/nginx-gateway-fabric/issues/1911
- [ ] https://github.com/nginxinc/nginx-gateway-fabric/issues/1912

[mpstefan]

Look at using NGINX otel: https://github.com/nginxinc/nginx-otel

[brianehlert]

We should use the nginx developed module. And expose its settings. It only has tracing today, but plans to add metrics and logs.

[ciarams87]

POC using the OTel module: https://github.com/ciarams87/nginx-gateway-fabric/tree/spike/try-otel

[sjberman]

Branch containing original implementation, can be used for reference: https://github.com/nginxinc/nginx-gateway-fabric/tree/feat/data-plane-crd