Switch from apm-agent-go to OpenTelemetry

[dmathieu]

We currently use the internal apm-agent-go library for self-instrumentation of this service. We should migrate to OpenTelemetry, so we can support sending this self instrumentation data in OTLP format.

We don't want to make a big bang migration either. So we should set up apmotel (See #11192) to be able to switch bits one after the other and only remove the apm agent once everything has been migrated.

[dmathieu]

One unknown we need to resolve with this migration is head sampling and metrics being generated at the ingestion level.