instr-http `headersToSpanAttributes` behaviour

[trentm]

trafficstars

I think the current headersToSpanAttributes may be getting things wrong.

Using this example instrumentation and HTTP-using script: https://gist.github.com/trentm/ce3568c01d1328d0769f136c3d24d6bb which configures the instrumentation with:

    new HttpInstrumentation({
      headersToSpanAttributes: {
        client: {
          requestHeaders: ['content-length'],
          responseHeaders: ['content-length'],
        },
        server: {
          requestHeaders: ['content-length'],
          responseHeaders: ['content-length'],
        },
      },
    }),

And running that via OTEL_SEMCONV_STABILITY_OPT_IN=http node play-http.js, I get these spans (trimmed for clarity):

// client span
{
  traceId: '231454febbc08ef404100d393fd9ad00',
  parentSpanContext: {
    traceId: '231454febbc08ef404100d393fd9ad00',
    spanId: 'bee01929e2b1182e',
    traceFlags: 1,
    isRemote: true
  },
  name: 'POST',
  id: '7d5f568d5fb5de3a',
  kind: 1,
  attributes: {
    'http.request.method': 'POST',
    'url.scheme': 'http',
    'server.address': 'localhost',
    'network.peer.address': '::1',
    'network.peer.port': 64804,
    'network.protocol.version': '1.1',
    'url.path': '/',
    'client.address': '::1',
    'server.port': 3000,
    'http.request.header.content_length': [ '13' ],
    'http.response.status_code': 200
  },
  status: { code: 0 },
  events: [],
  links: []
}

// server span
{
  traceId: '231454febbc08ef404100d393fd9ad00',
  parentSpanContext: undefined,
  traceState: undefined,
  name: 'POST',
  id: 'bee01929e2b1182e',
  kind: 2,
  timestamp: 1747067420549000,
  duration: 15742.708,
  attributes: {
    'http.request.method': 'POST',
    'server.address': 'localhost',
    'server.port': 3000,
    'url.full': 'http://localhost:3000/',
    'http.response.status_code': 200,
    'network.peer.address': '::1',
    'network.peer.port': 3000,
    'network.protocol.version': '1.1',
    'http.request.header.content_length': [ 13 ],
    'http.response.header.content_length': [ '4' ]
  },
  status: { code: 0 },
  events: [],
  links: []
}

1. Note in the server span: 'http.request.header.content_length': [ 13 ],. That value is a number rather than a string. That's inconsistent with other "content length" headers.
2. The normalization if the key (e.g. content_length): I don't think hyphens should be normalized to underscores. https://opentelemetry.io/docs/specs/semconv/http/http-spans/ The spec prose is a little ambiguous on normalization here, but the examples include usage of hyphens in the keys.
3. (Bonus item.) Usage of the headersToSpanAttributes is super-verbose. I wonder if a simpler single array of header string names (to be applied for all four case of client and server, incoming and outgoing headers) would work. Granted it would make it impossible to only capture a given header name for one of those directions, but I wonder how needed that is. Users needing that level of granularity could fallback to the hook callback options. I wonder if there is a pattern for configuring this in other language SDKs.

Originally posted by @trentm in https://github.com/open-telemetry/opentelemetry-js/pull/5665#discussion_r2085068527