Add standard attributes to all LOGS from request/context

[smyrick]

Is your feature request related to a problem? Please describe.

The Router logs in many places with messages that are not controlled by the eventing or tracing/spans system. I can add common resources to all this logs with hard-coded values in YAML or read them from environment variables but I want to also add values that come from my requests. If I could also save this into the context that might be a more generic way to read out data rather than support the entire request selector

Describe the solution you'd like

Support adding attributes to logs as values from the context or request (via the standard selectors) so that all my LOGS (not spans and traces) have a custom id (like a trace id) BUT IN ALL logs, not just single events, even ones that are not controlled by the eventing system. I.e. all logs even internal hardcoded debug or trace logs in Router

https://www.apollographql.com/docs/router/configuration/telemetry/exporters/logging/overview#resource-attribute

telemetry:
  exporters:
     logging:
       common:
         resource:
           # Existing option
           "environment.namespace": "{env.MY_K8_NAMESPACE_ENV_VARIABLE}"
           
           # New option 1
          "shanes-custom-correlation-id-1": "{context.my-custom-id|null}"

           # New option 2 from context
          "shanes-custom-correlation-id-2":
            request_context: "my-custom-id"
             
           # New option 2 from request header
          "shanes-custom-correlation-id-3":
            request_header: "my-custom-id-header"
           

Describe alternatives you've considered

I can add this id to specific log events but there are many logs events in Router that I can't not control and I want the id to appear everywhere if it is set

Additional context

We have customers who have legacy trace ids that do not follow the OpenTelemetry 16-bit standard: https://github.com/apollographql/router/issues/4892

We support parsing those different standards out but for logging purposes that id still came in in the old format so I want to be able to see all LOGS that had id xxx-yyy-zzz even if it gets properly converted to a trace-id xxxyyyzzz for OpenTelemetry TRACES

This of course is really only valid on logs that have the request/context present so this will not show up on start up logs, etc

[BrynCooke]

In theory this should give you what you need:

telemetry:
  instrumentation:
    spans:
      mode: spec_compliant
      router:
        attributes:
          shanes-custom-correlation-id:
            request_header: "x-correlation-id"

Span attributes are included on log events under the span_list key.

Resources are intentionally static and are sent in the envelope of otel payloads rather than on individual events/traces/metrics

[smyrick]

@BrynCooke That would add the value to all spans, but it does not get included in all text logs from Router. I want all logs, including ones internal to the Router debugging, to have some standard attributes

[smyrick]

Oh wait, I think I missed the key part there. There is a setting in the Router log configuration that allows you to include the span information IN the logs meaning that I can add standard attribute if they exist in the spans, correct?

https://www.apollographql.com/docs/router/configuration/telemetry/exporters/logging/stdout#json-configuration-reference

telemetry:
  instrumentation:
    spans:
      mode: spec_compliant
      default_attribute_requirement_level: recommended
       # Set values to add to all the different spans
      router:
        attributes:
          "shanes-custom-id":
            request_header: "x-shane-id"

  exporters:
    logging:
      stdout:
        enabled: true
        format:
          json:
            # Include span info in logs
            display_current_span: true
            display_span_list: true

Does this only get included as part of the standard router, supergraph, and subgraph spans though? What about logs that happen during query planning?

[smyrick]

One down side to this current configuration option is that is still logs all the span attributes which could be quite large. If I just need one header value to exist in the logs, and ideal configuration as proposed could be a lot smaller and use less resources

[BrynCooke]

Yeah, span list contains all the span data for all nested spans. I could see us adding something to allow more targeted attributes here.

[BrynCooke]

Suggest we make the JSON formatter for logging have a section that specifies allowed attributes.

[abernix]

@bnjjj Does this tie into any other work?

[bnjjj]

@abernix no I don't think so

[BrynCooke]

It could be solved with this https://github.com/apollographql/router/issues/5387