Document how to enable debug logging in collector

[ringerc]

trafficstars

The collector docs don't appear to mention how to control the collector's own log verbosity.

It uses Zap logging, and the k8s collector framework, so the admin may pass --zap-log-level=debug to set fine grained logging. This is documented for the operator SDK at https://sdk.operatorframework.io/docs/building-operators/golang/references/logging/

Mentioning the --zap-log-level param and linking to the SDK docs would help.

But it'd be nice to tell users how to actually set it, since Kustomize is a typical deployment method, but doing so through Kustomize is non-obvious. See https://github.com/operator-framework/operator-sdk/pull/5766/files for my patch against the operator SDK to add an explanation to its docs. A snippet like

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- https://url-of-opentelemetry-operator/
patches:
  - target:
      kind: Deployment
      name: opentelemetry-operator-controller-manager
      namespace: opentelemetry-opertor-system
    patch: |-
      - op: add
        path: "/spec/template/spec/containers/1/args/-"
        value: "--zap-log-level=debug"

or a kubectl hot-patch like

kubectl patch -n opentelemetry-operator-system $(\
  kubectl get -n opentelemetry-operator-system \
    -l app.kubernetes.io/name=opentelemetry-operator deployment -o name \
  ) \
  --type=json \
  -p '[{"op": "add", "path": "/spec/template/spec/containers/1/args/-", "value": "--zap-log-level=debug"}]' \
  --dry-run=client 

(To actually apply, remove --dry-run=client from the above)

TBH, it'd be a lot nicer to do this with an env-var binding, not a command-line argument. But documenting a simple way to adjust logging would help.

I don't see any increased logging with the above, so I'm not certain it's actually correct.

[pavolloffay]

Document how to enable debug logging in collector #873

Do you mean debug logging in the collector (instance created by the operator) or debug logging in the operator?

The operator log level is configurable via flag(s). The configuration has to be applied on the operator deployment and it depends how the operator is installed (via deployment or OLM).

docker run --rm -it  ghcr.io/open-telemetry/opentelemetry-operator/opentelemetry-operator:main --help                                                                                     2 ↵ ploffay@fedora
Usage of /manager:
      --auto-instrumentation-java-image string     The default OpenTelemetry Java instrumentation image. This image is used when no image is specified in the CustomResource. (default "ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:1.11.1")
      --auto-instrumentation-nodejs-image string   The default OpenTelemetry NodeJS instrumentation image. This image is used when no image is specified in the CustomResource. (default "ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-nodejs:0.27.0")
      --auto-instrumentation-python-image string   The default OpenTelemetry Python instrumentation image. This image is used when no image is specified in the CustomResource. (default "ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-python:0.30b1")
      --collector-image string                     The default OpenTelemetry collector image. This image is used when no image is specified in the CustomResource. (default "ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector:0.50.0")
      --enable-leader-election                     Enable leader election for controller manager. Enabling this will ensure there is only one active controller manager.
      --kubeconfig string                          Paths to a kubeconfig. Only required if out-of-cluster.
      --labels stringArray                         Labels to filter away from propagating onto deploys
      --metrics-addr string                        The address the metric endpoint binds to. (default ":8080")
      --target-allocator-image string              The default OpenTelemetry target allocator image. This image is used when no image is specified in the CustomResource. (default "ghcr.io/open-telemetry/opentelemetry-operator/target-allocator:0.1.0")
      --zap-devel                                  Development Mode defaults(encoder=consoleEncoder,logLevel=Debug,stackTraceLevel=Warn). Production Mode defaults(encoder=jsonEncoder,logLevel=Info,stackTraceLevel=Error)
      --zap-encoder encoder                        Zap log encoding (one of 'json' or 'console')
      --zap-log-level level                        Zap Level to configure the verbosity of logging. Can be one of 'debug', 'info', 'error', or any integer value > 0 which corresponds to custom debug levels of increasing verbosity
      --zap-stacktrace-level level                 Zap Level at and above which stacktraces are captured (one of 'info', 'error', 'panic').
      --zap-time-encoding time-encoding            Zap time encoding (one of 'epoch', 'millis', 'nano', 'iso8601', 'rfc3339' or 'rfc3339nano'). Defaults to 'epoch'.
pflag: help requested

The collector logging can be configured in collector yaml/CR under service node:

service:
  telemetry:
    logs:
      level: "DEBUG"
      development: true
      encoding: "json"

[pavolloffay]

@ringerc could you please comment?

[ringerc]

@pavolloffay I'm suggesting that the docs for the operator cover how to raise the debug level of the operator. Managing the log level of the deployed collector is simple and already well documented.

It should IMO at least mention the --zap-log-level=debug option in the README, and link to the collector SDK docs for details.

But it'd be really nice to provide a quick explanation of how to enable it for the most common deployment methods of the operator, like I gave for Kustomize above.

IMO the operator SDK should really accept an env-var for the debug level. I'm looking at adding one. But in the mean time, helping users find out how to turn on debug logging for the operator would be helpful, as it took me a while to (a) find the options and (b) figure out how to apply them.

[pavolloffay]

thanks for the clarification. Would you mind opening an issue against the operator-sdk and linking it with this one?

I totally agree that the log level (and other operator flags) should be configurable via env var.

[ringerc]

@pavolloffay I opened a PR for operator-sdk already @ https://github.com/operator-framework/operator-sdk/pull/5766

It looks like it'll be merged soon. So the opentelemetry-collector docs could probably just link to that. It won't be as nice as a canned example, but it'll do well enough. And I can see about proposing a patch on the sdk to add support for zap logger control via env-vars soon, in which case the otel operator would just inherit that support when it updates. Think the operator-sdk folks are likely to be accepting of the idea? It probably wouldn't be too big a patch for me.

[ringerc]

BTW @pavolloffay I'm thinking of attempting a much bigger change for the operator at https://github.com/open-telemetry/opentelemetry-operator/issues/901 that would address a large set of related open issues. But I really need some project owner input before I do anything, as it's a bit too large to attempt as a throw-away patch that might just get discarded

[jaronoff97]

I think this was closed by #1193 let me know if that's not the case

[ringerc]

@jaronoff97 While the linked PR is indeed helpful it lacks the "document" part. Mention of that in the README would be helpful.