Setting JAVA_TOOL_OPTIONS via configmap is not possible (replaced,ignored)

[5V715]

trafficstars

when using the operator in combination with ENVVAR from ConfigMap the original value of JAVA_TOOL_OPTIONS is ignored and therefore lost. see the append yaml and comments.

Containers:
  some-app:
    Container ID: xxx
    Image: xxx
    Image ID: xxx
    Port: 8080/TCP
    Host Port: 0/TCP
    ...
    Environment Variables from:
      some-app-config  ConfigMap  Optional: false # (<--) this also contains a value for JAVA_TOOL_OPTIONS
    Environment:
      AWS_METADATA_SERVICE_TIMEOUT: 5
      AWS_METADATA_SERVICE_NUM_ATTEMPTS: 20
      JAVA_TOOL_OPTIONS: -javaagent:/otel-auto-instrumentation/javaagent.jar # <-- replaces the old value from config map

[pavolloffay]

Correct, the implementation at the moment does not check env vars defined in the config map.

No objections on supporting it.

[pavolloffay]

Actually in this case I am not sure what would be the behaviour. I would say the operator should make changes to the config map - the problem would be to deal with reverting those changes.

Maybe it could copy the env var value from the config map if the modification is needed?

[5V715]

copying and setting it directly sounds cleaner to me. no changes to the config map needed. it's expressive and visible.

[KitSutliff]

Here’s a proposed implementation and a quick question. With a little more info I would be happy to knock this out.

Proposed implementation:

1. When the operator receives a pod we will check for config maps that reference the given pod
2. If a config map is present operator will look for JAVA_TOOL_OPTS env var
3. I there is a JAVA_TOOL_OPTS env var in the ConfigMap, that will be used to populate the JAVA_TOOL_OPTS env var in the container

Should the value form the ConfigMap overwrite the value form the container or should it be appended in some way?

[pavolloffay]

The operator should copy the value from the CM and override it.

[KitSutliff]

Awesome, thank you for the clarification! I'll start knocking this out.

[pavolloffay]

Perfect, can we make it generic enough and reuse for other/all variables?

We should as well document the behavior and that these env vars cannot be hardcoded in the image https://github.com/open-telemetry/opentelemetry-operator/issues/1884

[shazada]

Any update on this?

[yuriolisa]

#1393, I believe is requesting the same approach.

[jaronoff97]

Let's maybe discuss solutions here in our next SIG meeting. I don't initially love the idea of us reading values from arbitrary configmaps, i think that expands the permissions the operator needs and seems a bit unsafe, I can be persuaded otherwise.

[daguero-technisys]

Any update on this?

[pavolloffay]

The issue hasn't been resolved yet. This still needs a proposal that will be accepted.

[daguero-technisys]

I think that a possible solution for this case will be make a merge between the configmap variable value and the operator variable value, before inject JAVA_TOOL_OPTIONS variable value in each pod with the auto-instrumentation. Somethings like that:

JAVA_TOOL_OPTIONS="$JAVA_TOOL_OPTIONS -javaagent:/otel-auto-instrumentation/javaagent.jar"

What are you thinking about this proposal?

[pavolloffay]

it could work, but we would need to test it

[oldium]

I prepared a fix for this in #3373 - to read POD's env variables defined in ConfigMaps and Secrets.

With my fix the JAVA_TOOL_OPTIONS value is correctly merged - OTEL's -javaagent is appended to the existing value.