flink icon indicating copy to clipboard operation
flink copied to clipboard

[FLINK-33988][configuration] Fix the invalid configuration when using initialized root logger level on yarn deployment mode

Open RocMarshal opened this issue 1 year ago • 1 comments

What is the purpose of the change

Fix the invalid configuration when using initialized root logger level on yarn deployment mode

Brief change log

  • Fill the containerized env values of root logger level before submiting yarn application.

Verifying this change

This change added tests and can be verified as follows:

  • *Manually verified the change by running a 4 nodes cluster.

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): (yes / no)
  • The public API, i.e., is any changed class annotated with @Public(Evolving): (yes / no)
  • The serializers: (yes / no / don't know)
  • The runtime per-record code paths (performance sensitive): (yes / no / don't know)
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know)
  • The S3 file system connector: (yes / no / don't know)

Documentation

  • Does this pull request introduce a new feature? (yes / no)
  • If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

RocMarshal avatar Jan 08 '24 05:01 RocMarshal

CI report:

  • a746744c7e1e14d077316264c583c71bc6c6278d Azure: SUCCESS
Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

flinkbot avatar Jan 08 '24 05:01 flinkbot

Hi @1996fanrui, I've verified that the current release of Flink supports modifying the logger level in both Native K8s and Standalone mode. Since YARN prepares the container launching context in a special way, we'll need to do some special adaptation anyway. LGTM

X-czh avatar Jan 16 '24 02:01 X-czh

Thanks @X-czh for the double-check and @1996fanrui review.

I made a fix like the env-value passing-in at https://github.com/RocMarshal/flink/commit/624c199e756ed5f04f849e8e53033bc1ef258f93

Would we merge the additional fix into the current pr ? Please let me know what's your opinion~ :)

RocMarshal avatar Jan 16 '24 04:01 RocMarshal

I made a fix like the env-value passing-in at https://github.com/RocMarshal/flink/commit/624c199e756ed5f04f849e8e53033bc1ef258f93 Would we merge the additional fix into the current pr ? Please let me know what's your opinion~ :)

Same PR is fine for me, but separate JIRA and commits are necessary.

Separate JIRA and commits are easy to follow up and cherry pick for other users.

1996fanrui avatar Jan 16 '24 04:01 1996fanrui