[ST-Engine][Logs] Log ecosystem Integration

[hailin0]

Search before asking

• [X] I had searched in the feature and found no similar feature requirement.

Description

ST-Engine integration log ecosystem proposal

Backgroud & Motivation

Currently ST-Engine has not integrated log framework management engine and connector logs.

I think it is possible to integrate slf4j + log4j2 to manage logs output

Overall Design

Java log ecosystem & bridger

ecosystem

bridger

Integration hazelcast log

ST-Engine manage the cluster nodes using hazelcast. hazelcast supports integration with slf4j, log4j1.x, log4j2.x, jdk-logging, and provides http endpoint to dynamically manage logger level.

We chose to integrate log4j2.x and get the ability to dynamically manage loggger level.

• Configure hazelcast.logging.type: log4j2 into config files
• Add log4j-api、log4j-core package & config file into classpath

Integration ST-Engine dependencies package log

If ST-Engine dependencie packages use other log systems to output logs, e.g. slf4j、log4j1、 log4j2、logback、commons-logging.

We can use log bridge to integrate other logging systems：

• Add slf4j-api package into classpath
• Add log4j-slf4j-impl package into classpath
• Exclude commons-logging、slf4j-jcl from classpath, Add jcl-over-slf4j(commons-logging bridge) into classpath
• Exclude logback-classic、logback-core from classpath
• Exclude log4j、slf4j-log4j12、slf4j-reload4j from classpath, Add log4j-over-slf4j(log4j1 bridge) into classpath
• Exclude slf4j-jdk14 from classpath, Add jul-to-slf4j(jdk-logging bridge) into classpath
  • optional, low performance

ST-Engine ChildFirstClassLoader should not load packages contains with slf4j or log4j or commons-logging keyword. reference link

Integration connector log

Logging system used by each engine：

• flink use slf4j + log4j2 by default
• spark use slf4j + log4j1 by default
• ST-Engine use slf4j + log4j2 by default

If connector and their dependencie packages use other log systems to output logs, e.g. slf4j、log4j1、 log4j2、logback、commons-logging.

Exclude log system packages from connector shade package：

• Exclude commons-logging、slf4j-jcl from connector shade package
• Exclude logback-classic、logback-core from connector shade package
• Exclude log4j、slf4j-log4j12 from connector shade package
• Exclude log4j-api、log4j-core、log4j-slf4j-impl from connector shade package
• Exclude slf4j from connector shade package

Add log bridge to engine server libs.

Link the logs of same job in st-engine

We use slf4j mdc to link jobs and logs.

Define the slf4j mdc key values:

• ST-JID: indicates job ID
• ST-TID: indicates task ID
• ST-PID: indicates job pipeline ID

Inject metadata(ST-JID, ST-TID, ST-PID) to slf4j mdc at job start and cleanup slf4j mdc after job ends

Configure the output of mdc key-values for log pattern:

[%X{ST-JID, ST-TID, ST-PID}] [%p] %d{yyyy-MM-dd HH:mm:ss.SSS} [%t] %c:%L - %m%n

Rejected proposal

Job output single log file

The connector logs cannot be output to same log file as st-engine job logs, connector and dependency packages logger is declared by user custom.

Usage Scenario

No response

Related issues

No response

Are you willing to submit a PR?

• [X] Yes I am willing to submit a PR!

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[hailin0]

subtask #2722

[hailin0]

@2013650523 @531651225 @TyrantLucifer @laglangyue @zhuangchong @liugddx @iture123 @dijiekstra @guanboo @Hisoka-X @ic4y @EricJoy2048 @CalvinKirs @ashulin

I try @ everyone I know.

[CalvinKirs]

Great and +1

[TyrantLucifer]

+1

[guanboo]

+1

[Hisoka-X]

+1

[liugddx]

+1

[ashulin]

+1

[zhuangchong]

+1

[EricJoy2048]

Great, this issue clearly explains the ecology of all java logs. Thank you.

[hailin0]

subTask #3025, #3028