seatunnel icon indicating copy to clipboard operation
seatunnel copied to clipboard

[Feature][Log] Integrate slf4j and log4j2 for unified management logs

Open hailin0 opened this issue 3 years ago • 6 comments

Purpose of this pull request

proposal #2725

Change logs:

Maven dependencys:

  • Add slf4j & log4j2 provider & bridges scope into root pom.xml dependencyManagement
    • include: slf4j-api、log4j2、log4j2-slf4j-impl(slf4j provider)、xxx-to-slf4j
    • exclude(provided): logback、log4j1、commons-logging、slf4j-other-provider、log4j2-to-slf4j
  • Add slf4j & log4j2 & jcl-over-slf4j into root pom.xml dependencies
  • Exclude all logging system packages in maven-sheade-plugin

SeaTunnel engine module:

  • Enhanced logging http endpoint to dynamically change log4j2 level
  • Config hazelcast.logging.type: log4j2

SeaTunnel Dist module:

  • Use maven-assembly-plugin to copy slf4j & log4j2 & jcl-over-slf4j packages to lib/logging for use by starter shell

Engine Starter module[flink、spark]:

  • Use maven-dependency-plugin to copy slf4j & log4j2 & jcl-over-slf4j packages to target/logging-e2e for use by E2E testcase
  • Add lib/logging directory to java classpath in start shell for use by starter class
  • Add jcl-over-slf4j package into shade jar
  • Config config/log4j2-console.properties on engine-client
  • Exclude log4j & commons-logging from shade jar
  • Fix shell script tail last line text

Engine Starter module[seatunnel]:

  • Add disruptor jar and config -Dlog4j2.contextSelector to use asynchronous logger on engine-server
  • Config config/log4j2-file.properties on engine-server
  • Config config/log4j2-console.properties on engine-client
  • Add slf4j & log4j2 & jcl-over-slf4j package into starter shade jar
  • Support create new log file for submit local mode job

Connector-v1 module[seatunnel-connector-flink-clickhouse]:

  • Exclude commons-logging from shade jar

E2E:

  • Remove log4j.properties files
  • Add log4j2-test.properties output log to STDOUT or STDERR
  • Copy starter[flink、spark] target/logging-e2e into engine container
  • Copy config dir into engine container
  • Improve execute job logs

Check list

  • [x] Code changed are covered with tests, or it does not need tests for reason:
  • [ ] If any new Jar binary package adding in your PR, please add License Notice according New License Guide
  • [ ] If necessary, please update the documentation to describe the new feature. https://github.com/apache/incubator-seatunnel/tree/dev/docs

hailin0 avatar Oct 08 '22 15:10 hailin0

Add notes to v1 documents:Need to add runtime jar to flink engine at runtime

image

hailin0 avatar Oct 08 '22 16:10 hailin0

Improve maven package:

starter[spark、flink] change before: starter shade jar include: slf4j、 log4j1.x 企业微信截图_bf7a05da-21c3-4fb6-996c-8cd9bb29a144

change after: starter shade jar exclude: slf4j、 log4j1.x, include:jcl-over-slf4j 企业微信截图_f5f9b718-7404-49f1-a47c-d35d2e68c970

connector:example seatunnel-connector-flink-clickhouse change before: connector shade jar include: commons-logging, or other loggings 企业微信截图_574e6a49-c39a-45f7-a744-cb1fc068e6eb

change after: connector shade jar exclude all logging packages 企业微信截图_1d2163d6-351f-480f-a02e-73660793dbc8

hailin0 avatar Oct 10 '22 11:10 hailin0

Please fix CI problem.

EricJoy2048 avatar Oct 10 '22 12:10 EricJoy2048

@TyrantLucifer @EricJoy2048 @ashulin @Hisoka-X @CalvinKirs thx. PTAL

hailin0 avatar Oct 12 '22 03:10 hailin0

close #2474

hailin0 avatar Oct 12 '22 06:10 hailin0

resolve conflicts please.

EricJoy2048 avatar Oct 12 '22 14:10 EricJoy2048

image

hailin0 avatar Nov 02 '22 06:11 hailin0