seatunnel
seatunnel copied to clipboard
[Feature][Log] Integrate slf4j and log4j2 for unified management logs
Purpose of this pull request
proposal #2725
Change logs:
Maven dependencys:
- Add slf4j & log4j2 provider & bridges scope into root pom.xml dependencyManagement
- include: slf4j-api、log4j2、log4j2-slf4j-impl(slf4j provider)、xxx-to-slf4j
- exclude(provided): logback、log4j1、commons-logging、slf4j-other-provider、log4j2-to-slf4j
- Add slf4j & log4j2 & jcl-over-slf4j into root pom.xml dependencies
- Exclude all logging system packages in maven-sheade-plugin
SeaTunnel engine module:
- Enhanced logging http endpoint to dynamically change log4j2 level
- Config
hazelcast.logging.type: log4j2
SeaTunnel Dist module:
- Use
maven-assembly-pluginto copy slf4j & log4j2 & jcl-over-slf4j packages tolib/loggingfor use by starter shell
Engine Starter module[flink、spark]:
- Use
maven-dependency-pluginto copy slf4j & log4j2 & jcl-over-slf4j packages totarget/logging-e2efor use by E2E testcase - Add
lib/loggingdirectory to java classpath in start shell for use by starter class - Add jcl-over-slf4j package into shade jar
- Config
config/log4j2-console.propertieson engine-client - Exclude log4j & commons-logging from shade jar
- Fix shell script tail last line text
Engine Starter module[seatunnel]:
- Add
disruptorjar and config-Dlog4j2.contextSelectorto use asynchronous logger on engine-server - Config
config/log4j2-file.propertieson engine-server - Config
config/log4j2-console.propertieson engine-client - Add slf4j & log4j2 & jcl-over-slf4j package into starter shade jar
- Support create new log file for submit local mode job
Connector-v1 module[seatunnel-connector-flink-clickhouse]:
- Exclude commons-logging from shade jar
E2E:
- Remove
log4j.propertiesfiles - Add
log4j2-test.propertiesoutput log toSTDOUTorSTDERR - Copy starter[flink、spark]
target/logging-e2einto engine container - Copy config dir into engine container
- Improve execute job logs
Check list
- [x] Code changed are covered with tests, or it does not need tests for reason:
- [ ] If any new Jar binary package adding in your PR, please add License Notice according New License Guide
- [ ] If necessary, please update the documentation to describe the new feature. https://github.com/apache/incubator-seatunnel/tree/dev/docs
Add notes to v1 documents:Need to add runtime jar to flink engine at runtime
Improve maven package:
starter[spark、flink]
change before:
starter shade jar include: slf4j、 log4j1.x

change after:
starter shade jar exclude: slf4j、 log4j1.x, include:jcl-over-slf4j

connector:example seatunnel-connector-flink-clickhouse
change before:
connector shade jar include: commons-logging, or other loggings

change after:
connector shade jar exclude all logging packages

Please fix CI problem.
@TyrantLucifer @EricJoy2048 @ashulin @Hisoka-X @CalvinKirs thx. PTAL
close #2474
resolve conflicts please.