incubator-uniffle
incubator-uniffle copied to clipboard
[POC][Feature] Introduce the taskId in the spark client side log
What changes were proposed in this pull request?
[Feature] Introduce the taskId in the spark client side log
Why are the changes needed?
When multiple tasks are running in the same executor at the same time, it will be hard to analysis the rss log belonging to the specified task. To solve this, it's better to make log show task id in the rss client codebase.
https://user-images.githubusercontent.com/8609142/189322463-94bfc55b-0082-40aa-828b-88617f43cb28.png
Does this PR introduce any user-facing change?
Yes. Need to update the spark log4j.properties. like this
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}%X{taskId}: %m%n
How was this patch tested?
Manual tests.
PTAL @jerqi POC
Codecov Report
Merging #246 (df3ba98) into master (f1cb43f) will increase coverage by
0.03%. The diff coverage isn/a.
@@ Coverage Diff @@
## master #246 +/- ##
============================================
+ Coverage 59.31% 59.35% +0.03%
- Complexity 1345 1346 +1
============================================
Files 161 161
Lines 8780 8780
Branches 828 828
============================================
+ Hits 5208 5211 +3
+ Misses 3304 3302 -2
+ Partials 268 267 -1
| Impacted Files | Coverage Δ | |
|---|---|---|
| ...rg/apache/uniffle/storage/common/LocalStorage.java | 44.82% <0.00%> (+2.06%) |
:arrow_up: |
:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more
Maybe we just need to change the information that we want to log.
Maybe we just need to change the information that we want to log.
This is not enough general and duplicate params scattered enerywhere