incubator-uniffle icon indicating copy to clipboard operation
incubator-uniffle copied to clipboard

[POC][Feature] Introduce the taskId in the spark client side log

Open zuston opened this issue 3 years ago • 4 comments

What changes were proposed in this pull request?

[Feature] Introduce the taskId in the spark client side log

Why are the changes needed?

When multiple tasks are running in the same executor at the same time, it will be hard to analysis the rss log belonging to the specified task. To solve this, it's better to make log show task id in the rss client codebase.

https://user-images.githubusercontent.com/8609142/189322463-94bfc55b-0082-40aa-828b-88617f43cb28.png

Does this PR introduce any user-facing change?

Yes. Need to update the spark log4j.properties. like this

log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}%X{taskId}: %m%n

How was this patch tested?

Manual tests.

zuston avatar Sep 27 '22 10:09 zuston

PTAL @jerqi POC

zuston avatar Sep 27 '22 10:09 zuston

Codecov Report

Merging #246 (df3ba98) into master (f1cb43f) will increase coverage by 0.03%. The diff coverage is n/a.

@@             Coverage Diff              @@
##             master     #246      +/-   ##
============================================
+ Coverage     59.31%   59.35%   +0.03%     
- Complexity     1345     1346       +1     
============================================
  Files           161      161              
  Lines          8780     8780              
  Branches        828      828              
============================================
+ Hits           5208     5211       +3     
+ Misses         3304     3302       -2     
+ Partials        268      267       -1     
Impacted Files Coverage Δ
...rg/apache/uniffle/storage/common/LocalStorage.java 44.82% <0.00%> (+2.06%) :arrow_up:

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

codecov-commenter avatar Sep 27 '22 11:09 codecov-commenter

Maybe we just need to change the information that we want to log.

jerqi avatar Sep 27 '22 11:09 jerqi

Maybe we just need to change the information that we want to log.

This is not enough general and duplicate params scattered enerywhere

zuston avatar Sep 27 '22 16:09 zuston