incubator-uniffle
incubator-uniffle copied to clipboard
apache
[Feature] Introduce the taskId in the spark client side log
Motivation
When multiple tasks are running in the same executor at the same time, it will be hard to analysis the rss log belonging to the specified task. To solve this, it's better to make log show task id in the rss client codebase.
How to do
I think we should directly use the MDC to put the context info.
POC screenshot
Could you share me that what are you concerned about most? The cost of refactor or performance?
The cost of refactor looks easy for me. We just need to inject some infos into MDC which is hold by internal thread local vars.
Could we adjust the log4j to print thread ID?
This way looks a little bit ugly on propagating thread id/ task id in multiple threads when using thread pool.
It's a little abstraction for me, maybe you can raise a draft pr and let me look at it.
It's a little abstraction for me, maybe you can raise a draft pr and let me look at it.
Yes.