Junfan Zhang comments

Results 434 comments of


                                            Junfan Zhang

[Feature Request] Introduce the unique trace id to debug easily

Maybe it will. But we could introduce new config to enable debug, default is false.

[Feature Request] Introduce the unique trace id to debug easily

Got your point. Metric system is better for performance observation. And i think i can invest this topic by submitting some PRs later.

[Improvement] Process don't exit if exec start script using ansible

+1. We have similar problem.

Use NIO's Files API to replace FileInputStream/FileOutputStream in some paths

> Could you add some performance results? Should we only modify the critical code? Actually i didn't do any performance test. Just found this optimization while I was browsing spark...

Use NIO's Files API to replace FileInputStream/FileOutputStream in some paths

> @zuston From [apache/spark#20119](https://github.com/apache/spark/pull/20119), there has performance issue with NIO's Files API. The performance issue is caused by the default `InputStream.skip` implementation. So i dont replace it in the uniffle...

Use NIO's Files API to replace FileInputStream/FileOutputStream in some paths

Any progress? What do i need to do before merging @jerqi

Use NIO's Files API to replace FileInputStream/FileOutputStream in some paths

I think we could directly follow the Spark change. Besides, the performance test looks hard. @jerqi

[Improvement] Fine-grained shuffle manager fallback at the shuffle level

Could you help check this feature? @jerqi

[Improvement] Fine-grained shuffle manager fallback at the shuffle level

Sorry, maybe you misunderstand my thought. I hope if a spark job needs to do two shuffles, it can use the different shuffle manager. 1. Shuffle-ID-0, use the sortShuffleManager due...

[Improvement] Fine-grained shuffle manager fallback at the shuffle level

> When we reuse the Shuffle-ID-1, the rss don't have enough server nodes, what will it happen? I think it should fail because the shuffle data is in died server...