RickyMa comments

Results 168 comments of


                                            RickyMa

fix:read inconsistent when local file read handler init before flushing event to disk

ping @zuston @dingshun3016

[Improvement] Use FileChannel to replace FileOutputStream and DataOutputStream in LocalFileWriter

Close this because FileChannel will cause RSS(Resident Set Size) to grow unexpectedly.

[Improvement] Introduce local allocation buffer to store blocks in memory

This issue seems feasible. I'll take a look first. We need this too. Currently, there are a few things that we can do to make blocks smaller: 1. Set `spark.rss.writer.buffer.spill.size`...

[#1698] fix(test): Increase tests running memory for better stability

PTAL @zuston @jerqi

[#1698] fix(test): Increase tests running memory for better stability

What's your idea? Gently ping @zuston @jerqi

[#1717] improvement: Pick partitions instead of shuffles for flushing

BTW, I'm curious how much can the performance be improved?

[#1717] improvement: Pick partitions instead of shuffles for flushing

If I understand this correctly, is this PR similar to https://github.com/apache/incubator-uniffle/pull/1670 on the client-side, but implemented on the server-side? They all "flush" partial data.

[#1717] improvement: Pick partitions instead of shuffles for flushing

> This pr is for reducing the amount of data written to disk. When the server's memory is insufficient, it always has to flush to the disk. If the server's...

[#1717] improvement: Pick partitions instead of shuffles for flushing

I got your point, this PR will benefit in your cases. We don't know if it will still benefit in other cases, e.g. the shuffle size way larger than the...

[#1717] improvement: Pick partitions instead of shuffles for flushing

OK, I'll test this in the future.