John Huang
John Huang
still working
With my limited experience, I think s3 is a good choice in this issue. If we choose to support s3, then it will easily extend to other filesystems (NFS, CIFS,...
Hi, @zuston , I will look into this problem and tries to raise an PR. request to assign me this issue. feel free to ping me if there is any...
[#1163] improvement(netty): return the direct byte buffer when getting localfile data by using netty
try to resolve https://github.com/apache/incubator-uniffle/issues/1163
[#1163] improvement(netty): return the direct byte buffer when getting localfile data by using netty
> We know the managedBuffer is FileSegmentManagedBuffer . To keep performance, we should ensure the FileChannel is used, which is invoked by convertToNetty method. Right? Sure. I will fix this...
[#1163] improvement(netty): return the direct byte buffer when getting localfile data by using netty
> @pegasas In fact, this line of code in getShuffleData#transferTo method `channel.write(buffer.convertToNetty())` will use zero-copy api, we should keep using `FileSegmentManagedBuffer` instead of `NettyManagedBuffer`. It looks like we indeed use...
[#1163] improvement(netty): return the direct byte buffer when getting localfile data by using netty
> @pegasas Are you willing to raise a PR to implement the zero-copy logic of Encoder? also there is another problem. At present, only sendShuffleData uses netty. The three interfaces...
try to add feature: https://github.com/apache/incubator-uniffle/issues/1020
> > Hi, @zuston , @jerqi , > > I have a naive question. How to decide on the shuffle strategy (ESS/RSS/Uniffle) of each shuffleId for spark driver/executor？ When to...
> A use of rss or ess needs to be bound to a shuffle, which is determined by `registerShuffle` . > > Feel free to discuss if you want to...