xianjingfeng
xianjingfeng
I have thought about this, but i have no good ideas. May be we should also consider object storage here. I am going to store indexes in memory as we...
> @xianjingfeng > > 1. Do you use object storage? > 2. Why should we consider if we use object storage? > 3. I see [[FEATURE] Cache index files on...
LGTM except some minor comments. Please take another look @rickyma
For cli, it is ok. But for other modules, using uber jar may make it more difficult to find problems and it may lead to many useless dependencies that will...
How about using skip list to store blocks in `ShuffleBuffer`?
> > How about using skip list to store blocks in `ShuffleBuffer`? > > Maybe we can't, we should guarantee that the order is immutable. Why we should guarantee that...
@leixm @zuston I have discussed this issue with @jerqi offline. We think we can use skipList to store blocks in `ShuffleBuffer`, but the key of the skipList should be a...
@jerqi @zuston @advancedxy @sfwang218 PTAL
> > Maybe we should update the metrics in the following method. There may be other places that have been missed. > > How to update? The updated value maybe...
> @xianjingfeng @zuston Maybe the root cause is that we change the storage or storage manager in `hadoopThreadPoolExecutor` or `localFileThreadPoolExecutor` which is designed to handle specified storage type. The `storage`...