Junfan Zhang
Junfan Zhang
Thanks for your quick reply @Kerollmops . BTW, which data structure could be compatible with the `Roaring64NavigableMap` in java side.
> Hey [@zuston](https://github.com/zuston) 👋 I don't know which encoder they use. When [I read the doc](https://javadoc.io/doc/org.roaringbitmap/RoaringBitmap/0.7.30/org/roaringbitmap/longlong/Roaring64NavigableMap.html), it seems that they can store up to 2^32 integers, so maybe it's the...
Pleae attach your doc here @lwllvyb
I think this will bring some memory pressure, especially for some huge partitions.
> About 10M of data on average generates 1K index file. It means that if we have 10T data, there are about 1G index files. Emm... It's too big. For...
Besides, we'd better to have a unified memory manager in shuffle-server if using this cache.
PTAL @jerqi @xianjingfeng @leixm @smallzhongfeng @kaijchen
> If you use MEMORY_LOCALFILE_HDFS, it won't be a big problem. Yes. This is an optional solution.
It should fast fail
cc @xianjingfeng @jerqi