jokercurry
jokercurry
I can raise a pr if needed. @jerqi
In the versions used internally in our company, we use quotas to limit the number of apps that a single user can submit. I don't have much idea about the...
[https://docs.google.com/document/d/1MApSMFQgoS1VAoKbZjomqSRm0iTbSuKG1yvKNlWW65c/edit?usp=sharing](https://docs.google.com/document/d/1MApSMFQgoS1VAoKbZjomqSRm0iTbSuKG1yvKNlWW65c/edit?usp=sharing) If u have time, PTAL @jerqi @zuston
OK, I will add later.
Sorry, I forgot to give you permission, it has been updated.@jerqi @zuston
> Could you add some diagrams? I added a simple graphic to illustrate the process of Spark's resource limitation. A more complete pr will be proposed this week.
> so it's user level quota, what if single app produce large shuffle data, then impact other app, for example a app have large shuffle data and also have lots...
Of course, I'm looking forward to it.
``` 22/09/05 20:58:01 INFO impl.HdfsShuffleReadHandler: Read index files hdfs://ns1/tmp/4/308-308/xxxx-19999_0.index for 2 ms 22/09/05 20:58:01 WARN util.RssUtils: Read index data under flow java.nio.BufferUnderflowException at java.nio.Buffer.nextGetIndex(Buffer.java:506) at java.nio.HeapByteBuffer.getLong(HeapByteBuffer.java:412) at org.apache.uniffle.common.util.RssUtils.transIndexDataToSegments(RssUtils.java:204) at org.apache.uniffle.common.util.RssUtils.transIndexDataToSegments(RssUtils.java:188)...
> We should fail fast when encountering BufferUnderflowException instead of ignoring. What I am curious about is whether this is caused by the fact that the data of the stream...