celeborn

celeborn copied to clipboard

### What changes were proposed in this pull request? Adding a worker metrics for publish unreleased partition location count when worker was gracefully shutdown. ### Why are the changes needed?...

s0nskar

not a formal PR, just for discussion purposes. ### What changes were proposed in this pull request? Exclude JUnit 5 dependencies to enable running Java unit tests for modules such...

vastian180

[CELEBORN-1800] Introduce ApplicationTotalCount and ApplicationFallbackCount metric to record the total and fallback count of application

8

### What changes were proposed in this pull request? Introduce `ApplicationTotalCount` and `ApplicationFallbackCount` metric to record the total and fallback count of application. ### Why are the changes needed? There...

SteNicholas

[CELEBORN-1855] LifecycleManager return appshuffleId for non barrier stage when fetch fail has been reported

11

…stage when fetch fail has been reported ### What changes were proposed in this pull request? for non barrier shuffle read stage, LifecycleManager#handleGetShuffleIdForApp always return appshuffleId whether fetch status is...

buska88

[CELEBORN-1926] updated helmChart to allow different nodeSelector and tolerations for master and worker nodes

6

### What changes were proposed in this pull request? support different master/worker nodeSelector and tolerations ### Why are the changes needed? add possibility to run master and worker on different...

jesusch

[CELEBORN-1492] Introduce Celeborn Chaos Testing Framework

5

### What changes were proposed in this pull request? Introduce Celeborn Chaos Testing Framework. ### Why are the changes needed? A chaos testing framework is designed to simulate unpredictable and...

SteNicholas

CELEBORN-1334 [CIP-12] Support HARD_SPLIT in PushMergedData should not support new version client with old version worker

4

### What changes were proposed in this pull request? Current version client can't be compatibility with older worker version. ### Why are the changes needed? For backward compatibility. ### Does...

AngersZhuuuu

### What changes were proposed in this pull request? Set the max memory threshold to actual memory allocated to the task. This is reverse-calculated from how Spark determines it since...

ssejal1508

[CELEBORN-1919] Hardsplit batch tracking should be disabled when pushing only a single replica

5

### What changes were proposed in this pull request? Remove unused batch data tracking ### Why are the changes needed? When the optimization to handle skewed partition reads is enabled,...

RexXiong

[CELEBORN-1768][WRITER] Refactoring Shuffle Writer to extract common methods

11

### What changes were proposed in this pull request? Refactoring shuffleWriters, extract common methods as BasedShuffleWriter. ### Why are the changes needed? Currently, HashBasedShuffleWriter and SortBasedShuffleWriter have a lot of...

zwangsheng