Shawn Yang

## What do these changes do? This PR removes source_idxes/source_mappers from `FetchShuffle` to reduce scheduling overhead of shuffle meta and make operands relying on those attributes work on ray task-based...

[BUG] unique operand failed on ray shuffle with AssertionError

1

**Describe the bug** Mars unique operand failed on ray shuffle with AssertionError on check `assert len(subtask_chunk_graph.result_chunks) == 1`: ```python def _get_subtask_out_info( subtask_chunk_graph: ChunkGraph, is_mapper: bool, n_reducers: int = None ):...

[ray] Support scheduling ray tasks in Ray oscar deploy backend

2

## What do these changes do? This PR add support for scheduling ray tasks in Ray oscar deploy backend ## Related issue number Closes #3164 ## Check code requirements -...

[BUG] Ray task backend no progress

2

**Describe the bug** Ray task mode doesn't update progress until whole task finished:  **To Reproduce** To help us reproducing this bug, please provide information below: 1. Your Python version:...

## What do these changes do? This PR skip store shuffle object refs to reduce meta overhead and supervisor serialization bottleneck by disable autoscale-in when shuffle is executing. ## Related...

**Describe the bug** Mars access some private attribute pof ray which needs to be removed. **To Reproduce** To help us reproducing this bug, please provide information below: 1. Your Python...

[Ray] Support supervisor on oscar for ray task mode

1

**Is your feature request related to a problem? Please describe.** Currently ray dag mode run supervisor in client, which have some issues: - Client and Ray cluster are usually not...

Ray-based Shuffle For Mars

3

This issue is used for tracking Ray-based Shuffle For Mars progress: - [x] Ray Future-based Shuffle MEP：https://github.com/mars-project/meps/pull/2 - [x] Shuffle Meta optimization #3055 - [x] Ray Future-based Shuffle - [x]...

[Proposal] Apply operand closure cleanup

5

For function calls such as DataFrame.apply, DataFrameGroupby.apply, Series.apply, etc., the user will pass in a custom function, i.e. UDF. Mars will serialize the custom function multiple times during graph building,...

**Describe the bug** A clear and concise description of what the bug is. **To Reproduce** To help us reproducing this bug, please provide information below: 1. Your Python version 2....