mars

mars copied to clipboard

## What do these changes do? This PR removes source_idxes/source_mappers from `FetchShuffle` to reduce scheduling overhead of shuffle meta and make operands relying on those attributes work on ray task-based...

chaokunyang

[BUG] unique operand failed on ray shuffle with AssertionError

1

**Describe the bug** Mars unique operand failed on ray shuffle with AssertionError on check `assert len(subtask_chunk_graph.result_chunks) == 1`: ```python def _get_subtask_out_info( subtask_chunk_graph: ChunkGraph, is_mapper: bool, n_reducers: int = None ):...

chaokunyang

## What do these changes do? - [x] Share RayTaskState across tasks. - [x] Remove `create_task_state_actor_as_needed` option. We always lazy create RayTaskState and share the actor across tasks. ## Related...

fyrestone

## What do these changes do? To make the auto merge chunk feature works, Ray context should get meta from meta service if the key does not exist in current...

fyrestone

**Describe the bug** A clear and concise description of what the bug is. ```python Traceback (most recent call last): File "/home/vsts/miniconda/envs/test/lib/python3.9/site-packages/numexpr/necompiler.py", line 820, in evaluate compiled_ex = _numexpr_cache[numexpr_key] KeyError: ('(~((abs((sin((cosh(V_0))))))))',...

fyrestone

[WIP] Apply Operand Closure clean up

1

## What do these changes do? 1. Compute logic key and for Apply Operand before tiling. 2. While tiling, put entire UDF to storage when sum of sizes of cell_contents...

vcfgv

[ray] Support scheduling ray tasks in Ray oscar deploy backend

2

## What do these changes do? This PR add support for scheduling ray tasks in Ray oscar deploy backend ## Related issue number Closes #3164 ## Check code requirements -...

chaokunyang

[BUG] Ray task backend no progress

2

**Describe the bug** Ray task mode doesn't update progress until whole task finished:  **To Reproduce** To help us reproducing this bug, please provide information below: 1. Your Python version:...

chaokunyang

## What do these changes do? This PR skip store shuffle object refs to reduce meta overhead and supervisor serialization bottleneck by disable autoscale-in when shuffle is executing. ## Related...

chaokunyang

## What do these changes do? Implements scipy.stats.rankdata ## Related issue number Fixes #1588 ## Check code requirements - [x] tests added / passed (if needed) - [x] Ensure all...

shantam-8