mars icon indicating copy to clipboard operation
mars copied to clipboard

[WIP] Apply Operand Closure clean up

Open vcfgv opened this issue 3 years ago • 1 comments

What do these changes do?

  1. Compute logic key and for Apply Operand before tiling.
  2. While tiling, put entire UDF to storage when sum of sizes of cell_contents exceed a certain threshold, which is customizable for users to adjust or disable this feature.
  3. While executing, get UDF from storage and restore to op.func.
  4. Add storage service to supervisor to avoid store func on workers who may risk being released.

Under ray task mode, func key, i.e. ray.ObjectRef, shall be serialized by Ray rather than Mars. As a result, relevant tests are temporarily disabled for Github checks but passed locally with some modifications.

Lifecycle management of UDF storage object is planned in another PR.

Related issue number

Fixes #3193

Check code requirements

  • [x] tests added / passed (if needed)
  • [ ] Ensure all linting tests pass, see here for how to run them

vcfgv avatar Aug 04 '22 02:08 vcfgv

Some suggestions are here: https://github.com/mars-project/mars/issues/3193#issuecomment-1204708564

fyrestone avatar Aug 04 '22 03:08 fyrestone

Can we add some tests to check that closure cleanup are actually executed?

chaokunyang avatar Aug 16 '22 03:08 chaokunyang