mars icon indicating copy to clipboard operation
mars copied to clipboard

Some operands can be submitted and run when some inputs are ready

Open wjsi opened this issue 6 years ago • 0 comments

Is your feature request related to a problem? Please describe. Some operands, such as summation over a long series of chunks, can be started and run partially when some data are ready. This may effectively reduce the memory size during computing.

Describe the solution you'd like Several changes are needed.

  • [ ] Separate job submission and running in worker. This enables a job be executed multiple times.
  • [ ] Partial-execution support should be added to .
  • [ ] Jobs in workers are triggered given data arrival. When executed, these jobs are released from slots.
  • [ ] Remove OperandState.FREE and use methods like ref-counting to manage these data.
  • [ ] Reflect actual progress in Mars web for these operands.

Describe alternatives you've considered Streaming-based implementation is also considered. The reason not to use this solution is that the streaming does not share the data storage model currently implemented in worker. It may be implemented later.

Additional context N/A.

wjsi avatar Jan 10 '19 08:01 wjsi