mars
mars copied to clipboard
Some operands can be submitted and run when some inputs are ready
Is your feature request related to a problem? Please describe. Some operands, such as summation over a long series of chunks, can be started and run partially when some data are ready. This may effectively reduce the memory size during computing.
Describe the solution you'd like Several changes are needed.
- [ ] Separate job submission and running in worker. This enables a job be executed multiple times.
- [ ] Partial-execution support should be added to .
- [ ] Jobs in workers are triggered given data arrival. When executed, these jobs are released from slots.
- [ ] Remove
OperandState.FREEand use methods like ref-counting to manage these data. - [ ] Reflect actual progress in Mars web for these operands.
Describe alternatives you've considered Streaming-based implementation is also considered. The reason not to use this solution is that the streaming does not share the data storage model currently implemented in worker. It may be implemented later.
Additional context N/A.