legion
legion copied to clipboard
Data Executor Refactor
Summary
Make data-executor more scalable
Current State
Currently the data-executor executes a single job at once, even though it supports many workers. We also need to go through gRPC or OpenAPI. The RemoteStub
client & the data graph must also by async, for all this to work.
Tasks
- [ ] Investigate async data graph processing #1619
- [x] First design
- [ ] #1857 ...