legion icon indicating copy to clipboard operation
legion copied to clipboard

Data Executor Refactor

Open ndubois-legion opened this issue 2 years ago • 0 comments

Summary

Make data-executor more scalable

Current State

Currently the data-executor executes a single job at once, even though it supports many workers. We also need to go through gRPC or OpenAPI. The RemoteStub client & the data graph must also by async, for all this to work.

Tasks

  • [ ] Investigate async data graph processing #1619
  • [x] First design
  • [ ] #1857 ...

ndubois-legion avatar Apr 26 '22 19:04 ndubois-legion