dflow
dflow copied to clipboard
Reform Bohrium executor with virtual Kubelet or Argo executor plugin
Current Bohrium executor runs an actual Pod with a go process or a DPDispatcher script running in the container to poll the job status. When massive steps with Bohrium executor are running simultaneously, the resource usage is considerable.
To reduce the cost, we can use virtual Kubelet to submit Bohrium job and poll the job status in the framework of https://github.com/dptech-corp/virtual-kubelet. A prerequisite of this project is a unified artifact storage (e.g. OSS) as the interface of Argo Workflows and Bohrium.
Another possible solution is using executor plugins of Argo to write an executor for Bohrium tasks. It is needed to develop a server which implements APIs to submit Bohrium jobs and to acquire job statuses.