dflow icon indicating copy to clipboard operation
dflow copied to clipboard

Reform Bohrium executor with virtual Kubelet or Argo executor plugin

Open zjgemi opened this issue 2 years ago • 1 comments

Current Bohrium executor runs an actual Pod with a go process or a DPDispatcher script running in the container to poll the job status. When massive steps with Bohrium executor are running simultaneously, the resource usage is considerable.

To reduce the cost, we can use virtual Kubelet to submit Bohrium job and poll the job status in the framework of https://github.com/dptech-corp/virtual-kubelet. A prerequisite of this project is a unified artifact storage (e.g. OSS) as the interface of Argo Workflows and Bohrium.

zjgemi avatar Jan 24 '23 08:01 zjgemi

Another possible solution is using executor plugins of Argo to write an executor for Bohrium tasks. It is needed to develop a server which implements APIs to submit Bohrium jobs and to acquire job statuses.

zjgemi avatar Jun 13 '23 00:06 zjgemi