How can I serve replicas of an executor in different hosts?

[nhtlongcs]

First, I thank Jina for an easy deployment/scaling framework. In my scenario, I have independent physical instance GPU resources, so is there any way to host replicas of an executor type in a different place? For example, is a external replica exist? Or suppose I have to do that manually. In that case, any tool in the Jina ecosystem or third party can help me to solve this problem?

[hanxiao]

oh, @JoanFM @samsja @JohannesMessner remember what I told you 2 months ago in the office? The load balancer idea 😉

let's do it,let's do it,let's do it, i feel excited 😆

[JoanFM]

Hey @nhtlongcs,

This is not possible right now. This solution for now is delivered via Kubernetes where K8s would schedule different Pods in different machines.

[nhtlongcs]

@hanxiao @JoanFM Thanks for your replies, hope this feature will available soon. I have learned many about common ml-ops practices and system design from Jina

[hanxiao]

let's keep the ticket open, i will push this feature to be landed

[JoanFM]

Hey @nhtlongcs, my question is how u plan to orchestrate these different Replicas in different machines? If one of these replicas go down, do you need it to be automatically restarted, etc...? This is why we rely on K8s to be our most professional and advanced Container Orchestrator rather than reinventing the wheel ourselves.

[nhtlongcs]

Hi @JoanFM. In my situation, when opening this issue, I only expected Jina could help me with horizontal scaling easiest and simplest way. For example, in my case, if one of my replicas goes down, the load balancer can redirect automatically from 4 to 3 (for example) and notify me to take some manual actions (or may I can monitor that), which is acceptable (in my opinion).

Honestly, I have never done this before. A professional tool designed to solve multiple scenarios, such as k8s, is too advanced and challenging for me to utilize in my specific case (but thank you for mentioning that, I will find out how to use it). Maybe that is one of the reasons that led me to choose Jina instead of many professional DevOps platforms to achieve my scope.

[JoanFM]

Okey, we will take into consideration when designing this feature, thanks

[hanxiao]

Raised by community is added because it was raised by community and me

[JohannesMessner]

I have started an implementation at https://github.com/jina-ai/jina/pull/5217. Let's move more implementation specific discussions there.

[JohannesMessner]

Hey @nhtlongcs your requested feature has just been merged, with this you can have replicas distributed across machines, without resorting to k8s. Jina ~~3.11~~ 3.10.1 will include this feature and should be released today.

Thanks for the suggestion, I think this is a cool feature to have!

Edit: It's out, you can find the docs here.