mlem icon indicating copy to clipboard operation
mlem copied to clipboard
verified publisher icon iterative

Serving models with gRPC

Open dacbd opened this issue 4 years ago • 8 comments

https://github.com/iterative/mlem/blob/8d2be329b8f97a993a5835eb6d97d61e4d6c585f/README.md?plain=1#L30

Sorry, I haven't been able to really play with this yet and give any kind of real feedback, But I noticed the linked line and I'm really interested in seeing an example.

Would you consider supporting a gRPC interface as well?

dacbd avatar Dec 17 '21 18:12 dacbd

Thanks for the feedback! Yes, we'll support it for sure. Do you work with gRPC instead of REST API?

aguschin avatar Dec 20 '21 16:12 aguschin

Currently, most of our models produce a defined output that is consumed by another resource on a fixed cadence.

We are moving to more "real-time" inferences, to which both gRPC and REST would be acceptable interfaces.

I feel that gRPC's more standardized framework offers better stability, not sure I'm finding the right words to describe my reasoning for why I would prefer gRPC > REST.

dacbd avatar Dec 20 '21 17:12 dacbd

Hi, can I be assigned this? I haven't worked with gRPC before so I am hoping this can be something new for me + could appreciate some sense of direction. Thanks!

madhur-tandon avatar Jan 13 '22 01:01 madhur-tandon

Note: @madhur-tandon was doing that in https://github.com/iterative/mlem/pull/156, but we decided to leave it hanging for some time due to complicated implementation.

aguschin avatar Aug 31 '22 10:08 aguschin

Is anyone working on this? If there is no time pressure, I would be happy to contribute

francesco086 avatar Oct 08 '22 10:10 francesco086

Hi @francesco086! Happy to hear this :)

Yes, there is no time pressure on this one. @madhur-tandon implemented that to great extend in #156, but in the process we found out many details and problems that should be solved in the meantime. So we decided to put that on hold.

If you're going to pick that up, you'll need to fetch the latest updates from main or release/0.3.0 branches first. I'm suggesting to use release/0.3.0 since we're going to merge it in main soon.

Also, in this case I would suggest to have a quick call with @madhur-tandon and @mike0sv, where they could explain the work being done and problems we encountered there.

aguschin avatar Oct 10 '22 05:10 aguschin

Thanks @aguschin , I read through the PR you mentioned and got a feeling of the kind of problems.

Perhaps I will spend a little more time in looking at the code and understand the context before a call, ok? :)

francesco086 avatar Oct 10 '22 08:10 francesco086

Sure! Just ping us once you're ready :)

aguschin avatar Oct 10 '22 10:10 aguschin