llama.cpp icon indicating copy to clipboard operation
llama.cpp copied to clipboard
Published 3 months ago • verified publisher icon ggml-org

add a grpc server for embedding and completion

Open extrame opened this issue 2 years ago • 1 comments

We add a grpc server based on http server code and some change. And I wish it will be helpful.

Buf according to grpc c++ version requirement. It will need upgrading c++11 to c++14. Maybe it will be a problem

extrame avatar Jun 09 '23 05:06 extrame

I would recommend adding a lint step for the .proto so that the protocol can evolve more rapidly and with ease as time goes on, and helps enforce that breaking API changes for the RPC mechanism are versioned up safely. I use buf, which also has a Github Action available available, so may just slide right in.

canardleteer avatar Jun 10 '23 13:06 canardleteer

Is this PR still active? It would be great to have gRPC server.

jfulem avatar Oct 17 '23 11:10 jfulem

It's quite outdated at this point.

I don't know what gRPC means, but just earlier today I saw this: https://github.com/go-skynet/LocalAI/pull/1170 Might be useful

ggerganov avatar Oct 17 '23 19:10 ggerganov