GPTCache
GPTCache copied to clipboard
zilliztech
[Feature]: JavaScript/TypeScript Port
Python is obviously the ecosystem to be in for ML at the moment, but offering an easy interface for JavaScript/TypeScript (which is extremely common to use for both frontend and backend) could help increase library adoption.
I'd be curious to hear thoughts on the best approach for this, or if it's even a good idea, from those more knowledgeable. Would basically embedding this library be the best way, so that the code generally only needs written once? Or would a full port in JS probably be worth it, as to my understanding this isn't actually doing anything data-intensive or ML-intensive like creating the embeddings or the database system, and it would mostly just be syntactical changes instead of creating anything new?
Good idea, we will try to support it. please stay tuned. Of course, we also welcome contributions.
I'm happy to be a prominent contributor for it, would just need a bit of guidance on high-level architecture before I get started on a PoC. Would a full rewrite make sense, or would basically embedding this library and calling via node's child_process be better? Do we draw the line at just Python + JS/TS, or try and generalize to the degree that future ports are easier? Super general question, just seeking thoughts and guidance of those that may have done something related like this before.
A full rewrite isn't necessary. I think the easiest way may be to wrap this library into a server, and this is in our plan. And if you want to know more about this project, I think the readme doc and usage doc will help you a lot.
I'm not an expert on both nodejs and python but what about another wrapper?
@SimFG @aacitelli @shanghaikid (https://www.npmjs.com/package/node-calls-python)
I'm not an expert on both nodejs and python but what about another wrapper?
@SimFG @aacitelli @shanghaikid (https://www.npmjs.com/package/node-calls-python)
From performance perspective, calling python from nodejs is not a good option, personally I agree with @SimFG, a server is a better option.
does it really matters for a 100ms responses?
Personally i want to keep GPTCache light before we moved to C/S style
My idea is to wrap it into a very simple docker service, so that other languages can experience GPTCache through the interface.
works for me. just a very simple python with grpc or restful?
yes, provide two restful interfaces, the names are search and save, or like redis, called get and set
does it really matters for a 100ms responses?
Agree that performance shouldn't be a primary concern, more just a bonus. Seems to me like, seeing as a network round-trip is core to this library, optimizing for performance isn't really worth it, especially considering the library itself isn't doing the heavy computation.
does it really matters for a 100ms responses?
Agree that performance shouldn't be a primary concern, more just a bonus. Seems to me like, seeing as a network round-trip is core to this library, optimizing for performance isn't really worth it, especially considering the library itself isn't doing the heavy computation.
I agree with you, anyway, calling python from node is not a good practice and its scope will be limited within backend and it's hard to debug. Personally I prefer a typescript rewrite, but some features are not portable for node, because they are not available in node, for example faiss.
I'd ultimately prefer a full TypeScript rewrite as well. Probably better for the long/term future and maturity of the project, and decoupling the two so they can make different decisions if necessary is probably a good idea.
@andenacitelli We have implemented a simple version of the server (similar match), but currently requires a python environment to run. After installing gptcache, you can directly run the gptcache_server.py file. Further development we are in progress, such as packaging into binary and docker
we have published the docker image, which means you can use GPTCache in any language, details: https://github.com/zilliztech/GPTCache/blob/main/docs/usage.md#use-gptcache-server so i will close the issue
I prefer a typescript rewrite, but some features are not portable for node, because they are not available in node, for example faiss.
That is not true nowadays:
const embedding = new OpenAIEmbeddings()
const store = await FaissStore.loadFromPython('vectorstore', embedding)
retriever = store.asRetriever(5)
we have published the docker image, which means you can use GPTCache in any language, details: https://github.com/zilliztech/GPTCache/blob/main/docs/usage.md#use-gptcache-server so i will close the issue
that is like saying everybody should start using microservices...