[Enhancement]: Support langchain.schema.ChatGeneration

[Sendery]

Is your feature request related to a problem? Please describe.

I'm using langchain and started a GPTCache integration, but after a few attempts I did manage to config everything as I whised then i start testing and:

ValueError: GPTCache only supports caching of normal LLM generations, got <class 'langchain.schema.ChatGeneration'>

I understand that this is not a real error but from langchain the start suggesting use this schemas, wich suport text and embbedings mixed.

Describe the solution you'd like.

I did track the error to this function:

RETURN_VAL_TYPE IS A langchain.schema.Generation and well the return_val is not, so the code works as expected

    def update(self, prompt: str, llm_string: str, return_val: RETURN_VAL_TYPE) -> None:
        """Update cache.
        First, retrieve the corresponding cache object using the `llm_string` parameter,
        and then store the `prompt` and `return_val` in the cache object.
        """
        for gen in return_val:
            if not isinstance(return_val, Generation):
                raise ValueError(
                    "GPTCache only supports caching of normal LLM generations, "
                    f"got {type(gen)}"
                )
        from gptcache.adapter.api import put

        _gptcache = self._get_gptcache(llm_string)
        handled_data = json.dumps([generation.dict() for generation in return_val])
        put(prompt, handled_data, cache_obj=_gptcache)
        return None

---

I did try to this changes:

from langchain.schema import Generation, ChatGeneration
# ....
def update(self, prompt: str, llm_string: str, return_val) -> None:
# ... 
            if not isinstance(return_val, Generation) and not isnstance(return_val, ChatGeneration):
# ....

      But didnt do the trick

Describe an alternate solution.

No response

Anything else? (Additional Context)

Thank you for your time, and work!

[SimFG]

Do you mean that the gptcache in langchain can't work well now? reference: https://python.langchain.com/docs/modules/model_io/models/llms/integrations/llm_caching#gptcache

[Sendery]

It work if you use this langchain implementation:

import langchain
from langchain.llms import OpenAI

# To make the caching really obvious, lets use a slower model.
llm = OpenAI(model_name="text-davinci-002", n=2, best_of=2)

#####

langchain.llm_cache = GPTCache(init_gptcache)

llm("Tell me a joke") # simple query to the llm
# This query uses/returns  langchain.schema.Generation

but with for more complex querys with prompt template usage:

import langchain

chat_chain=langchain.LLMChain(
    llm=ChatOpenAI(model="gpt-3.5-turbo",temperature=0,cache=True), 
    prompt=prompt_template, 
    verbose=True, 
)

langchain.llm_cache= GPTCache(init_gptcache)
chat_chain.predict(prompt_template_input_text_1="Tell me a joke",
                                 prompt_temlpate_input_text_2=contex_embeddings) # More 'complex' query
# This query uses/returns langchain.schema.ChatGeneration

GPTCache does not support langchain.schema.ChatGeneration

[SimFG]

Can other caches work well? like sqlite cache

[SimFG]

I'm not sure if it's because langchain has modified something that caused the cache not to work or for other reasons, because GPTCache implements the cache capability based on the interface provided by langchain before, and there may be incompatibility now.

[sebustam]

@SimFG For chat models, LangChain is inheriting from langchain.chat_models.base.BaseChatModel instead of langchain.llms.base.BaseLLM.

[hueiyuan]

@Sendery I would like to check this problem whether is fixed or solved?