Add caching support to BaseChatModel

[asadovsky]

Looks like BaseLLM supports caching (via langchain.llm_cache), but BaseChatModel does not.

[asadovsky]

Here is a suboptimal workaround that I came across: https://gist.github.com/BlackHC/49a37aaa6e9f3e31928596ce477897ad

Note that it only works for generate, not agenerate, and only for ChatOpenAI.

[UmerHA]

Working on it - will be done today or tomorrow

[kaikun213]

Amazing! Any news on this? @UmerHA

[UmerHA]

Yes: I'm working on it (https://github.com/UmerHA/langchain/tree/1644-BaseChatModel-Caching), but it got delayed as I had to fix another PR first.

[UmerHA]

Amazing! Any news on this? @UmerHA

https://github.com/hwchase17/langchain/pull/5089 :)

[gururise]

@UmerHA Does this work for agenerate? Still getting: ValueError: GPTCache only supports caching of normal LLM generations, got <class 'langchain.schema.ChatGeneration'>

[UmerHA]

@gururise Hey, the lc team adapted the pr and now it doesn't, but it was a conscious decision from their pov. I assume the reasoning is that some caches only make sense with single-message inputs. GPTCache would be one of them.

[pors]

@gururise Hey, the lc team adapted the pr and now it doesn't, but it was a conscious decision from their pov. I assume the reasoning is that some caches only make sense with single-message inputs. GPTCache would be one of them.

Could you please elaborate a bit on that? Why won't caching make sense for a chat model?

[jakobsa]

Thanks for the merge! I have been looking into RedisCache and noticed the cache throwing KeyError 'message' when cache value would be available in the cache.

The reason seems to be, that it is not intended for use with ChatGenerations. There are ValueErrors in the code making statements along those lines. The limitation is fine, I will subclass and override. However I wanted to come back to the PR as the intended ValueError is never thrown. ChatGeneration is a subclass of Generation and therefor not hitting:

            if not isinstance(gen, Generation):
                raise ValueError(
                    "RedisCache only supports caching of normal LLM generations, "
                    f"got {type(gen)}"
                )