Add caching to BaseChatModel (issue #1644)

[UmerHA]

Add caching to BaseChatModel

Fixes #1644

(Sidenote: While testing, I noticed we have multiple implementations of Fake LLMs, used for testing. I consolidated them.)

Who can review?

Community members can review the PR once tests pass. Tag maintainers/contributors who might be interested: Models

• @hwchase17
• @agola11

Twitter: @UmerHAdil | Discord: RicChilligerDude#7589

[kaikun213]

Any comments on this? @hwchase17 @agola11

Would be great to have caching included!

[abdulzain6]

Someone please take a look at this. Really need this :) Thanks

[realjustinwu]

Need this too.

[ielmansouri]

I hope it's reviewed soon, we need caching for ChatModels !

[Rienkim]

langchain.llm_cache = SQLiteCache(database_path=".langchain.db")
chat = ChatOpenAI(temperature=0, openai_api_key=get_openai_api_key())
messages = [
    SystemMessage(content="You are a helpful assistant that translates English to French."),
    HumanMessage(content="I love programming.")
]
start = time.time()
print(chat(messages))
print(f"first time = {time.time() - start}")
start = time.time()
print(chat(messages))
print(f"first time = {time.time() - start}")

I test this code with this PR. First request miss cache, so It works. But, second request hit cache, and error occur.

cls = <class 'langchain.schema.ChatGeneration'>
values = {'generation_info': None, 'text': "J'adore la programmation."}

    @root_validator
    def set_text(cls, values: Dict[str, Any]) -> Dict[str, Any]:
>       values["text"] = values["message"].content
E       KeyError: 'message'

With InMemoryCache, test code work fine

langchain.llm_cache = InMemoryCache()
chat = ChatOpenAI(temperature=0, openai_api_key=get_openai_api_key())
messages = [
    SystemMessage(content="You are a helpful assistant that translates English to French."),
    HumanMessage(content="I love programming.")
]
start = time.time()
print(chat(messages))
print(f"first time = {time.time() - start}")
start = time.time()
print(chat(messages))
print(f"first time = {time.time() - start}")

My guess is that InMemoryCache is just python dictionary, so it save data as ChatGeneration type. However, SQLiteCache is local database, so it save data as Generation type. If cache hit with SQLiteCache (and other type cache), loaded data is Generation type, not ChatGeneration. So, there is no "message" property in loaded data. Fast(?) solution is implementing langchain.chat_model seperately, which save and load ChatGeneration type. But, it need two cache code, llm and chat_model for all cache implementation. To solve this problem, BaseCache need to be modified (I think). But, It is complicated.

[UmerHA]

Hey @Rienkim, thanks for pointing that out! I'll take a look & add more tests that use more different caching options.

ETA should be this week. In the meanwhile, I'll turn this PR into a draft.

[UmerHA]

@Rienkim Fixed it & added more tests

[deepblue]

@UmerHA thank you for working on this.

I found that _combine_llm_outputs implemented in this PR can be an issue with OpenAICallbackHandler https://github.com/hwchase17/langchain/blob/master/langchain/callbacks/openai_info.py#LL99C42-L99C42

[sam-cohan]

Hi folks, what is the latest on this? Is there a timeline for merging this?

[UmerHA]

@UmerHA thank you for working on this.

I found that _combine_llm_outputs implemented in this PR can be an issue with OpenAICallbackHandler https://github.com/hwchase17/langchain/blob/master/langchain/callbacks/openai_info.py#LL99C42-L99C42

Can you post example code with actual & expected behavior? To me it seems fine, but maybe I'm missing something:

langchain.llm_cache = SQLiteCache(".langchain.test.db")
langchain.llm_cache.clear()

oai_cb = OpenAICallbackHandler()

llm = ChatOpenAI(callbacks=[oai_cb])

messages = [
    SystemMessage(content="You are a helpful assistant that translates English to French."),
    HumanMessage(content="I love programming.")
]

print(llm(messages))
print(oai_cb)

print(llm(messages))
print(oai_cb)

print(llm(messages))
print(oai_cb)

gives first

content="J'adore programmer." additional_kwargs={} example=False
Tokens Used: 34
	Prompt Tokens: 28
	Completion Tokens: 6
Successful Requests: 1
Total Cost (USD): $6.8e-05

then

content="J'adore programmer." additional_kwargs={} example=False
Tokens Used: 34
	Prompt Tokens: 28
	Completion Tokens: 6
Successful Requests: 2
Total Cost (USD): $6.8e-05

and then

content="J'adore programmer." additional_kwargs={} example=False
Tokens Used: 34
	Prompt Tokens: 28
	Completion Tokens: 6
Successful Requests: 3
Total Cost (USD): $6.8e-05

What would you expect to be different? :)

[pors]

@UmerHA thank you for working on this. I found that _combine_llm_outputs implemented in this PR can be an issue with OpenAICallbackHandler https://github.com/hwchase17/langchain/blob/master/langchain/callbacks/openai_info.py#LL99C42-L99C42

Can you post example code with actual & expected behavior? To me it seems fine, but maybe I'm missing something:

langchain.llm_cache = SQLiteCache(".langchain.test.db")
langchain.llm_cache.clear()

oai_cb = OpenAICallbackHandler()

llm = ChatOpenAI(callbacks=[oai_cb])

messages = [
    SystemMessage(content="You are a helpful assistant that translates English to French."),
    HumanMessage(content="I love programming.")
]

print(llm(messages))
print(oai_cb)

print(llm(messages))
print(oai_cb)

print(llm(messages))
print(oai_cb)

gives first

content="J'adore programmer." additional_kwargs={} example=False
Tokens Used: 34
	Prompt Tokens: 28
	Completion Tokens: 6
Successful Requests: 1
Total Cost (USD): $6.8e-05

then

content="J'adore programmer." additional_kwargs={} example=False
Tokens Used: 34
	Prompt Tokens: 28
	Completion Tokens: 6
Successful Requests: 2
Total Cost (USD): $6.8e-05

and then

content="J'adore programmer." additional_kwargs={} example=False
Tokens Used: 34
	Prompt Tokens: 28
	Completion Tokens: 6
Successful Requests: 3
Total Cost (USD): $6.8e-05

What would you expect to be different? :)

@deepblue ^^^ (in case you missed it)

[jakobsa]

@deepblue could you elaborate on your concerns? I am waiting for the feature :)

[deepblue]

Just realized I made an error in my code- used BaseChatModel._combine_llm_outputs in this PR instead of the default ChatOpenAI._combine_llm_outputs for testing. Retested with the right method and it's all clear. The original method doesn't affect our subclassed code. Apologies for the mix-up and the late response.

I tested and confirmed that it's working as expected

[pors]

@hwchase17 @agola11

Good to go?

[kaikun213]

@hwchase17 @agola11

Any update on this?

[vercel[bot]]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
langchain	⬜️ Ignored (Inspect)			Jun 24, 2023 6:44pm

[ghost]

Getting the below error when I use MomentoCache. @UmerHA please let me know if this is a bug or if I am doing anything wrong.

Edit: The error pops up for all calls after the cache has atleast 1 key set.

@root_validator
    def set_text(cls, values: Dict[str, Any]) -> Dict[str, Any]:
>       values["text"] = values["message"].content
E       KeyError: 'message'

Code:

import langchain
from datetime import timedelta
from langchain.cache import MomentoCache

langchain.llm_cache = MomentoCache.from_client_params("langchain_momento", imedelta(days=1))

# Further code for constructing and calling the chain using ChatOpenAI

[UmerHA]

Getting the below error when I use MomentoCache. @UmerHA please let me know if this is a bug or if I am doing anything wrong.

Edit: The error pops up for all calls after the cache has atleast 1 key set.

@root_validator
    def set_text(cls, values: Dict[str, Any]) -> Dict[str, Any]:
>       values["text"] = values["message"].content
E       KeyError: 'message'

Code:

import langchain
from datetime import timedelta
from langchain.cache import MomentoCache

langchain.llm_cache = MomentoCache.from_client_params("langchain_momento", imedelta(days=1))

# Further code for constructing and calling the chain using ChatOpenAI

Can you post the full code, error message, and stack trace?