void [Feature] Prompt Caching

Apr 16 '25 07:04 andrewpareles

Hi @andrewpareles, could you please assign this issue to me? I'd like to start working on it right away and get it resolved as soon as possible. Thanks!

Apr 17 '25 21:04 karankulshrestha

@andrewpareles I prompt caching means thinking in this direction like prompt caching refers to storing the results of previously sent prompts (or messages) to an LLM (Large Language Model) so that subsequent requests with the same or similar prompts can retrieve the cached response instead of sending the request to the LLM again.

Apr 17 '25 22:04 karankulshrestha

Sure, just assigned you!

Yep, prompt caching here means sending the provider (Anthropic, OpenAI, OpenRouter) a message to cache the conversation. A lot of providers offer discount rates for cached inputs.

Apr 17 '25 23:04 andrewpareles

@karankulshrestha hey, how is this going?

May 01 '25 05:05 andrewpareles

hey @karankulshrestha.. hope everything’s going well! Just checking in on the prompt caching feature. there’s been a bit of buzz around it lately, especially since it ties into cost management (and indirectly to cost tracking PR #485 ). Totally understand if other things have come up... if you’re swamped no worries at all. would be great to hear your feedback. really appreciate you taking it on!

May 29 '25 05:05 vrtnis