feat: Cache generated responses

[whiterabbit1983]

trafficstars

:rocket: This description was created by Ellipsis for commit 0529b486d05b3052d6dbc6f591d0169677b72565

Summary:

Introduces caching for generated responses in the agents-api, including new utility functions, a caching decorator, a database migration for cache storage, and a new dependency.

Key points:

• Added get_cached_response and set_cached_response functions in /agents-api/agents_api/models/session/ for caching.
• Implemented cache decorator in /agents-api/agents_api/routers/sessions/session.py to enable caching for the generate method.
• Created migration script /agents-api/migrations/migrate_1716013793_session_cache.py for session_cache table.
• Added xxhash dependency in pyproject.toml for generating cache keys.

---

Generated with :heart: by ellipsis.dev

[sweep-ai-deprecated[bot]]

Sweep: PR Review

agents-api/agents_api/models/session/get_cached_response.py

Introduced a new function get_cached_response to query a session cache using a provided key, and added the necessary import for the cozo_query decorator.

Sweep Found These Issues

• The function get_cached_response does not handle the case where the key is not found in the session_cache, which could lead to unexpected behavior or errors.

https://github.com/julep-ai/julep/blob/0529b486d05b3052d6dbc6f591d0169677b72565/agents-api%2Fagents_api%2Fmodels%2Fsession%2Fget_cached_response.py#L5-L11 View Diff

---

agents-api/agents_api/models/session/set_cached_response.py

Introduced a new function set_cached_response to cache key-value pairs in a session_cache.

Sweep Found These Issues

• The function does not handle potential exceptions that may occur during the query execution, which could lead to unhandled errors.

https://github.com/julep-ai/julep/blob/0529b486d05b3052d6dbc6f591d0169677b72565/agents-api%2Fagents_api%2Fmodels%2Fsession%2Fset_cached_response.py#L5-L16 View Diff

Potential Issues

Sweep isn't 100% sure if the following are issues or not but they may be worth taking a look at.

• The function does not validate the input parameters key and value, which could lead to unexpected behavior or security vulnerabilities if invalid data is passed.

https://github.com/julep-ai/julep/blob/0529b486d05b3052d6dbc6f591d0169677b72565/agents-api%2Fagents_api%2Fmodels%2Fsession%2Fset_cached_response.py#L5-L16 View Diff

• The function does not validate the input parameters key and value, which could lead to unexpected behavior or security vulnerabilities if invalid data is passed.

https://github.com/julep-ai/julep/blob/0529b486d05b3052d6dbc6f591d0169677b72565/agents-api%2Fagents_api%2Fmodels%2Fsession%2Fset_cached_response.py#L5-L16 View Diff

---

agents-api/agents_api/routers/sessions/session.py

The changes introduce a caching mechanism using the xxhash library and new functions to store and retrieve cached responses, applied to the generate method of the BaseSession class.

Potential Issues

Sweep isn't 100% sure if the following are issues or not but they may be worth taking a look at.

• The cache decorator assumes that the result returned by get_cached_response can be directly converted into a ChatCompletion object, which might not always be the case, potentially causing a ValueError.

https://github.com/julep-ai/julep/blob/0529b486d05b3052d6dbc6f591d0169677b72565/agents-api%2Fagents_api%2Frouters%2Fsessions%2Fsession.py#L77 View Diff

---

agents-api/migrations/migrate_1716013793_session_cache.py

The changes introduce a new migration script to create and remove a "session_cache" table with key-value pairs.

Sweep Found These Issues

• The SQL command for creating the "session_cache" table uses a non-standard syntax with :create and ::remove, which may not be recognized by the database engine.

https://github.com/julep-ai/julep/blob/0529b486d05b3052d6dbc6f591d0169677b72565/agents-api%2Fmigrations%2Fmigrate_1716013793_session_cache.py#L9-L16 View Diff

---

The following files were not reviewed because our filter identified them as typically non-human-readable or less important files (e.g., dist files, package.json, images). If this is an error, please let us know.

• agents-api/poetry.lock
• agents-api/pyproject.toml

[creatorrr]

Lgtm