OpenAdapt
OpenAdapt copied to clipboard
Support Gemini 1.5 via `openadapt.adapter.gemini` + `LosslessReplayStrategy`
Feature request
From https://ateam-members.slack.com/archives/C050HTZ8HE0/p1708072362007929:
Google released Gemini 1.5 and tested it with 10 million tokens: https://blog.google/technology/ai/google-gemini-next-generation-model-february-2024/#sundar-note
Gemini v1 API documentation: https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/gemini#sample_requests
PROBLEM STATEMENT:
We wish to implement an openadapt.adapter
for Gemini 1.5, and parameterize/refactor StatefulReplayStrategy
.
COMPLETION CRITERIA:
This Issue will be complete once there is:
- An
openadapt.adapter
that implements the Gemini API - A refactored
StatefulReplayStrategy
that: a. supports the larger context window size b. implements a more general prompt (e.g. implement screenshots in https://github.com/OpenAdaptAI/OpenAdapt/blob/main/openadapt/strategies/stateful.py#L126 and remove / simplify https://github.com/OpenAdaptAI/OpenAdapt/blob/main/openadapt/events.py#L719)
Motivation
Support the largest context window model available in order to include full resolution Set-of-Mark prompts (i.e. via https://github.com/OpenAdaptAI/OpenAdapt/issues/543 + https://github.com/microsoft/SoM/pull/19) across full recording.