OpenAdapt icon indicating copy to clipboard operation
OpenAdapt copied to clipboard

Support Gemini 1.5 via `openadapt.adapter.gemini` + `LosslessReplayStrategy`

Open abrichr opened this issue 11 months ago • 0 comments

Feature request

From https://ateam-members.slack.com/archives/C050HTZ8HE0/p1708072362007929:

Google released Gemini 1.5 and tested it with 10 million tokens: https://blog.google/technology/ai/google-gemini-next-generation-model-february-2024/#sundar-note

Gemini v1 API documentation: https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/gemini#sample_requests

PROBLEM STATEMENT:

We wish to implement an openadapt.adapter for Gemini 1.5, and parameterize/refactor StatefulReplayStrategy.

COMPLETION CRITERIA:

This Issue will be complete once there is:

  1. An openadapt.adapter that implements the Gemini API
  2. A refactored StatefulReplayStrategy that: a. supports the larger context window size b. implements a more general prompt (e.g. implement screenshots in https://github.com/OpenAdaptAI/OpenAdapt/blob/main/openadapt/strategies/stateful.py#L126 and remove / simplify https://github.com/OpenAdaptAI/OpenAdapt/blob/main/openadapt/events.py#L719)

Motivation

Support the largest context window model available in order to include full resolution Set-of-Mark prompts (i.e. via https://github.com/OpenAdaptAI/OpenAdapt/issues/543 + https://github.com/microsoft/SoM/pull/19) across full recording.

abrichr avatar Feb 28 '24 01:02 abrichr