Richard Abrich

### Feature request We would like to support prompting the model to generate PyAutoGUI code, like in OSWorld and elsewhere. This involves: - creating a new module under ‘openadapt.strategies’ -...

### Feature request We would like users to submit recordings via the Upload Recording functionality to be implemented in https://github.com/OpenAdaptAI/OpenAdapt/issues/724. ### Motivation Finetuning on data

Upload recordings from dashboard

1

### Feature request We would like to give users the option to share their recording data with us. This involves: - creating a script that configures an s3 bucket if...

### Feature request We would like to provide the user more context in the dashboard by animating the keyframe / parent action, e.g. with `requestAnimationFrame` and `canvas`. Each of the...

### Feature request We would like to let users select whether audio should be recorded immediately before starting the recording ### Motivation UX

[Bug]: model generated JSON snippet exceeds output context length

3

### Describe the bug ``` python -m openadapt.replay SegmentReplayStrategy "replay verbatim" ``` This happens when there are too many segments due to a lack of filtering, e.g. ``` ('```json\n' 14:20:57...

### Feature request This involves refactoring `openadapt.adapters.[openai|anthropic|google]` into `openadapt.drivers`, and implementing a new `openadapt.adapters.prompt` to manage them. ### Motivation https://github.com/OpenAdaptAI/OpenAdapt/issues/674: > OpenAI often gives this error when parsing images: "Your...

### Feature request We would like to implement multi shot prompting. This involves running openadapt.replay on multiple actions simultaneously. This also involves selecting relevant recordings, using user specified natural language...

Design: Fine-Tuning

5

### Feature request We would like to implement fine-tuning. This task involves considering the tradeoffs between various approaches to improving action completions and outcome evaluation via fine-tuning. More generally, this...

Implement TextGrad (bounty)

9

### Feature request We would like to implement https://github.com/zou-group/textgrad. ### Motivation > It works out-of-the-box for a variety of tasks, where the users only provide the objective function without tuning...