OpenAdapt
OpenAdapt copied to clipboard
feat(cursor): Add CursorReplayStrategy with red dot painting and self-correction
Fixes #760 /claim #760
What kind of change does this PR introduce?
Feature
Summary
This PR introduces the CursorReplayStrategy feature, which allows the AI model to suggest coordinates for a red dot on a screenshot representing the intended action target location. This new strategy provides the ability to visualize and self-correct initial suggestions, improving replay accuracy. It also includes an update to the screenshot capturing functionality to support capturing all monitors.
Checklist
- [x] My code follows the style guidelines of OpenAdapt
- [x] I have performed a self-review of my code
- [ ] If applicable, I have added tests to prove my fix is functional/effective
- [x] I have linted my code locally prior to submission
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation (e.g. README.md, requirements.txt)
- [ ] New and existing unit tests pass locally with my changes
How can your code be run and tested?
- Test the
CursorReplayStrategyby running the AI model and verifying the red dot is correctly suggested and visualized on the screenshot. - Use the self-correction mechanism to adjust the red dot position and verify the corrections are applied accurately.
Other information
No additional context needed.
Thank you @onyedikachi-david !
Can you please update your PR description with images containing the screenshot with the red dot? Ideally multiple over time as the model self corrects.