OpenAdapt icon indicating copy to clipboard operation
OpenAdapt copied to clipboard

feat(cursor): Add CursorReplayStrategy with red dot painting and self-correction

Open onyedikachi-david opened this issue 1 year ago • 1 comments


Fixes #760 /claim #760

What kind of change does this PR introduce?

Feature

Summary

This PR introduces the CursorReplayStrategy feature, which allows the AI model to suggest coordinates for a red dot on a screenshot representing the intended action target location. This new strategy provides the ability to visualize and self-correct initial suggestions, improving replay accuracy. It also includes an update to the screenshot capturing functionality to support capturing all monitors.

Checklist

  • [x] My code follows the style guidelines of OpenAdapt
  • [x] I have performed a self-review of my code
  • [ ] If applicable, I have added tests to prove my fix is functional/effective
  • [x] I have linted my code locally prior to submission
  • [x] I have commented my code, particularly in hard-to-understand areas
  • [ ] I have made corresponding changes to the documentation (e.g. README.md, requirements.txt)
  • [ ] New and existing unit tests pass locally with my changes

How can your code be run and tested?

  1. Test the CursorReplayStrategy by running the AI model and verifying the red dot is correctly suggested and visualized on the screenshot.
  2. Use the self-correction mechanism to adjust the red dot position and verify the corrections are applied accurately.

Other information

No additional context needed.


onyedikachi-david avatar Jul 04 '24 10:07 onyedikachi-david

Thank you @onyedikachi-david !

Can you please update your PR description with images containing the screenshot with the red dot? Ideally multiple over time as the model self corrects.

abrichr avatar Jul 05 '24 22:07 abrichr