maestro
maestro copied to clipboard
Allow for customizing the prompt
User feedback on assertWithAI
I like
assertWithAIfor detecting defects. It helps to limit the unexpected changes but I miss something to tell the model what defects it should ignore. For example, localization changes, I have one situation where I test the app in a spanish locale but since the email for the login has english names, the model reported a defect that the locale doesn't match.
To be honest, we expected this feedback (see https://github.com/mobile-dev-inc/maestro/issues/1222#issuecomment-2273404992), but preferred to not implement it until users actually request it.
Ideas
(ordererd from most basic to most powerful)
1. Per-command list of false positives (injected into the final prompt)
- assertWithAI:
assertion: There is a login screen visible
falsePositives:
- Both English and Spanish text being visible
- <something-else>
Cons:
- why limit the flexibility? Feels wrong to me. Better just give users the control of the whole prompt.
2. Per-command list customizable prompt
- assertWithAI:
assertion: There is a login screen visible
prompt: Ignore problems related to both English and Spanish being visible
Cons:
- our current prompt is quite long and including it all every time, on the commands, would be tedious.
3. Customize the default prompts for AI-powered commands
Allow for customization on per-workspace level, with possible override on per-command level.
Define prompts
We'd let you configure the prompt template in .maestro/config.yaml:
ai:
defaultPrompts:
assertNoDefectsWithAI: ./prompts/my_assert_no_defects_ai_prompt_template.txt
assertWithAI: ./prompts/my_assert_with_ai_prompt_template.txt
customPrompts:
noBadWords: ./prompts/badwords_filter.txt
That's what the workspace directory hierarchy would look like in this example:
.
├── config.yaml
└── prompts
├── badwords_filter.txt
├── my_assert_no_defects_ai_prompt_template.txt
└── my_assert_with_ai_prompt_template.txt
Of course, users would be free to structure it however they want.
Use prompts
We would introduce some syntax to refer to the prompts prefefined on the per-workspace level in the per-command level. For example:
- assertWithAI:
prompt: noBadWords # refers to the contents of the "noBadWords" prompt
Or, to not introduce new arguments, we could re-use JavaScript for this:
- assertWithAI:
prompt: ${prompts.noBadWords} # "prompts" would be the JS object, set by Maestro