make integration test a little looser

[rbren]

@li-boxuan how bad an idea is this?

The idea is that instead of string-matching the prompts, we just iterate through the responses, one by one

[li-boxuan]

The idea is that instead of string-matching the prompts, we just iterate through the responses, one by one

I thought so at the beginning as well. The upside is clear - CI won't fail if one just modifies the prompt by a little bit.

The downsides are debatable though:

1. One can break the agent, say, by always sending nonsense to LLM, and still get all these tests pass. A malicious attempt will mostly likely be caught by reviewers, but a true bug may not.
2. I feel like keeping the prompt files as test artifacts is a good way for developers and even users to learn what are being sent to the LLM.

[rbren]

Going to close this--seems like there hasn't been as much integration test pain lately. But we should definitely work on a better way to generate the prompt files

[li-boxuan]

That’s on my radar! Planning to work on that next week.