camel icon indicating copy to clipboard operation
camel copied to clipboard

[BUG] Excessive Usage of OpenAI APIs on Every Git Push

Open dandansamax opened this issue 1 year ago • 2 comments

Required prerequisites

Problem description

We have identified a critical issue where the OpenAI APIs are being invoked on every push to the Git repository. This behavior is highly inefficient and potentially costly, as it results in unnecessary API calls regardless of whether they are needed for the particular code changes being pushed.

The expected behavior is that the OpenAI APIs should only be triggered under specific conditions, such as before merging a PR.

dandansamax avatar Nov 10 '23 14:11 dandansamax

What is more, we should require all unit tests that call LLM to provide text generated by LLM to simulate its output, so we can see in CI/CD whether the generated text is correctly formatted. Finally, before deciding to merge PR, we run a real output of LLM once.

Appointat avatar Nov 10 '23 17:11 Appointat

Do we really need the real output of LLM? As I checking several famous agent projects, all of them just use mock outputs. A possible reson is most of tests do not require the output contents but only require their formats. Also they do not have so many tests as we have. I wonder if our strict tests limited our development progress?

Can we remove some tests that are actually useless?

dandansamax avatar Nov 12 '23 12:11 dandansamax