All-Hands-AI
Add Codecov test coverage
Now that we have a fair amount of test coverage, we could start tracking coverage using Codecov. I know we have frontend tests as well but they don't seem to be part of CI yet. I'll leave that to our frontend experts to solve :)
I ran this on my fork:
With unit-test coverage only, we have 29.67% coverage for backend.
With integration test coverage, we have 58.16% coverage for backend.
NOTE: for the CodeCov to actually work on CI, someone with write access to the repo, a.k.a. maintainer, needs to authorize CodeCov to access this repo (by logging into https://app.codecov.io/ using their GitHub account, and link opendevin/opendevin), and then add the token to repository secrets: https://docs.codecov.com/docs/adding-the-codecov-token
As someone who got bit by the massive codecov breach this makes me a bit nervous 😅
Are there any open source python coverage tools we could use? Or is this the best option?
As someone who got bit by the massive codecov breach this makes me a bit nervous
That's very interesting. I've been using codecov for years but I wasn't aware of this. If I understand correctly, this is more of a concern when hosting codecov in your own network? On a public GitHub repo, there's no concern regarding code leakage. What about secrets? According to https://docs.github.com/en/actions/security-guides/using-secrets-in-github-actions, secrets are not available to GitHub Actions (CI) unless they are explicitly provided. Currently we only expose secrets (LLM api key) to dogfood tasks (issue solver and pr reviewer). The test actions that CodeCov runs in, have no access to secrets.
Are there any open source python coverage tools we could use?
Codecov-action, the GitHub Action that we use to upload the xml reports, is indeed open-source: https://github.com/codecov/codecov-action
Codecov server itself is also "source available": https://github.com/codecov/self-hosted
There are alternatives to codecov but according to https://github.com/marketplace?category=code-quality&type=actions&verification=&query=sort%3Apopularity-desc+coverage, codecov is the most popular one. Apparently, it's far more popular than other ones.
The unit test failure seems to be a transient failure (might be fixed by https://github.com/OpenDevin/OpenDevin/pull/1381). I don't have permission to rerun the CI.