Bug Report

[superlopuh]

I recently switched to codecov/codecov-action@v4, and since then I've not been seeing any line by line coverage, on the website or in any of the actions.

Here is a recent example:

https://github.com/xdslproject/xdsl/pull/2621

Here's the relevant action for that PR:

https://github.com/xdslproject/xdsl/actions/runs/9188980377/job/25269976878

And the config:

    - name: Upload coverage to Codecov
      if: matrix.python-version == '3.10'
      uses: Wandalen/wretry.action@v1
      with:
        action: codecov/codecov-action@v4
        attempt_delay: 10000
        attempt_limit: 10
        with: |
          fail_ci_if_error: true
          verbose: true
          files: coverage.xml
          token: ${{ secrets.CODECOV_TOKEN }}

On the codecov website, I see this for every file:

There was a problem getting the source code from your provider. Unable to show line by line coverage.

How do we get code coverage back?

[drazisil-codecov]

Hi @superlopuh ,

Can you try changing the fetch depth, as explained here https://docs.codecov.com/docs/environment-specific-requirements#github-actions ?

(That's newly documented as of a few days ago, it was missed when we switched to v4.

[superlopuh]

Thank you for the suggestion. I tried it out in these two PRs:

https://github.com/xdslproject/xdsl/pull/2650 https://github.com/xdslproject/xdsl/pull/2651

And the diff is still missing.

[drazisil-codecov]

Hi @superlopuh ,

Interesting , it still isn't finding your repo files. How are you calling the action? Is it in a normal job with the repo checked out? A standalone job? A Docker container? Some other way? :)

[superlopuh]

I'm not sure that I understand your question, here's the action file: https://github.com/xdslproject/xdsl/blob/bd7ffdfa722561bb5bb501a5fc400229c3aff366/.github/workflows/ci-mlir.yml

It runs on every PR and when merged to main. It runs on the GitHub CI, runs a bunch of tests, combines the coverage, and then uploads to CodeCov.

[superlopuh]

One of the things I'm a little confused by is this message in the CodeCov website:

There was a problem getting the source code from your provider. Unable to show line by line coverage.

Which suggests that there's a different problem to the PR not being able to display the diff, but rather in how our coverage data is collected and uploaded.

[drazisil-codecov]

Hi @superlopuh

You actually have two issues:

• Codecov can't match your coverage files to any files in your repo, due to not being able to get your repo file list at upload time

• You are seeing an error in codecov.io because we are unable to fetch your repo file (at the time you are trying to view it) This is usually due to either a bad path, or bad access permissions from the token we use (should be yours at that time)

Let me address one at a time, please 🙏

[drazisil-codecov]

For the first, can you please add root_dir: xdsl to your codecov-action values? Let's see if that helps.

[superlopuh]

Thank you! What is the best way to see whether it's helped? Is it to merge the PR and see if the changes are reflected on the codecov.io website? Or is there something I can see in the logs?

Here are the logs for the run with the root_dir: xdsl included:

info - 2024-05-29 15:01:29,236 -- ci service found: github-actions
debug - 2024-05-29 15:01:29,238 -- versioning system found: <class 'codecov_cli.helpers.versioning_systems.NoVersioningSystem'>
debug - 2024-05-29 15:01:29,239 -- versioning system found: <class 'codecov_cli.helpers.versioning_systems.NoVersioningSystem'>
warning - 2024-05-29 15:01:29,240 -- No config file could be found. Ignoring config.
debug - 2024-05-29 15:01:29,240 -- No codecov_yaml found
debug - 2024-05-29 15:01:29,241 -- Starting create commit process --- {"commit_sha": "d6020e7bafcd221934dac65df9d4d29a6c361ec5", "parent_sha": null, "pr": "2650", "branch": "sasha/codecov", "slug": "xdslproject/xdsl", "token": "2******************", "service": "github", "enterprise_url": null}
info - 2024-05-29 15:01:29,367 -- Process Commit creating complete
debug - 2024-05-29 15:01:29,368 -- Commit creating result --- {"result": "RequestResult(error=None, warnings=[], status_code=201, text='{\"message\":null,\"timestamp\":\"2024-05-29T15:01:29.332896Z\",\"ci_passed\":null,\"state\":null,\"repository\":{\"name\":\"xdsl\",\"is_private\":false,\"active\":true,\"language\":\"python\",\"yaml\":{\"fixes\":[\"xdsl/::\"],\"ignore\":[\"(?s:.*/[^\\\\\\\\/]*\\\\\\\\.ipynb)\\\\\\\\Z\"],\"codecov\":{\"notify\":{\"wait_for_ci\":true},\"require_ci_to_pass\":true},\"coverage\":{\"range\":[80.0,90.0],\"round\":\"down\",\"status\":{\"patch\":{\"default\":{\"enabled\":false}},\"project\":{\"default\":{\"target\":1.0,\"enabled\":true,\"threshold\":0.1}}},\"precision\":2}}},\"author\":null,\"commitid\":\"d6020e7bafcd221934dac65df9d4d29a6c361ec5\",\"parent_commit_id\":null,\"pullid\":2650,\"branch\":\"sasha/codecov\"}')"}
==> Running command '/home/runner/work/_actions/Wandalen/codecov-action/dist/codecov -v create-report'
/home/runner/work/_actions/Wandalen/codecov-action/dist/codecov -v create-report --git-service github -C d6020e7bafcd221934dac65df9d4d29a6c361ec5 -Z
info - 2024-05-29 15:01:30,072 -- ci service found: github-actions
debug - 2024-05-29 15:01:30,075 -- versioning system found: <class 'codecov_cli.helpers.versioning_systems.NoVersioningSystem'>
debug - 2024-05-29 15:01:30,076 -- versioning system found: <class 'codecov_cli.helpers.versioning_systems.NoVersioningSystem'>
warning - 2024-05-29 15:01:30,077 -- No config file could be found. Ignoring config.
debug - 2024-05-29 15:01:30,077 -- No codecov_yaml found
debug - 2024-05-29 15:01:30,078 -- Starting create report process --- {"commit_sha": "d6020e7bafcd221934dac65df9d4d29a6c361ec5", "code": "default", "slug": "xdslproject/xdsl", "service": "github", "enterprise_url": null, "token": "2******************"}
info - 2024-05-29 15:01:30,182 -- Process Report creating complete
debug - 2024-05-29 15:01:30,182 -- Report creating result --- {"result": "RequestResult(error=None, warnings=[], status_code=201, text='{\"external_id\":\"b07a7dfe-d631-4e0c-9d52-5ea978106109\",\"created_at\":\"2024-05-29T15:01:30.147324Z\",\"commit_sha\":\"d6020e7bafcd221934dac65df9d4d29a6c361ec5\",\"code\":null}')"}
info - 2024-05-29 15:01:30,182 -- Finished creating report successfully --- {"response": "{\"external_id\":\"b07a7dfe-d631-4e0c-9d52-5ea978106109\",\"created_at\":\"2024-05-29T15:01:30.147324Z\",\"commit_sha\":\"d6020e7bafcd221934dac65df9d4d29a6c361ec5\",\"code\":null}"}
==> Running command '/home/runner/work/_actions/Wandalen/codecov-action/dist/codecov -v do-upload'
/home/runner/work/_actions/Wandalen/codecov-action/dist/codecov -v do-upload -Z -f coverage.xml --git-service github -C d6020e7bafcd221934dac65df9d4d29a6c361ec5 --network-root-folder xdsl
info - 2024-05-29 15:01:30,885 -- ci service found: github-actions
debug - 2024-05-29 15:01:30,887 -- versioning system found: <class 'codecov_cli.helpers.versioning_systems.NoVersioningSystem'>
debug - 2024-05-29 15:01:30,889 -- versioning system found: <class 'codecov_cli.helpers.versioning_systems.NoVersioningSystem'>
warning - 2024-05-29 15:01:30,889 -- No config file could be found. Ignoring config.
debug - 2024-05-29 15:01:30,889 -- No codecov_yaml found
debug - 2024-05-29 15:01:30,890 -- Starting upload processing --- {"branch": "sasha/codecov", "build_code": "9288421232", "build_url": "https://github.com/xdslproject/xdsl/actions/runs/9288421232", "commit_sha": "d6020e7bafcd221934dac65df9d4d29a6c361ec5", "disable_file_fixes": false, "disable_search": false, "enterprise_url": null, "env_vars": {}, "files_search_exclude_folders": [], "files_search_explicitly_listed_files": ["coverage.xml"], "files_search_root_folder": "/home/runner/work/xdsl/xdsl", "flags": [], "git_service": "github", "handle_no_reports_found": false, "job_code": "CI - MLIR-based Testing", "name": "9288421232", "network_filter": null, "network_prefix": null, "network_root_folder": "xdsl", "plugin_names": ["xcode", "gcov", "pycoverage"], "pull_request_number": "2650", "report_code": "default", "slug": "xdslproject/xdsl", "token": "2******************", "upload_file_type": "coverage"}
debug - 2024-05-29 15:01:30,891 -- Selected preparation plugins --- {"selected_plugins": ["<class 'codecov_cli.plugins.xcode.XcodePlugin'>", "<class 'codecov_cli.plugins.gcov.GcovPlugin'>", "<class 'codecov_cli.plugins.pycoverage.Pycoverage'>"]}
debug - 2024-05-29 15:01:30,891 -- Running preparation plugin: <class 'codecov_cli.plugins.xcode.XcodePlugin'>
debug - 2024-05-29 15:01:30,891 -- Running xcode plugin...
warning - 2024-05-29 15:01:30,891 -- xcrun is not installed or can't be found.
debug - 2024-05-29 15:01:30,891 -- Running preparation plugin: <class 'codecov_cli.plugins.gcov.GcovPlugin'>
debug - 2024-05-29 15:01:30,891 -- Running gcov plugin...
warning - 2024-05-29 15:01:30,979 -- No gcov data found.
debug - 2024-05-29 15:01:30,979 -- Running preparation plugin: <class 'codecov_cli.plugins.pycoverage.Pycoverage'>
info - 2024-05-29 15:01:31,051 -- Generating coverage.xml report in /home/runner/work/xdsl/xdsl/xdsl

I'm a little concerned by the fact that it can't find the codecov.yml in the root folder.

[drazisil-codecov]

@thomasrockhu-codecov I think I need to tag you in here. It's unclear to me why the CLI can't find the file network. Using the action.

[superlopuh]

Hi @drazisil-codecov @thomasrockhu-codecov, any follow-up on this? I'd be happy with any pointers, I'm just not sure where to start looking.

[thomasrockhu-codecov]

hi @superlopuh sorry I haven't gotten a chance to dig into this. I'll take a look over the weekend

[superlopuh]

Thank you!

[sblauth]

I have a similar issue with my codecov reports, so I think I can chime in here. I noticed that for your builds, the coverage.xml file created in the CI is missing a list of files, which should be at the top of the file. See, e.g., here: https://app.codecov.io/gh/xdslproject/xdsl/commit/e8de1367d7db2557dc53b6047c0c5aaa8236176c (if you click on the download button, I cannot link the file directly).

The same is happening for my project, e.g., here https://app.codecov.io/github/sblauth/cashocs/commit/77db69b35f06f0584754ac0a85ae49a05b12a106

I noticed that on an older branch, the file includes all files in the git, see here: https://app.codecov.io/github/sblauth/cashocs/commit/da532163c8524dd37e59b85221c083c9c64aee7a

Note that this was created with v3 of the action. So something has changed, perhaps with the codecov-action, so that this is not present anymore. And from what I've gathered, it is important for codecov, that this list is present in order to properly display the line by line coverage.

Maybe someone of the codecov team can take a deeper look into this?

[superlopuh]

Turns out we were missing the codecov token in GH. There are still some things that aren't very clear to me, such as why it can't find the codecov.yml, but that seems like a topic for a different issue. Thank you all for your help!