opentelemetry-dotnet icon indicating copy to clipboard operation
opentelemetry-dotnet copied to clipboard

Published 20 hours ago • verified publisher icon open-telemetry

Investigating Codecov

Open TimothyMothra opened this issue 2 years ago • 1 comments

I'm investigating why Codecov often reports Impacted Files that are not modified by the PR.

Take this PR as an example: #3546.

This PR changes 3 files related to Serilog, but Codecov is reporting 19 Impacted Files from unrelated projects! (Codecov)

SCREENSHOT: CODECOV FOR PR 3546

image

Two things stand out to me...

  1. Codecov is reporting all the files that I recently increased coverage to 100%. (PR 3476) The reason for this is that 3546 was forked off an earlier version of main, before 3476 was merged. I think it's safe to ignore these as they should be omitted from future PRs, or if the current PR is rebased.

    SCREENSHOT: COMMIT IDs

    Codecov shows the base commit (a789bc0): image

    Commit history: image

  2. There are a few EventSource classes in this report! This is reporting that some methods were added to coverage, and some were removed.

    SCREENSHOT: src/OpenTelemetry/Internal/OpenTelemetrySdkEventSource.cs

    image

    SCREENSHOT: src/OpenTelemetry.Exporter.ZPages/Implementation/ZPagesExporterEventSource.cs

    image

    SCREENSHOT: src/OpenTelemetry.Exporter.Prometheus.HttpListener/Internal/PrometheusExporterEventSource.cs

    image

    The current theory is that some of our functional/integration tests are experiencing concurrency issues and executing different blocks during their tests.

Next Steps

  1. Can Codecov be configured to compare PRs against the current head of main? Currently comparing from where a PR was branched creates noise in PRs.
  2. Monitor which files are showing up in Codecov Reports for future investigations.
    • There may be a bug in either the test or the product code.
    • If these were 100% covered by dedicated unit tests, they shouldn't show up in the variability of other tests.
  3. Consider removing Integration Tests from Codecov. Assuming these are the source of the variability, this would help reduce noise in PRs.

TimothyMothra avatar Aug 06 '22 00:08 TimothyMothra

Shared my findings in today's SIG meeting (2022-08-09).

Will continue to work to improve code coverage, this will remove some of the variability. I'll also follow up with Eddy if he knows any better configurations for Codecov.

TimothyMothra avatar Aug 09 '22 18:08 TimothyMothra