Feedback on the new 'Flags' tab

[aj-codecov]

Thanks for dropping by! 👋

We've recently released a whole new area of the app focused on flags. If you're not seeing it yet, keep your eyes peeled as we're slowly rolling it out to our users.

• What do you think about the new Flags tab?
• What would you'd like to see improved?
• Has it posed any challenges or questions?
• How do you use it?
• How could the report better help your workflow?
• Any general thoughts you'd like to share! We greatly appreciate your time and thoughts - looking forward to hearing from you ❤

Codecov team

This issue is intended to share and collect feedback about the tool. If you have support needs or questions, please see our support page.

[austinpray-mixpanel]

There should be an easy way to delete flags from the flags tab.

All of our flags are carry forward:

flag_management:
  default_rules:
    carryforward: true

Which means we have lots of dangling flags left over from when we were configuring codecov for the first time. We experimented with different naming conventions and such. There's no way to clear these flags out, and I'm worried they will contribute to weird coverage numbers.

[guohui-gao]

There should be an easy way to delete flags from the flags tab.

All of our flags are carry forward:

flag_management:
  default_rules:
    carryforward: true

Which means we have lots of dangling flags left over from when we were configuring codecov for the first time. We experimented with different naming conventions and such. There's no way to clear these flags out, and I'm worried they will contribute to weird coverage numbers.

Agree +1

[aj-codecov]

We really appreciate the feedback! We've heard the need for clearing out older flags and are looking at it as work to be done in the next few months. We also have a couple other improvements to the flags tab that will allow you to search and select only a few flags you want to see at a time.

[mattlewis92]

Hi there,

We have a few pieces of feedback that I think would really improve the flags page:

1. Add a branch selector so you can see the flag coverage for a particular branch, right now it's an average of all flags across all branches which isn't really that helpful:
2. Show the date range start and end coverage percentages, instead of this average coverage over the selected time period:

Our use case is that we want the ability to track individual team coverage over time (where each team is a flag) on our monorepo. Right now because of the way that the flags report is averaged over time, across all PRs this severely limits the usefulness of it.

[FelixS90]

After enabling the feature, the status stays at "Pulling historical data" now for roughly half an hour - although we enabled it right after the first scan on the default branch.

What could be the reason? Let me know if you need more info.

[eliatcodecov]

@FelixS90 our historical data pull feature relies on an asynchronous task queue to function. This is essential since, in many cases, users have a a large amount of historical coverage data that we need to pull, aggregate, analyze, and store in a time series friendly format.

With this in mind, there are two reasons a historical data pull can take awhile:

1. The repo in question has lots of historical data (doesn't seem to be your case)
2. The historical data task queue is first in first out. So if there were other repos queued ahead of yours they would need to have been processed first before processing yours.

We generally try to autoscale processing for that queue to keep queue length under control, but it could have just been a particularly busy time in your case. Apologies for the delay, but is the feature working for that particular repository now?

[austinpray-mixpanel]

I get a consistent "There was a problem getting flags data" after a considerable wait time.

JS console output: app.codecov.io-1667315704716.log

[RobertBrunhage]

I don't understand the flags.

In the flag section it's stated 5.5%

But looking a the normal page and the folder that is tagged, that folder is 17.5%

My idea was to have each package have it's own coverage but they don't match in the flag section.

[Wayonb]

I am not seeing data for my configured flags 🤔

[anyweez]

I also see "There was a problem getting flags data" - it appears to work once every 10-15 attempts, but only after a lengthy delay.

From browser console:

Sentry Logger [log]: Event processor "InboundFilters" dropped event
main.ec98bddc.js:2 Sentry Logger [log]: Adding outcome: "event_processor:error"
main.ec98bddc.js:2 Sentry Logger [log]: An event processor returned null, will not send event.
script.js:1 Uncaught Error: It looks like your website URL has been changed. To ensure the proper functioning of your banner, update the registered URL on your CookieYes account (More > Site Settings) and reload this page to retry. If the issue persists, contact us at https://www.cookieyes.com/support.
    at script.js:1:86908
    at script.js:1:97140
    at script.js:1:97143
(anonymous) @ script.js:1
(anonymous) @ script.js:1
(anonymous) @ script.js:1
manifest.json:1 Manifest: Line: 1, column: 1, Syntax error.
main.ec98bddc.js:2 Sentry Logger [log]: Sending outcomes: [{…}]
flags:1 Access to fetch at 'https://api.codecov.io/graphql/gh' from origin 'https://app.codecov.io' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.
main.ec98bddc.js:2          POST https://api.codecov.io/graphql/gh net::ERR_FAILED 502

[aj-codecov]

Hi all, we are actively working on the issues causing the error states, when you encounter those, changing the time scale can oftentimes unblock you.

@RobertBrunhage I'd love to dig in further with you on your particular issue, can you give me a specific repo/flag (if it's public) so I can hopefully get you an answer?

[Wayonb]

Is this a known issue where some of the flags are empty?

[aj-codecov]

@Wayonb flags stick around once they have been created at any point in time, for example 'sdk-ptyhon' was likely a flag at one time, which was corrected to 'sdk-python'. In the case of the empties you see in your flags tab, is it possible each of those flags was uploaded to once and then never again? The lack of data makes me wonder if they were even uploaded to with a blank coverage report? It's hard to say exactly without more info, but those are the possible avenues I'm aware of that might cause this scenario.

[Wayonb]

@aj-codecov Thanks. I guess the only way to remove those is to reset all the data?

[sungam3r]

From https://github.com/codecov/Codecov-user-feedback/issues/27#issuecomment-1314018066 :

My idea was to have each package have it's own coverage

I thought so as well. Is that true for flags feature? My case is Xabaril/AspNetCore.Diagnostics.HealthChecks repo https://app.codecov.io/github/Xabaril/AspNetCore.Diagnostics.HealthChecks/flags with ~40 projects and I would like to track coverage independently for each project (package). Can I archive that by flags feature? I planned to add flags argument to each GitHub CI workflow. Note that in my case I run only those workflows for which some files were changes, i.e. if PR touches only Redis project then only Redis CI file run with only tests for Redis (and coverage data only for Redis), not all ~40 projects. As I understand in that case each time I upload totally different coverage results to codecov. Maybe I do upload totally wrong and it can be achived by other means, not flags?

[ascopes]

I've followed the instructions, but all I get is

Clicking the link to show the reported uploads shows all the coverage data, so no idea what is going on. I've had to disable this entirely now since it no longer reports any coverage at all when I use flags, which is a bit of a pain.

There is nothing to actually indicate what the issue is though, so for now I won't be able to make use of this, which is unfortunate as this feature would be really useful in my project for separating out integration and unit test coverage.

If there is a way to feed back errors more clearly on these pages, it would be greatly appreciated and of great use in the future.

https://app.codecov.io/gh/ascopes/java-compiler-testing/commit/0cb6c43a695b20c446a824c25b8095632b8eeed8

[aj-codecov]

@sungam3r I believe you've got the right idea, only thing I'd say is if you're not uploading all coverage every time you commit, you're going to want to use Carryforward flags as described here: https://docs.codecov.com/docs/carryforward-flags

[aj-codecov]

@ascopes In your first screenshot I can see you've attempted to tie two flags to one upload, Flags must be associated with only one upload to work properly, so you would need to do an upload of coverage for integration tests and an upload of coverage for unit tests where each is flagged with their corresponding label further reading here: https://docs.codecov.com/docs/flags#one-to-one-relationship-of-flags-to-uploads I'm not sure why we're flagging the upload as empty offhand, it could be because Codecov isn't recognizing the format, there was an error processing, or because the upload is actually empty, it's hard to say without a dive into the coverage report itself. Error messaging is a perpetual work in progress and this is certainly an area we aren't quite cutting it at the moment. If we can get to the root of what happened in your case I'm happy to get a relevant error message into our product pipeline.

[sungam3r]

@aj-codecov I have configured carryforward option for each project: https://github.com/Xabaril/AspNetCore.Diagnostics.HealthChecks/blob/master/.github/codecov.yml

Also I set individual flags in each CI pipeline like this one https://github.com/Xabaril/AspNetCore.Diagnostics.HealthChecks/blob/d4c751bdeaa363dec39bb30bd49cead817aae5f5/.github/workflows/healthchecks_applicationstatus_ci.yml#L64

but after the latest upload https://github.com/Xabaril/AspNetCore.Diagnostics.HealthChecks/pull/1616 I still see only coverage from only touched files and overall coverage dropped again after full report upload.

[ascopes]

@aj-codecov Hi, thanks for the response

@ascopes In your first screenshot I can see you've attempted to tie two flags to one upload, Flags must be associated with only one upload to work properly, ...

I was under the impression that the feature flags would handle this automatically since they consume specific paths related to specific flags. Is this not the case?

flag_management:
  default_rules:
    carryforward: true
    statuses:
      - type: project
        target: auto
        threshold: 1%
      - type: patch
        target: 90%

  individual_flags:
    # Unit testing coverage
    - name: unit
      carryforward: true
      paths:
        - '**/target/site/jacoco/unit/jacoco*.xml'
      statuses:
        - type: project
          target: auto
          threshold: 65%
        - type: patch
          target: 90%

    # Integration testing coverage
    - name: integration
      carryforward: true
      paths:
        - '**/target/site/jacoco/int/jacoco*.xml'
      statuses:
        - type: project
          target: auto
          threshold: 0%
        - type: patch
          target: 0%

[vigenere23]

My flags page is not loading the coverages:

Any idea what's happening?

Logs

The connection to https://clientstream.launchdarkly.com/eval/60f9b3cf32beb9244b10f3d1/<SOME_LONG_TOKEN> was interrupted while the page was loading. main.ad642d24.js:2:550053
[LaunchDarkly] Error on stream connection: {"isTrusted":true}, will continue retrying every 1000 milliseconds. main.ad642d24.js:2:407351
Sentry Logger [error]: Error while triggering instrumentation handler.
Type: fetch
Name: <anonymous>
Error: TypeError: o is undefined
    Et https://app.codecov.io/static/js/main.ad642d24.js:2
    en https://app.codecov.io/static/js/main.ad642d24.js:2
    m https://app.codecov.io/static/js/main.ad642d24.js:2
    h https://app.codecov.io/static/js/main.ad642d24.js:2
main.ad642d24.js:2:414414
    99204/s/</t[n]/< https://app.codecov.io/static/js/main.ad642d24.js:2
    u https://app.codecov.io/static/js/main.ad642d24.js:2
    n https://app.codecov.io/static/js/main.ad642d24.js:2
    m https://app.codecov.io/static/js/main.ad642d24.js:2
    h https://app.codecov.io/static/js/main.ad642d24.js:2
Sentry Logger [log]: No outcomes to send main.ad642d24.js:2:414414
Some cookies are misusing the recommended “SameSite“ attribute 7
Segment snippet included twice. flags line 512 > injectedScript:1:126
Sentry Logger [log]: Integration installed: InboundFilters main.ad642d24.js:2:414414
Sentry Logger [log]: Integration installed: FunctionToString main.ad642d24.js:2:414414
Sentry Logger [log]: Integration installed: TryCatch main.ad642d24.js:2:414414
Sentry Logger [log]: Integration installed: Breadcrumbs main.ad642d24.js:2:414414
Sentry Logger [log]: Global Handler attached: onerror main.ad642d24.js:2:414414
Sentry Logger [log]: Global Handler attached: onunhandledrejection main.ad642d24.js:2:414414
Sentry Logger [log]: Integration installed: GlobalHandlers main.ad642d24.js:2:414414
Sentry Logger [log]: Integration installed: LinkedErrors main.ad642d24.js:2:414414
Sentry Logger [log]: Integration installed: Dedupe main.ad642d24.js:2:414414
Sentry Logger [log]: Integration installed: HttpContext main.ad642d24.js:2:414414
Sentry Logger [log]: [Tracing] Starting pageload transaction on scope main.ad642d24.js:2:414414
Sentry Logger [log]: Setting idle transaction on scope. Span ID: 89231477b0e5b4a9 main.ad642d24.js:2:414414
Sentry Logger [log]: [Tracing] Discarding transaction because it's not included in the random sample (sampling rate = 0.2) main.ad642d24.js:2:414414
Sentry Logger [log]: Integration installed: BrowserTracing main.ad642d24.js:2:414414
Sentry Logger [log]: Integration installed: Replay main.ad642d24.js:2:414414
Sentry Logger [log]: [Tracing] Starting 'ui.react.mount' span on transaction '/gh/vigenere23/disma/flags' (89231477b0e5b4a9). main.ad642d24.js:2:414414
Sentry Logger [log]: [Tracing] Finishing 'ui.react.mount' span on transaction '/gh/vigenere23/disma/flags' (89231477b0e5b4a9). main.ad642d24.js:2:414414
Sentry Logger [log]: [Tracing] Starting 'http.client' span on transaction '/gh/vigenere23/disma/flags' (89231477b0e5b4a9). 2 main.ad642d24.js:2:414414
Sentry Logger [log]: [Tracing] Starting 'http.client' span on transaction '/gh/vigenere23/disma/flags' (89231477b0e5b4a9). 2 main.ad642d24.js:2:414414
Loading failed for the <script> with source “https://static.ads-twitter.com/uwt.js”. flags:1:1
Sentry Logger [log]: [Replay] Using compression worker main.ad642d24.js:2:414414
Ignoring unsupported entryTypes: element. main.ad642d24.js:2:153138
Ignoring unsupported entryTypes: largest-contentful-paint. main.ad642d24.js:2:153138
Ignoring unsupported entryTypes: layout-shift. main.ad642d24.js:2:153138
Ignoring unsupported entryTypes: longtask. main.ad642d24.js:2:153138
Sentry Logger [log]: [Tracing] Starting 'http.client' span on transaction '/gh/vigenere23/disma/flags' (89231477b0e5b4a9). main.ad642d24.js:2:414414
Sentry Logger [log]: [Tracing] Finishing 'http.client' span on transaction '/gh/vigenere23/disma/flags' (89231477b0e5b4a9). 4 main.ad642d24.js:2:414414
Sentry Logger [log]: [Tracing] Starting 'http.client' span on transaction '/gh/vigenere23/disma/flags' (89231477b0e5b4a9). main.ad642d24.js:2:414414
Sentry Logger [log]: [Tracing] Starting 'http.client' span on transaction '/gh/vigenere23/disma/flags' (89231477b0e5b4a9). main.ad642d24.js:2:414414
Sentry Logger [log]: [Tracing] Finishing 'http.client' span on transaction '/gh/vigenere23/disma/flags' (89231477b0e5b4a9). main.ad642d24.js:2:414414
Sentry Logger [log]: [Tracing] Starting 'http.client' span on transaction '/gh/vigenere23/disma/flags' (89231477b0e5b4a9). main.ad642d24.js:2:414414
Sentry Logger [log]: [Tracing] Finishing 'http.client' span on transaction '/gh/vigenere23/disma/flags' (89231477b0e5b4a9). main.ad642d24.js:2:414414
Sentry Logger [log]: [Tracing] Starting 'http.client' span on transaction '/gh/vigenere23/disma/flags' (89231477b0e5b4a9). main.ad642d24.js:2:414414
Sentry Logger [log]: [Tracing] Finishing 'http.client' span on transaction '/gh/vigenere23/disma/flags' (89231477b0e5b4a9). main.ad642d24.js:2:414414
Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://clientstream.launchdarkly.com/eval/60f9b3cf32beb9244b10f3d1/<SOME_LONG_TOKEN>. (Reason: CORS request did not succeed). Status code: (null).
Sentry Logger [log]: [Tracing] Finishing 'http.client' span on transaction '/gh/vigenere23/disma/flags' (89231477b0e5b4a9). main.ad642d24.js:2:414414
Sentry Logger [log]: [Tracing] Finishing 'http.client' span on transaction '/gh/vigenere23/disma/flags' (89231477b0e5b4a9). main.ad642d24.js:2:414414
Sentry Logger [log]: [Tracing] Starting 'http.client' span on transaction '/:provider/:owner/:repo/flags' (89231477b0e5b4a9). main.ad642d24.js:2:414414
Request to access cookie or storage on “https://googleads.g.doubleclick.net/pagead/viewthroughconversion/781097871/?random=1674002832456&cv=11&fst=1674002832456&bg=ffffff&guid=ON&async=1&gtm=2wg1a1&u_w=1440&u_h=900&hn=www.googleadservices.com&frm=0&url=https%3A%2F%2Fapp.codecov.io%2Fgh%2Fvigenere23%2Fdisma%2Fflags&tiba=Codecov&rdp=1&auid=1383083016.1674002832&rfmt=3&fmt=4” was blocked because it came from a tracker and content blocking is enabled.
Sentry Logger [log]: [Tracing] Starting 'http.client' span on transaction '/:provider/:owner/:repo/flags' (89231477b0e5b4a9). main.ad642d24.js:2:414414
Sentry Logger [log]: [Tracing] Finishing 'http.client' span on transaction '/:provider/:owner/:repo/flags' (89231477b0e5b4a9). main.ad642d24.js:2:414414
Sentry Logger [log]: [Tracing] Starting 'http.client' span on transaction '/:provider/:owner/:repo/flags' (89231477b0e5b4a9). main.ad642d24.js:2:414414
Sentry Logger [log]: [Tracing] No active IdleTransaction main.ad642d24.js:2:414414
Sentry Logger [log]: [Tracing] Discarding transaction because its trace was not chosen to be sampled. main.ad642d24.js:2:414414
Sentry Logger [log]: Adding outcome: "sample_rate:transaction" main.ad642d24.js:2:414414
Sentry Logger [log]: [Tracing] Finishing 'http.client' span on transaction '/:provider/:owner/:repo/flags' (89231477b0e5b4a9). main.ad642d24.js:2:414414
Sentry Logger [log]: [Tracing] Finishing 'http.client' span on transaction '/:provider/:owner/:repo/flags' (89231477b0e5b4a9). main.ad642d24.js:2:414414
Sentry Logger [log]: [Tracing] No active IdleTransaction main.ad642d24.js:2:414414
Sentry Logger [log]: [Tracing] Starting navigation transaction on scope main.ad642d24.js:2:414414
Sentry Logger [log]: Setting idle transaction on scope. Span ID: 80df6c1167066ba2 main.ad642d24.js:2:414414
Sentry Logger [log]: [Tracing] Discarding transaction because it's not included in the random sample (sampling rate = 0.2) main.ad642d24.js:2:414414
Sentry Logger [log]: [Tracing] Starting 'http.client' span on transaction '/gh/vigenere23/disma/flags' (80df6c1167066ba2). main.ad642d24.js:2:414414
Sentry Logger [log]: [Tracing] Starting 'http.client' span on transaction '/:provider/:owner/:repo/flags' (80df6c1167066ba2). main.ad642d24.js:2:414414
Sentry Logger [log]: [Tracing] Finishing 'http.client' span on transaction '/:provider/:owner/:repo/flags' (80df6c1167066ba2). main.ad642d24.js:2:414414
Sentry Logger [log]: [Tracing] Finishing 'http.client' span on transaction '/gh/vigenere23/disma/flags' (80df6c1167066ba2). main.ad642d24.js:2:414414
Sentry Logger [log]: [Tracing] Starting 'http.client' span on transaction '/:provider/:owner/:repo/flags' (80df6c1167066ba2). main.ad642d24.js:2:414414
XHRPOSThttps://api.feedback.us.pendo.io/widget/pendo_ping
[HTTP/2 400 Bad Request 145ms]

Sentry Logger [log]: [Tracing] Finishing 'http.client' span on transaction '/:provider/:owner/:repo/flags' (80df6c1167066ba2). main.ad642d24.js:2:414414
Sentry Logger [log]: [Tracing] No active IdleTransaction main.ad642d24.js:2:414414
Sentry Logger [log]: [Tracing] Discarding transaction because its trace was not chosen to be sampled. main.ad642d24.js:2:414414
Sentry Logger [log]: Adding outcome: "sample_rate:transaction" main.ad642d24.js:2:414414

[vigenere23]

My flags page is not loading the coverages:

Any idea what's happening? Logs

As an update, I've realized that all my recent uploads have failed, with the message upload is empty, which is weird since their content is not empty at all. I'll try the proposed fixes of https://github.com/codecov/codecov-action/issues/844 and see if that fixes the problem.

[vigenere23]

Another update : after fixing the coverage upload, it does load the information now, thanks!

[pmalek]

This feature is pretty neat but I have 2 comments on it:

• the UI doesn't allow sorting or filtering flags which wouldn't require specifying concrete flags to list.

  Use case: I use version of a dependency I use as a flag. After some versions rolled out, I don't test against version n - 10 but it still shows up in the UI as the first one entry because it was one of the first ones to test. Now, at this moment I don't test against it nor do I care about its coverage yet it still show up with No report uploaded yet message with more recent (and more relevant for me at this stage) versions at the very end of the list, hidden behind Load More button.

• ~I'm not sure if I missed it somehow in the UI or it's not possible to achieve but in another project I have unit and integration tests running and I have corresponding flags for that purpose. It would be super cool to have information whether particular lines are covered by unit and/or integration tests.~

  ~Use case: target which test cases to extend e.g. unit or integration for increased coverage.~

  For this it seems there's already a tracking issue at https://app.codecov.io/gh/feedback/p/view-source-with-flags

[Artalus]

Dunno if I am using Flags the right way, but I felt they might provide a good introspection for which parts of my code are covered with which tests (unit, integration, system). Unfortunately, the UI only gives the total percentage of coverage, without any detalization per file.

What I would like to see is something like this:

• you click flag foo
• Codecov shows you a regular file tree, but the percentages now are only accounting for the tests from flag foo
• you click some file
• the lines are now colored in 3 (4? 5?) colors, unlike the regular view:
  • green - line covered by a test from foo
  • blue/teal/pink/whatever - line covered by some test, but not from foo
  • red - line not covered at all

[0xleizhang]

It would be very helpful to display the latest report upload time on the Flags page!

[tsinis]

I like the feature, but I'm also having issues with weird coverage showing, for example:

1. https://app.codecov.io/github/tsinis/sealed_world/flags world_countries should have 85+, I have no idea where 68.68 comes from :-/
2. badges are messy, they are not showing data for all flags but only for the most recently uploaded one (although there is coverage data for all of them), for example: vs

P.S. repo is open-source.

[sungam3r]

badges are messy, they are not showing data for all flags but only for the most recently uploaded one

+1

[Benoss]

I would really like to have:

• Branch context dropdown
• Last report uploaded time column (and be able to filter/order by this column so that we can check if CI/CD is uploading correctly the flag data)
• Order by Coverage% / Filter by more than/less than % (Find the flags under the baseline that we need to focus on)
• Order by Trend%/ Filter by more than/less than % (Find the flags that are not going well)
• Click on the flag to have a flag detail page with more info about this flag

[nickpalmer]

We would like the ability to delete a flag we have retired so we do not keep getting it on comments with carry forward at the same level.