Codecov Report Generation Constantly Delayed

[Ryang20718]

Describe the bug We rely on codecov to gate PR's. Specifically status checks. However, the SLA's are very hand wavy.

Often times, ingestion is down and we'll need to merge PR's without coverage report information.

Environment (please complete the following information): We are on linux and leverage codecov's cli tool to generate reports.

To Reproduce Sporadically, we see delays with status checks not getting sent.

Expected behavior Status checks should get sent within 5-10 minutes of coverage being uploaded.

Example ticket with relevant information https://codecovpro.zendesk.com/hc/en-us/requests/148605

We constantly are trying to track codecov report stability, but support usually gives a vague answer. Is there an epic tracking stability please?

ideally some sort of UI to view/track coverage status ingestion would be great. Currently we're just sitting ducks with no visibility on the ingestion status. Codecov status doesn't report an outage either

[Ryang20718]

cc @thomasrockhu-codecov

Was hoping to get more engineering/product observability here since it seems there's not a clear timeline on when/if this issue is being tracked/triaged/solved?

[jiaxin-lin]

We are experiencing this today severely, basically the upload says completed in github action but no coverage gets updated

[thomasrockhu-codecov]

@Ryang20718 @JLLLinn apologies, we are aware of the current issue and have a status page up here

[Ryang20718]

@thomasrockhu-codecov by chance is there a UI or status somewhere where we can track when these updates (in advance) are occurring?

codecov status checks currently are gating CI for us.

[hotrush]

Status checks should get sent within 5-10 minutes of coverage being uploaded.

I hope you meant 5-10 seconds 😄

@thomasrockhu-codecov we are still experiencing issues. Your status page saying it was resolved but our PRs opened an hour ago are still waiting for status to be reported. And it keeps happening with 50% probability in many repos.

Expected — Waiting for status to be reported

Should i create an issue or you are aware?

[pencil]

I'm still waiting for a GitHub status check for a commit I pushed 38 minutes ago despite the status page indicating that the issue has been resolved.

[Ryang20718]

for those of use on codecov pro, are there SLA's?

[pfuhrmann]

We have been waiting for the status to be reported for 8h now! We are on the Pro plan, but it seems like queues have no priorities or are just overloading or failing. This constantly blocks our pipelines, which impacts the business.

[thomasrockhu-codecov]

@pfuhrmann @Ryang20718 apologies it looks like our queues were still backed up. This should be resolved now

[Ryang20718]

@thomasrockhu-codecov we're seeing some cases of this again

[Ryang20718]

Hitting this again