feat: monitor daily webhooks state cron

[Udit-takkar]

trafficstars

What does this PR do?

• Fixes https://github.com/calcom/cal.com/issues/15121
• Fixes CAL-3763 (Linear issue number - should be visible at the bottom of the GitHub issue description)

Mandatory Tasks (DO NOT REMOVE)

• [ ] I have self-reviewed the code (A decent size PR without self-review might be rejected)
• [ ] I have added a Docs issue here if this PR makes changes that would require a documentation change
• [ ] I have added or modified automated tests that prove my fix is effective or that my feature works (PRs might be rejected if logical changes are not properly tested)

How should this be tested?

[linear[bot]]

CAL-3763 Trigger notification when daily webhook are not working

[github-actions[bot]]

Thank you for following the naming conventions! 🙏 Feel free to join our discord and post your PR link.

[vercel[bot]]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
ai	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Jun 14, 2024 8:24am

2 Ignored Deployments

Name	Status	Preview	Comments	Updated (UTC)
cal	⬜️ Ignored (Inspect)	Visit Preview		Jun 14, 2024 8:24am
calcom-web-canary	⬜️ Ignored (Inspect)	Visit Preview		Jun 14, 2024 8:24am

[graphite-app[bot]]

Graphite Automations

"Add consumer team as reviewer" took an action on this PR • (05/20/24)

1 reviewer was added to this PR based on Keith Williams's automation.

[github-actions[bot]]

📦 Next.js Bundle Analysis for @calcom/web

This analysis was generated by the Next.js Bundle Analysis action. 🤖

This PR introduced no changes to the JavaScript bundle! 🙌

[deploysentinel[bot]]

Current Playwright Test Results Summary

✅ 321 Passing - ⚠️ 16 Flaky

Run may still be in progress, this comment will be updated as current testing workflow or job completes...

(Last updated on 05/20/2024 12:31:26pm UTC)

Run Details

Running Workflow PR Update on Github Actions

Commit: b44d7020e9c7a5db2b8fe2853e0a3e85b7e69f7b

Started: 05/20/2024 12:29:25pm UTC

⚠️ Flakes

📄   apps/web/playwright/managedBooking/advancedOptions.e2e.ts • 1 Flake

Test Case Results

Test Case	Last 7 days Failures	Last 7 days Flakes
Check advanced options in a managed team event type Check advanced options in a managed team event type without offer seats Retry 1 • Initial Attempt	0.43% (1) 1 / 231 run failed over last 7 days	48.92% (113) 113 / 231 runs flaked over last 7 days

---

📄   packages/embeds/embed-core/playwright/tests/preview.e2e.ts • 1 Flake

Test Case Results

Test Case	Last 7 days Failures	Last 7 days Flakes
Preview Preview - embed-core should load if correct embedLibUrl is provided Retry 1 • Initial Attempt	0% (0) 0 / 224 runs failed over last 7 days	31.70% (71) 71 / 224 runs flaked over last 7 days

---

📄   packages/app-store/routing-forms/playwright/tests/basic.e2e.ts • 1 Flake

Test Case Results

Test Case	Last 7 days Failures	Last 7 days Flakes
Routing Forms Seeded Routing Form Test preview should return correct route Retry 1 • Initial Attempt	0.87% (2) 2 / 231 runs failed over last 7 days	31.60% (73) 73 / 231 runs flaked over last 7 days

---

📄   apps/web/playwright/event-types.e2e.ts • 2 Flakes

Top 1 Common Error Messages

null	2 Test Cases Affected

Test Case Results

Test Case	Last 7 days Failures	Last 7 days Flakes
Event Types tests -- future user Different Locations Tests Can add Link Meeting as location and book with it Retry 1 • Initial Attempt	0.42% (1) 1 / 240 run failed over last 7 days	13.75% (33) 33 / 240 runs flaked over last 7 days
Event Types tests -- legacy user enabling recurring event comes with default options Retry 1 • Initial Attempt	1.69% (4) 4 / 236 runs failed over last 7 days	9.32% (22) 22 / 236 runs flaked over last 7 days

---

📄   packages/embeds/embed-core/playwright/tests/action-based.e2e.ts • 9 Flakes

Top 1 Common Error Messages

null	9 Test Cases Affected

Test Case Results

Test Case	Last 7 days Failures	Last 7 days Flakes
Popup Tests should open embed iframe on click - Configured with light theme Retry 1 • Initial Attempt	2.14% (5) 5 / 234 runs failed over last 7 days	60.68% (142) 142 / 234 runs flaked over last 7 days
Popup Tests should be able to reschedule Retry 1 • Initial Attempt	-163.22% (-142) -142 / 87 runs failed over last 7 days	163.22% (142) 142 / 87 runs flaked over last 7 days
Popup Tests should open Routing Forms embed on click Retry 1 • Initial Attempt	-160.92% (-140) -140 / 87 runs failed over last 7 days	160.92% (140) 140 / 87 runs flaked over last 7 days
Popup Tests Floating Button Popup Pro User - Configured in App with default setting of system theme should open embed iframe according to system theme when configured with 'auto' theme using Embed API Retry 1 • Initial Attempt	-158.14% (-136) -136 / 86 runs failed over last 7 days	158.14% (136) 136 / 86 runs flaked over last 7 days
Popup Tests Floating Button Popup Pro User - Configured in App with default setting of system theme should open embed iframe according to system theme when no theme is configured through Embed API Retry 1 • Initial Attempt	-156.32% (-136) -136 / 87 runs failed over last 7 days	157.47% (137) 137 / 87 runs flaked over last 7 days
Popup Tests Floating Button Popup Pro User - Configured in App with default setting of system theme should open embed iframe(Event Booking Page) with dark theme when configured with dark theme using Embed API Retry 1 • Initial Attempt	-160% (-136) -136 / 85 runs failed over last 7 days	160% (136) 136 / 85 runs flaked over last 7 days
Popup Tests Floating Button Popup Pro User - Configured in App with default setting of system theme should open embed iframe(Booker Profile Page) with dark theme when configured with dark theme using Embed API Retry 1 • Initial Attempt	-158.14% (-136) -136 / 86 runs failed over last 7 days	158.14% (136) 136 / 86 runs flaked over last 7 days
Popup Tests prendered embed should be loaded and apply the config given to it Retry 1 • Initial Attempt	-160% (-136) -136 / 85 runs failed over last 7 days	160% (136) 136 / 85 runs flaked over last 7 days
Popup Tests should open on clicking child element Retry 1 • Initial Attempt	-159.52% (-134) -134 / 84 runs failed over last 7 days	159.52% (134) 134 / 84 runs flaked over last 7 days

---

📄   packages/embeds/embed-core/playwright/tests/namespacing.e2e.ts • 1 Flake

Test Case Results

Test Case	Last 7 days Failures	Last 7 days Flakes
Namespacing Inline Embed Add inline embed using a namespace without reload Retry 1 • Initial Attempt	0.43% (1) 1 / 231 run failed over last 7 days	62.77% (145) 145 / 231 runs flaked over last 7 days

---

📄   apps/web/playwright/organization/booking.e2e.ts • 1 Flake

Test Case Results

Test Case	Last 7 days Failures	Last 7 days Flakes
Bookings Team Event Can create a booking for Round Robin EventType Retry 2 • Retry 1 • Initial Attempt	7.53% (18) 18 / 239 runs failed over last 7 days	27.20% (65) 65 / 239 runs flaked over last 7 days

---

View Detailed Build Results

---

[github-actions[bot]]

This PR is being marked as stale due to inactivity.

[PeerRich]

is this being sent to users or to us admins?

because it is way too technical for users. and i wonder if we can track this as admins outside of our code base? 🤔

also, why high priority?

[Udit-takkar]

is this being sent to users or to us admins? because it is way too technical for users. and i wonder if we can track this as admins outside of our code base? 🤔 also, why high priority?

When a recording is ready to be downloaded (after it has been stopped by user) we have a daily webhook (https://docs.daily.co/reference/rest-api/webhooks endpoint /api/recorded-daily-video ) that receives the payload with recording id, room name etc to be sent to email of attendees and trigger RECORDING_READY event.

Now if this endpoint return a non success status code like 40x or 500 three times consecutively then the daily webhooks goes to FAILED state and doesn't trigger /api/recorded-daily-video enpoint unless we restart it again.

[hariombalhara]

is this being sent to users or to us admins? because it is way too technical for users. and i wonder if we can track this as admins outside of our code base? 🤔 also, why high priority?

When a recording is ready to be downloaded (after it has been stopped by user) we have a daily webhook (docs.daily.co/reference/rest-api/webhooks endpoint /api/recorded-daily-video ) that receives the payload with recording id, room name etc to be sent to email of attendees and trigger RECORDING_READY event.

Now if this endpoint return a non success status code like 40x or 500 three times consecutively then the daily webhooks goes to FAILED state and doesn't trigger /api/recorded-daily-video enpoint unless we restart it again.

So, it affects all the users using the Daily location after that?

Also, not sure if it is discussed but in addition to this email we should restart it automatically(if possible).

[Udit-takkar]

I have discussed an alternate solution with daily team. I'll update the retry type to "exponential"