posthog.com icon indicating copy to clipboard operation
posthog.com copied to clipboard

Create incident handbook entry

Open timgl opened this issue 3 years ago • 3 comments

Changes

Our response to an incident today was okay, but there were a few areas for improvement:

  • There were a couple of separate threads with different people solving the same problem
  • we didn't communicate well to customers
  • We didn't take any good notes until the very end
  • We're not sure whether it was actually resolved

I propose we use incident.io to respond to incidents. This will

  • Automatically create a channel for new incidents with a zoom link so everyone knows where to look. It'll also post that an incident is ongoing in #incidents, so the rest of the company knows what's going on
  • It'll provide us with a better log of what happened
  • It'll help us with incident analysis afterwards.

In the future:

  • we'll be able to automatically update our status page (might need to move back to statuspage.io)
  • we'll be able to have a feed of incidents in posthoguser slack (they don't support this out of the box, but perhaps zapier)
  • We might even be able to have a feed directly in our product (api?)

Checklist

timgl avatar Aug 23 '22 21:08 timgl

Deploying with  Cloudflare Pages  Cloudflare Pages

Latest commit: 4dfb3e1
Status:🚫  Build failed.

View logs

@timgl still looking to merge this PR?

charlescook-ph avatar Sep 15 '22 08:09 charlescook-ph

@fuziontech suggested we just use pagerduty instead of incident.io, but haven't had a change to look at that

timgl avatar Sep 15 '22 08:09 timgl

Deploy Preview for posthog failed.

Name Link
Latest commit 72628d405cfbbc48f48474b78a75232697692e8f
Latest deploy log https://app.netlify.com/sites/posthog/deploys/635a61f4ede5af00088bb12a

netlify[bot] avatar Oct 27 '22 10:10 netlify[bot]

👋 @guidoiaquinti @yakkomajuri

I've made some adjustments as ik Tim doesn't have the bandwidth to push this through atm. As you both reviewed this before, mind giving it a once over? I'd like to get this merged asap if possible.

Thanks!

ellie avatar Oct 27 '22 10:10 ellie