sentry

sentry copied to clipboard

🚨 This PR is not meant to be merged. It serves as a place for discussion about the architecture and as a reference for the motives behind the changes. 🚨

Context: Sentry used to only have the concept of events, and those events were errors associated with an issue (or group). When we decided to add transactions, we then had two kinds of events, errors and transactions. In many places of the code, events used to be synonymous errors. So eventstream was originally built with just errors in mind.

What is the eventstream responsible for?

• Ingestion: errors and transactions are processed and then need to be sent somewhere so they can be saved and post processed (if needed)
• Mutations: merges, unmerges, reprocessing, etc. If the event type can be mutated in some way, the eventstream provides an api for doing so.

Snuba vs Kafka eventstream The default eventstream is "sentry.eventstream.snuba.SnubaEventStream".

• Snuba: This is used in local dev environments. If you don't need Kafka running and don't need any post processing for the work you are doing, we can simple send the error or transaction to Snuba and save the data to ClickHouse.
• Kafka: What we use in production. The eventstream will produce the errors/transactions to a Kafka topic (currently transactions and errors share the events topic). Snuba consumer consume from the topic, and save the data to ClickHouse. Post process forwarder consumers also consume from the topic but wait on the commit log for the topic before actually doing the post processing since we need to know that the data is indeed saved on ClickHouse.

Current: We've split out a lot of pipeline already (new subscriptions infra is not included in this and that is also split out into separate errors and transactions flows.

Ideal: @lynnagara is already working on splitting the events topic so that it just has errors, and then making a new transactions topic. With separate topics, it makes sense to split the eventstream as well.