Dedupe triggers on dropped events

[Prillan]

The dedupe integration drops events that were never sent before.

1. An error is triggered.
2. before_send drops the event.
3. Another event is triggered based on the same error somewhere else.
4. dedupe drops the event.
5. (before_send would have accepted the event)

Ideally, dedupe would run after before_send or there should be a possibility to hook into the event processing earlier. A third option would be to bring back the option to ignore specific loggers from raven.

Example code:

from flask import Flask
import logging
import sentry_sdk
from sentry_sdk.integrations.flask import FlaskIntegration


# Drop logging based messages from inside a third-party library
def before_send_hook(event, hint):
    if event['level'] == 'error' and 'logger' in event:
        if event['logger'] == 'internal-logger':
            return None

    return event


app = Flask(__name__)
sentry_sdk.init(
    # RANDOM DSN
    'https://[email protected]/1234567',
    integrations=[FlaskIntegration()],
    before_send=before_send_hook,
    debug=True,
)


# This function is hidden inside a third-party library
def complex_internal_function():
    class InternalError(Exception):
        def __init__(self, original):
            super().__init__()
            self.original = original

    try:
        raise ValueError('Oh noes!')
    except Exception as ex:
        logger = logging.getLogger('internal-logger')
        e = InternalError(ex)
        logger.error('ERROR!', exc_info=e)
        raise e


@app.route('/')
def index():
    try:
        complex_internal_function()
    except Exception as ex:
        sentry_sdk.capture_exception(ex)
        return 'error', 500

    return 'ok'

if __name__ == '__main__':
    app.run()

Output from the flask server:

ERROR!
Traceback (most recent call last):
  File "example.py", line 32, in complex_internal_function
    raise ValueError('Oh noes!')
ValueError: Oh noes!
 [sentry] INFO: before send dropped event ({'level': 'error', 'exception': ...})
 [sentry] INFO: event processor (<function DedupeIntegration.setup_once.<locals>.processor at 0x7f8a88b10ea0>) dropped event ({'level': 'error', 'exception': ...})
127.0.0.1 - - [21/May/2019 16:09:01] "GET / HTTP/1.1" 500 -

[untitaker]

To solve your intermediate problem, you can use ignore_logger: https://docs.sentry.io/platforms/python/logging/#ignoring-a-logger

Not sure what's best to do about the more general bug yet. Thanks for raising it though!

[github-actions[bot]]

This issue has gone three weeks without activity. In another week, I will close it.

But! If you comment or otherwise update it, I will reset the clock, and if you label it Status: Backlog or Status: In Progress, I will leave it alone ... forever!

---

"A weed is but an unloved flower." ― Ella Wheeler Wilcox 🥀

[antonpirker]

This is probably a bug that is worth looking into. Putting it in the internal backlog at low priority

[sentrivana]

Note: might be easier to fix once we have lifecycle hooks.

[thianesh]

Can anybody provide a solution?

Same here, Sentry is dropping my exceptions.

Exception: Test Exception [sentry] INFO: event processor (<function DedupeIntegration.setup_once..processor at 0x7f6470145d00>) dropped event INFO:sentry_sdk.errors:event processor (<function DedupeIntegration.setup_once..processor at 0x7f6470145d00>) dropped event INFO: connection closed

[sentrivana]

Hey @thianesh, one thing you can do is disable the DedupeIntegration (see docs), though that can of course lead to, well, duplicates in other parts of the system.

Could you give us a bit more info on where you're seeing this? Do you also have a before_send like the OP describes in the first comment?

[sentrivana]

So to sum up, the problem is that event processors (which is what the dedupe integration is using to detect duplicates) run before the before_send callback, meaning dedupe runs before the event is dropped in before_send and it has no knowledge of the fact.

The way dedupe works now is that it holds the last exception in a ContextVar called last-seen and then when a new exception comes along, compares the two. If they're the same, the new exception is dropped. Otherwise the new exception is set as last-seen.

I see the following options here:

1. We rework the DedupeIntegration to not use an event processor. This would probably mean deleting the integration altogether, moving last-seen elsewhere, and adding some code to after before_send is called.
2. We augment the DedupeIntegration so that we can tell it to remember a previous exception instead in case before_send dropped the event. For this it'd need to keep more exceptions in memory.
3. If before_send drops an event, we simply set dedupe's last-seen to None so that we guarantee the next exception is sent.

Both 1. and 2. are quite invasive. I'm thinking 3., while not perfect, would be a good enough improvement here.