sentry

sentry copied to clipboard

Introduce the ServerModeTest decorator in testutils, to dynamically add test methods that run in a server component mode.

Add a script for automatically adding ServerModeTest decorators to existing unit tests.

I would prefer if this was available as context manager or something similar that integrates with bare function-based tests as well.

So would I. I'd also like to be able to put the decorator on individual test methods, instead of putting it at the class level and messing around with names and predicates. Let me share some of the problems I've encountered with that so far.

My first pass was something like this (pardon the pseudocode):

def server_mode_test(silo_mode):
    def wrapper(wrapped_method):
        with override_settings(monolith_mode):
            wrapped_method()
        with override_settings(silo_mode):
            wrapped_method()
    return wrapper

Some test methods still work as intended that way, but many don't because the first invocation of wrapped_method changes the preconditions of the second. If the test is supposed to create an object, it's already created when we enter the second run. I ran into a ton of SQL integrity violation errors, plus it could silently cause false negatives.

I experimented with other ways around this, such as manually calling the test class's setUp and tearDown methods at the appropriate time, which didn't work (and, even if it had, would still require the decorator to go on the class level, not the method). Ultimately I settled on splicing a new method into the class, which ensures the testing harness treats it the way we want, and as an added benefit exposes it to a human user as a distinct case with an informative name.

However, considering that bare function-based tests don't generally rely on a class to do setup and teardown in the first place, maybe the "run it twice" approach would suffice? We could just have a second, simpler decorator which is used only for those.

@untitaker Since you specifically mentioned a context manager, I'm not sure how to make that work, as opposed to a decorator. As I understand it, a limitation of Python context managers is that they can only insert code at the "enter" and "exit" points, but can't control the flow of the enclosed code. That is, in

with foo():
    bar()

there's no way to write foo such that bar is run twice, or conditionally. (As opposed to similar constructs in, e.g., Ruby, where that would be possible.) Let me know if you have any implementation suggestions; otherwise it will probably be a decorator.

Let me know if you have any implementation suggestions; otherwise it will probably be a decorator.

If you need to run multiple tests twice, perhaps check out fixture parametrization in pytest. You could provide a pytest fixture/marker for tests that aren't using a unittest-based baseclass, and the decorator for unittest-based tests.

Unfortunately pytest parametrization won't work for unittest-based tests... i don't even know y.

I also solved a similar problem here: https://github.com/getsentry/sentry/pull/36397

There I just went with a marker on every test that needs to be rerun twice, and an additional CI job.

Unfortunately pytest parametrization won't work for unittest-based tests... i don't even know y.

it's because the unittest support via pytest is testgen which is incompatible with the parametrize testgen