relay icon indicating copy to clipboard operation
relay copied to clipboard

feat(relay): snapshot testing for python integration tests

Open Litarnus opened this issue 6 months ago • 3 comments

Adds snapshot testing in python integration tests for relay.

Dynamic values can be redacted to prevent flakyness (or enable snapshot testing in general).

For example, event_id get's automatically replaced by <event_id> in the snapshot.

The list is configurable through direct parameters, e.g. relay_snapshot("json", additional_keys={"foo"}).

Keys need to match the full path, so timestamp will only match the root level timestamp key but not the one in spans (see snapshot in PR).

Litarnus avatar Jun 30 '25 07:06 Litarnus

Just some high level questions:

  1. Can we have the snapshots inline, I prefer that much over having them in separate files
  2. Instead of ignoring certain keys, can we assert that they match some bound (e.g. the assertions we have with time_within (or other generic functions that can produce some output)

I think ideally we have both of these capabilities before we start replacing uses. This may still be useful for some very large assertions though.

Dav1dde avatar Jul 02 '25 18:07 Dav1dde

Can we have the snapshots inline, I prefer that much over having them in separate files

It should be relatively easy to add, regular python asserts give a good diff of what the differences between two dicts are.

Instead of ignoring certain keys, can we assert that they match some bound (e.g. the assertions we have with time_within (or other generic functions that can produce some output)

For inline snapshots it shouldn't be too hard to add it since you could just pass the comparison functions in the local snapshot but for file snapshots it would be more tricky since it would need to have a way to configure them upfront based on key name or something like that.

Litarnus avatar Jul 03 '25 09:07 Litarnus

After giving it more thought I'm not sure what the benefit would be if this could also do inline snapshots. assert is pretty much an inline snapshot.

While I agree that inline snapshots are nicer and easier to review, I think this could be useful for PII related snapshot tests where we can not only test if fields are properly scrubbed but also make sure that _meta is correctly populated without having to write _meta ourselves or write helper functions that generate the meta. For those tests it would also not be relevant if the timestamp redacted or within a timeframe since that would not be the scope of said tests.

Litarnus avatar Jul 04 '25 09:07 Litarnus