[Feature] Store snapshots from multiple simulators side-by-side

[tikitu]

We're just starting to use this wonderful library, and already we're scope-creeping it to solve problems it was never intended to...

Namely: it looks like it will do a beautiful job of cross-OS-version/cross-device regression testing, except for the fact that the snapshot filename doesn't depend on any of the simulator details. I see #128 is related, where you suggest the workaround of naming the file by hand, but the problem is more general than just display scale: in particular we're interested in regression testing for OS behaviour changes over different versions, and the by-hand approach gets out of control pretty quickly.

I'm wondering if we could do something more systematic, by encoding some relevant details of the simulator setup in the filename: definitely OS version and device display scale, there might be more. I guess it should be opt-in, or use the current naming scheme as a fallback, or something similar.

I'd be happy to take a crack at a PR if y'all would be open to this idea, what do you think?

[stephencelis]

Hey @tikitu!

Yeah this is coming up often enough that we definitely think it needs to be baked in. It'll be a breaking change so let's add a new global configuration to opt into it, and then we can enabled it by default in SnapshotTesting 2.0.

We'd want to bake in:

• OS
• Color gamut
• Device scale
• (Anything else?)

Some examples:

// unnamed first (index 1) snapshot of "testMyView" test, iOS 12.1, SRGB gamut
[email protected] 

// "customName"-named snapshot of "testMyView" test, iOS 12.0, P3 gamut
[email protected]

This means the current delimiter is changing from . to -, but it makes the OS version read a bit more nicely.

One other requirement: we should only record automatically if no screen shot exists for this test. If you run on a simulator, record a snapshot, and then run on an incompatible simulator, it should not record another snapshot, it should fail with an error about an existing snapshot for a different simulator type. In order to recover the ability to record for multiple simulators, one could either:

1. Explicitly set record = true
2. We could introduce another global, SnapshotTesting.supportedPlatforms, that turns auto-record on for some explicit platforms.

Whatcha think?

[tikitu]

Sounds good, thanks for the feedback! For recording to multiple simulators I'll have to try out some variations, I don't yet have a good feel for the kinds of errors one can make (how often am I going to be surprised by record = true on the wrong simulator if that's the API?).

My team will be using this once we get it figured out, so I can contribute in work hours: I'll make a first attempt in the coming week.

[tikitu]

Do we have any non-image use cases for this machinery? If not, I'd like to somehow avoid forcing e.g. text-snapshots to be re-recorded multiple times. A first suggestion for how would be to get Snapshotting<Value, Format> to answer the question "do I care about image properties?": if yes we do the filename dance, if no we keep it simple. (A more elaborate version puts Snapshotting in charge of the filename additions too, but that complicates SnapshotTesting.supportedPlatforms or equivalent, so I'd like to see a use case before going that way, unless y'all feel strongly that it's preferable.)

[tikitu]

~Hoho, and I guess I also need to ask with which device/simulator you generated the tests for your library, because indeed a few of the tests fail for what I suspect to be exactly this reason!~ "iOS Simulator,name=iPhone XR,OS=12.1"

[Kaspik]

Hey here! Any update on this? :)

[tikitu]

Whoops, I should come back to this but I'm struggling to find time to work on it. It's still something my team would like, though, so I will eventually be able to pick it up again.

[Zentaur0]

Hi everyone! Any updates on this issue/feature?