swift-snapshot-testing

swift-snapshot-testing copied to clipboard

Motivation

Xcode 13 introduced new parameters to xcodebuild that allow to run tests repeatedly (wwdc talk). Unfortunately SnapshotTesting doesn't handle such situation in a correct way.

Proposal

We can use XCTestObservation in order to clean counter map between different test cases.

Resolved issues

Closes #577

I have two questions regarding this

1. Would we still have SnapshotTesting increment the counter for multiple snapshots inside one test case? e.g. snapshot assertion from different lines but the same test case
2. I'm using Quick/Nimble for this and if the test originates from the same line, i.e. a toEventually expectation, would it use the same file name for each execution?

I have two questions regarding this

1. Would we still have SnapshotTesting increment the counter for multiple snapshots inside one test case? e.g. snapshot assertion from different lines but the same test case

Yes. TestObserver is called after test case is executed so the feature of counting multiple assertSnapshot calls works as expected.

2. I'm using Quick/Nimble for this and if the test originates from the same line, i.e. a toEventually expectation, would it use the same file name for each execution?

I've never used Quick or Nimble. Would you be so kind to check with my forked version if it works with your project? I've already checked with our where we have planty of different scenarios but definitely would be cool to have confirmation also from someone else.

@krzysztofpawski I tried it out and it still produces many snapshots. The way Quick works is that it executes the same condition with a polling until it passes.

Maybe it will make sense to make it unique based on the test line. The only thing that would break it is if someone adds two assertions on the same line

@tahirmt could you check it one more time? I found one more case when it was producing too many reference files in our repo. If not maybe then you could prepare a very simple project reproducing the issue mentioned by you and attach it here?

@krzysztofpawski I created a demo repo to test this using your fork https://github.com/tahirmt/test-toeventually-behavior-snapshots.

@tahirmt I'm afraid it can't be fixed in swift-snapshot-testing library without some bigger refactor. But the solution I see for you is that you can use named argument (pass some non empty string) to omit this counting mechanism. It's not perfect but it should do the work for your matcher.

@tahirmt I'm afraid it can't be fixed in swift-snapshot-testing library without some bigger refactor. But the solution I see for you is that you can use named argument (pass some non empty string) to omit this counting mechanism. It's not perfect but it should do the work for your matcher.

That's what I was thinking too. I was thinking of many alternatives but I can't think of a better solution for now. Thanks for looking into it though.

@stephencelis any possibility to review this PR?