Write snapshot artifacts to disk

[mac-gallagher]

Hi all! 👋

Justification

Right now, there is no way to write all of the reference, failure, and difference files found as XCTAttachments to disk. This makes it difficult to fix snapshot tests that fail on CI but pass locally.

The SNAPSHOT_ARTIFACTS environment variable gets us a third of the way there by writing the failure snapshot to disk, but not the other two. This PR allows for all three to be written to disk via the SNAPSHOT_ARTIFACTS variable.

This enhancement was discussed in https://github.com/pointfreeco/swift-snapshot-testing/issues/176.

Proposed Solution

This PR introduces the SnapshotArtifact struct which is used to create the XCTAttachments as well as optionally write the artifacts to disk. The enum ArtifactType identifies the artifact as one of the following types: reference, failure, and difference.

The proposed file structure looks something like this:

- SNAPSHOT_ARTIFACTS
  - MyClassTests
    - testFunctionImageFormat.1
      - testFunctionImageFormat.1_difference.png
      - testFunctionImageFormat.1_failure.png
      - testFunctionImageFormat.1_reference.png
    - testFunctionStringFormat.1
      - testFunctionStringFormat.1_difference.txt

Open to any and all suggestions! Thanks!

[stephencelis]

Hi @mac-gallagher! Thanks so much for the pull request! This looks great and definitely addresses a problem with how things are currently handled. @mbrandonw and I will take a closer look at this soon and deliver any feedback we may have!

[mluisbrown]

@mac-gallagher just an FYI that you can use https://github.com/TitouanVanBelle/XCTestHTMLReport to get an HTML version of the Xcode test log, which will include any attachments. The created HTML report can then be saved in your CI artefacts, to allow browsing of snapshot failures, with diffs, for when they fail on CI.

It's a really nice workaround, which we have been using.

[mac-gallagher]

@mac-gallagher just an FYI that you can use https://github.com/TitouanVanBelle/XCTestHTMLReport to get an HTML version of the Xcode test log, which will include any attachments. The created HTML report can then be saved in your CI artefacts, to allow browsing of snapshot failures, with diffs, for when they fail on CI.

It's a really nice workaround, which we have been using.

@mluisbrown thanks for the suggestion! I'll definitely give it a try 👍

[mac-gallagher]

Hey @stephencelis, any update on this PR?

[mac-gallagher]

Hey @stephencelis! Thanks for the suggestions - I just pushed a commit adding backwards compatibility. A few things:

• I added a new computed var artifactDiff instead of replacing the existing diff variable. I also added an overloaded initializer.
• All of the internal snapshot strategies now use the artifactDiff variable, so it should be easy to soft deprecate the old method if/when the time comes!
• It was a little tricky to handle the backwards compatibility in AssertSnapshot and AssertInlineSnapshot in a clean way. The respective verify functions will check for the artifactDiff variable first, and then the diff variable when the former is nil. The initializers are set up in a way where exactly one of them will be nil in the case of a snapshot failure.

Again, I'm open to any suggestions! Thanks!

[mac-gallagher]

Hey guys, any plans to merge this in? Thanks! @mbrandonw @stephencelis

[MaxDesiatov]

@mbrandonw @stephencelis this would be a super-useful feature to be merged to master, any updates on this?

[megs-seek]

Agreed, would be great to get this merged in!

[ixrevo]

Hi everyone, with Xcode 11 you can get all snapshot artifacts from CI pretty easy. Check out this issue for more info https://github.com/pointfreeco/swift-snapshot-testing/issues/291

[justinguo]

This would definitely be a nice addition for the framework and cannot wait for using it!

[stephencelis]

@mac-gallagher Thanks for making those changes. This fell off our radar but we're going to take a look at your changes tomorrow. Hope to get things merged quickly as well!

[carsten-wenderdel]

Hi @stephencelis I would also like to see this merged. Using the whole xcresult file as artifact (proposed above in #291) has some issues, for example the file size is bigger than what our CI system allows.

[TomHumphrey150]

@stephencelis any updates on this? 🙏

I'm working on something similar to write the snapshots to disk in the location that XCPretty uses for screenshots in its HTML report. Would only be a minor tweak if this merged 😬

[andrewmarmion]

@stephencelis Is there any update on this? It would be such a great addition.

[teameh]

This would be a great addition! @stephencelis anything we can do to help get this merged?

[Jake-Prickett]

Echoing many of the voices on here, this would be a great addition that we would really benefit from - anything I can do to help move this forward @stephencelis?

I did come across this comment in a separate issue which describes archiving the xcresult file on CI. We went down this route initially, but for large projects the xcresult file size gets pretty big (~19MB) which isn't ideal.

This approach would minimize the amount of CI resources consumed and provide clear insights into failures that occur on CI.

Are there concerns with the implementation or approach taken? Could this be something enabled/disabled via an environment variable?

[mac-gallagher]

Sorry for all the activity above - I struggled to rebase my changes because I didn't realize the base branch changed from master to main! Anyways, resolved all the merge conflicts.

However, it doesn't look like that environment variable SNAPSHOT_ARTIFACTS is working right now... I'm getting the error You can’t save the file ___ because the volume ___ is read only. on the main branch right now. This will need to be fixed before these changes go in. It looks like there was some discussion about this in #193

[nunogoncalves]

Hello! @stephencelis Any news on this one? We would really like to have this available too. :)
Thank you very much for this work and the lib!! :)