swift-testing icon indicating copy to clipboard operation
swift-testing copied to clipboard

Provide an API for custom expectations with diagnostics

Open allevato opened this issue 2 years ago • 1 comments

Description

Spawned from the discussion in https://forums.swift.org/t/brainstorming-customizing-matchers/67456/11:

When a custom comparison is needed, it was recommended that users could just write whatever comparison they need as a Bool-returning function and pass that into #expect. That's fine from the point of view of keeping the #expect API simple, but expectations will often want to provide better diagnostics than what can be extracted from the arguments.

For example, pretend we didn't have isSuperset(of:) and we wrote our own terrible predicate:

extension Collection where Element: Hashable {
  func isThisASuperset(of smaller: some Collection<Element>) -> Bool {
    let diff = Set(smaller).subtracting(Set(self))
    return diff.isEmpty
  }
}

@Test func foo() {
  let x = [1, 3]
  let y = [1, 2, 4, 5]
  #expect(x.isThisASuperset(of: y))
}

When running this test, the runner produces the following diagnostic:

Expectation failed: (x → [1, 3]).isItASuperset(of: y → [1, 2, 4, 5])

which is helpful! But what I'd really like is something like:

Expectation failed: argument contained [2, 4, 5] which were not in the receiver -- (x → [1, 3]).isItASuperset(of: y → [1, 2, 4, 5])

One way to achieve this would be to create an #expect overload that takes, say, an ExpectationResult instead. That could look like the following:

enum ExpectationResult {
  case success
  case failure(reason: String)
}

and then the predicate becomes:

extension Collection where Element: Hashable {
  func isThisASuperset(of smaller: some Collection<Element>) -> ExpectationResult {
    let diff = Set(smaller).subtracting(Set(self))
    return diff.isEmpty
      ? .success
      : .failure(reason: "argument contained \(diff) which were not in the receiver")
  }
}

This would be a bare minimum API for these kinds of diagnostics, and simple enough for most purposes. It wouldn't be advanced enough to factor in the named of the arguments though; if we wanted the output to be something like this, we'd need something more complex:

Expectation failed: 'y' contained [2, 4, 5] which were not in 'x' -- (x → [1, 3]).isItASuperset(of: y → [1, 2, 4, 5])

Expected behavior

No response

Actual behavior

No response

Steps to reproduce

No response

swift-testing version/commit hash

0.0.0-initial

Swift & OS version (output of swift --version && uname -a)

No response

allevato avatar Sep 22 '23 22:09 allevato

Tracked internally with rdar://106832903. Using this OSS issue to track going forward.

grynspan avatar Sep 25 '23 16:09 grynspan