MockSequence and VerifyAll

[henriquemotaesteves]

When using a "MockSequence" the behavior of the "VerifyAll" method is modified. Is this correct?

public interface IFoo
{
    void Do();
}

/*
 * Throws MockVerificationException
 */
var foo = new Mock<IFoo>(MockBehavior.Strict);

foo.Setup(f => f.Do());
foo.VerifyAll();

/*
 * Does not throw MockVerificationException
 */
var foo = new Mock<IFoo>(MockBehavior.Strict);
var seq = new MockSequence();

foo.InSequence(seq).Setup(f => f.Do());
foo.VerifyAll();

[kzu]

Good point. What should be the expected behavior?

/kzu from mobile On Dec 18, 2013 7:33 PM, "Henrique Esteves" [email protected] wrote:

When using a "MockSequence" the behavior of the "VerifyAll" method is modified. Is this correct?

public interface IFoo{ void Do();} /* * Throws MockVerificationException /var foo = new Mock<IFoo>(MockBehavior.Strict); foo.Setup(f => f.Do());foo.VerifyAll(); / * Does not throw MockVerificationException */var foo = new Mock<IFoo>(MockBehavior.Strict);var seq = new MockSequence(); foo.InSequence(seq).Setup(f => f.Do());foo.VerifyAll();

— Reply to this email directly or view it on GitHubhttps://github.com/Moq/moq4/issues/75 .

[kzu]

But isn't that what verifying the sequence achieves?

/kzu from mobile On Dec 19, 2013 10:20 AM, "Henrique Esteves" [email protected] wrote:

In my opinion, regardless if you are using a sequence or not, the behavior of a strict mock must be the same. If "VerifyAll" is called it should check if all methods were called.

When using a sequence, the "VerifyAll" method should be more restrictive, not less, as in the example above. Besides checking all the methods were called, it should check whether they were called in the correct order.

public interface IFoo{ void DoOne(); void DoTwo();} /* * Does not throw MockVerificationException /var foo = new Mock<IFoo>(MockBehavior.Strict); foo.Setup(f => f.DoOne());foo.Setup(f => f.DoTwo()); foo.DoOne();foo.DoTwo(); foo.VerifyAll(); / * Does not throw MockVerificationException /var foo = new Mock<IFoo>(MockBehavior.Strict); foo.Setup(f => f.DoOne());foo.Setup(f => f.DoTwo()); foo.DoTwo();foo.DoOne(); foo.VerifyAll(); / * Does not throw MockVerificationException /var foo = new Mock<IFoo>(MockBehavior.Strict);var seq = new MockSequence(); foo.InSequence(seq).Setup(f => f.DoOne());foo.InSequence(seq).Setup(f => f.DoTwo()); foo.DoOne();foo.DoTwo(); foo.VerifyAll(); / * Throws MockVerificationException */var foo = new Mock<IFoo>(MockBehavior.Strict);var seq = new MockSequence(); foo.InSequence(seq).Setup(f => f.DoOne());foo.InSequence(seq).Setup(f => f.DoTwo()); foo.DoTwo();foo.DoOne(); foo.VerifyAll();

— Reply to this email directly or view it on GitHubhttps://github.com/Moq/moq4/issues/75#issuecomment-30927589 .

[henriquemotaesteves]

I'm sorry, I accidentally removed the previous comment.

I had written something like this.

In my opinion, regardless if you are using a sequence or not, the behavior of a strict mock must be the same. If "VerifyAll" is called it should check if all methods were called.

When using a sequence, the "VerifyAll" method should be more restrictive, not less, as in the example above. In addition to check if all the methods were called, it should check whether they were called in the correct order.

public interface IFoo
{
    void DoOne();
    void DoTwo();
}

/*
 * Does not throw MockVerificationException
 */
var foo = new Mock<IFoo>(MockBehavior.Strict);

foo.Setup(f => f.DoOne());
foo.Setup(f => f.DoTwo());

var obj = foo.Object;

obj.DoOne();
obj.DoTwo();

foo.VerifyAll();

/*
 * Does not throw MockVerificationException
 */
var foo = new Mock<IFoo>(MockBehavior.Strict);

foo.Setup(f => f.DoOne());
foo.Setup(f => f.DoTwo());

var obj = foo.Object;

obj.DoOne();
obj.DoTwo();

foo.VerifyAll();

/*
 * Does not throw MockVerificationException
 */
var foo = new Mock<IFoo>(MockBehavior.Strict);
var seq = new MockSequence();

foo.InSequence(seq).Setup(f => f.DoOne());
foo.InSequence(seq).Setup(f => f.DoTwo());

var obj = foo.Object;

obj.DoOne();
obj.DoTwo();

foo.VerifyAll();

/*
 * Throws MockVerificationException
 */
var foo = new Mock<IFoo>(MockBehavior.Strict);
var seq = new MockSequence();

foo.InSequence(seq).Setup(f => f.DoOne());
foo.InSequence(seq).Setup(f => f.DoTwo());

var obj = foo.Object;

obj.DoTwo();
obj.DoOne();

foo.VerifyAll();

[henriquemotaesteves]

The previous examples were not very good because I've just described the behavior that is already correct.

The following example demonstrates how a "MockSequence" causes the "VerifyAll" method to be less restrictive, while in my opinion, it should be more restrictive.

/*
 * Does not throw MockVerificationException, but in my opinion it should
 */
var foo = new Mock<IFoo>(MockBehavior.Strict);
var seq = new MockSequence();

foo.InSequence(seq).Setup(f => f.DoOne());
foo.InSequence(seq).Setup(f => f.DoTwo());

var obj = foo.Object;

obj.DoOne();

foo.VerifyAll();

[kzu]

there's a key issue here that I think is missing: a sequence can be used across several mocks.

So any single mock in that sequence will not be able to verify that it was called before/after any of the calls in the sequence for the other mocks.

I believe that's a great powerful feature of sequences as implemented currently, and I can't see how we could add this verification behavior without introducing a very confusing behavior for cross-mocks verification.

I think verifying the sequence when you have sequenced calls is the right thing to do.

/kzu

Daniel Cazzulino

On Thu, Dec 19, 2013 at 1:40 PM, Henrique Esteves [email protected]:

The previous examples were not very good because I've just described the behavior that is already correct.

The following example demonstrates how a "MockSequence" causes the "VerifyAll" method to be less restrictive, while in my opinion, it should be more restrictive.

/*

• Does not throw MockVerificationException, but in my opinion it should */ var foo = new Mock<IFoo>(MockBehavior.Strict); var seq = new MockSequence();

foo.InSequence(seq).Setup(f => f.DoOne()); foo.InSequence(seq).Setup(f => f.DoTwo());

var obj = foo.Object;

obj.DoOne();

foo.VerifyAll();

— Reply to this email directly or view it on GitHubhttps://github.com/Moq/moq4/issues/75#issuecomment-30944013 .

[henriquemotaesteves]

Based on previous comments we have two issues:

1 - The behavior of the "VerifyAll" method of a strict mock in particular when a "MockSequence" is used. 2 - How to verify that a sequence of calls involving different mocks was respected.

Regarding the issue 1, I still think that a "MockSequence" should not interfere with the behavior of the "VerifyAll" method of a strict mock in particular. In the example below an exception should be thrown because a method ("DoTwo") was not invoked, regardless if the sequence was respected or not.

/*
 * Does not throw MockVerificationException, but in my opinion it should
 */
var foo = new Mock<IFoo>(MockBehavior.Strict);
var seq = new MockSequence();

foo.InSequence(seq).Setup(f => f.DoOne());
foo.InSequence(seq).Setup(f => f.DoTwo());

var obj = foo.Object;

obj.DoOne();

foo.VerifyAll();

Regarding the issue 2, I agree with you. It would be useful if a "MockSequence" had a method, like "VerifyAll", to verify that a sequence of calls involving different mocks was respected.

[kzu]

Yup, I agree on both issues.

[udlose]

So is there a way to verify on a MockSequence across all mocks that use that sequence? I'm using v4.5.30 and I don't see it. I was hoping that since this thread was from 3 yrs ago maybe it would have been added by now?

[cmeeren]

I'm also wondering if there is a way to verify called sequences across mocks.

[georgiosd]

Nothing four years later? :(

[georgiosd]

Found this: https://github.com/dwhelan/Moq-Sequences PR with updates here: https://github.com/georgiosd/Moq-Sequences

[Ray-B]

Also would love to see VerifyAll() for MockSequence.

Until then, another alternative approach is to use callbacks() for verifying sequence across multiple mocks: StackOverFlow Example

Then you don't need to drag YAP (yet another package) into your project.

[stakx]

I'd like to suggest the following changes:

• Let's make sure a MockSequence itself has Verify[All] methods.

• If mock partakes in a MockSequence, and it is the only mock in that sequence, then mock.Verify[All] simply delegates verification to the sequence.

• If mock partakes in a MockSequence, but that sequence spans across several different mocks, then mock.Verify[All] fails with an InvalidOperationException. (This is a breaking change, but would alert users to possibly incorrect usage.)

Any thoughts on this?

[udlose]

1. i like it

2. i'd argue that an exception should be thown if only one Mock is plced into a MockSequence

3. i like it. Communication to users about API misuse is becoming increasingly important!

[stakx]

@udlose:

2. i'd argue that an exception should be thown if only one Mock is [placed] into a MockSequence

I guess for consistency's sake with my third bullet point? My reasoning for suggesting that this would simply delegate was to have one less breaking change, but I think you're right in principle. I'd happy with either behaviour.

[stakx]

Closing this dormant issue, but marking it as "unresolved" so it can be easily found again. Please see #642 for details. If you'd like to pick this up and work on it, please post here briefly and we'll see what we can do!

[TinkerPhu]

Being a newbie in this blog I might not see the complete picture. But I think testing with sequences is essential. The moq functionality without sequence was really just a playground for testing but if you want to test protocols and issues during communication you need to be able to use sequences.

@stakx: your proposal goes into the right direction but I think the mock.Verify() and the sequence.Verify() function should be independent. And there is no need for exceptions to educate the user.

1. mock.Verify() keeps its semantics unchanged and independent of involved sequences by testing whether the set-up calls have been called at least once. These are non vital calls for sequences.

2. sequence.Verify() shall verify all set-up calls done with InSequence(sequence) on this very sequence. (including order and whether all calls of the sequence have actually been executed)

3. no exceptions for sequence.Verify() if no call / mock is involved at all or if it is only one. I think there is no necessity to force more than one mock or any at all. The point is to verify an order of 0...N calls setup by 0..N mocks.

Inspired by this post ( https://stackoverflow.com/questions/32585355/trying-to-understand-mocksequence ) I continued its idea which I would prefer to have completed in Moq:

public class SequenceVerifyer
{
    public MockSequence Sequence { get; private set; } = new MockSequence();

    public Action NextCallback()
    {
        var callNo = setupCount++;
        return () => { AssertCallNo(callNo);};
    }
    public void VerifyAll()
    {
        Assert.AreEqual(setupCount, executionCount, $"less calls ({executionCount}) executed than previously set up ({setupCount}).");
    }

    private int executionCount = 0;
    private int setupCount = 0;

    private void AssertCallNo(int expectedCallNo)
    {
        Assert.AreEqual(executionCount, expectedCallNo, $"order of call is wrong. this call is marked with No ({expectedCallNo}), but ({executionCount}) was expected.");
        executionCount++;
    }
}

usage:

public interface IFoo
{
    bool foo(int n);
}

public interface IBar
{
    int bar(string a);
}

[Test]
public void SequenceVerifyer_with_full_sequence()
{
    var fooMock = new Mock<IFoo>(MockBehavior.Strict);
    var barMock = new Mock<IBar>(MockBehavior.Strict);
    var seq = new SequenceVerifyer();

    fooMock.Setup(f => f.foo(3)).Returns(false);
    barMock.Setup(b => b.bar("world")).Returns(4);

    fooMock.InSequence(seq.Sequence).Setup(f => f.foo(4)).Returns(true).Callback(seq.NextCallback());
    barMock.InSequence(seq.Sequence).Setup(b => b.bar("hello")).Returns(2).Callback(seq.NextCallback());
    fooMock.InSequence(seq.Sequence).Setup(f => f.foo(4)).Returns(false).Callback(seq.NextCallback());

    fooMock.Object.foo(3); //non sequence
    fooMock.Object.foo(4);
    barMock.Object.bar("hello");
    barMock.Object.bar("world"); //non sequence
    fooMock.Object.foo(4);

    fooMock.VerifyAll();
    barMock.VerifyAll();
    seq.VerifyAll();
}

[weitzhandler]

In my scenario, I'm using non-strict mocks. I just want to use the sequence in the verification stage, so that the calls to the various underlying services were made by the SUT in the correct order.

One way to achieve that is to create a disposable sequence, maybe such that all mocks are added to it, that collects data about underlying calls and underlying verifications, then compares the order of the calls to the order of verifications.

I'm willing to implement this feature.

[Fact]
public void Test1()
{
  // arrange
  var jobServiceMock = new Mock<IJobService>();
  var workServiceMock = new Mock<IWorkService>();

  using var sequence = Sequence.Create(jobServiceMock, workServiceMock); //maybe no need to provide mocks

  var sut = new SystemUnderTest(jobServiceMock.Object, workService.Object);

  // act
  sut.DoWorkAndJob();

  // assert             
  workServiceMock.Verify(work => work.DoWork(), Times.Once);
  jobServiceMock.Verify(job => job.DoJob(), Times.Once);

  sequence.Dispose(); //redundant - verifies recorded calls match their verification order
}

[weitzhandler]

Otherwise, VerifyInSequence instead of Verify, something like:

[Fact]
public void Test1()
{
  // arrange
  var jobServiceMock = new Mock<IJobService>();
  var workServiceMock = new Mock<IWorkService>();

  using var sequence = Sequence.Create(jobServiceMock, workServiceMock); //maybe no need to provide mocks

  var sut = new SystemUnderTest(jobServiceMock.Object, workService.Object);

  // act
  sequence.Execute(() => sut.DoWorkAndJob());

  // assert             
  workServiceMock.VerifyInSequence(sequence, work => work.DoWork(), Times.Once);
  jobServiceMock.VerifyInSequence(sequence, job => job.DoJob(), Times.Once);

  sequence.Dispose(); //redundant - verifies recorded calls match their verification order
}

[stakx]

Thanks for your willingness to work on this @weitzhandler.

I have to say, however, that ambient sequences (the API approach taken by the 3rd party package Moq.Sequences) can be problematic for a variety of reasons, it would perhaps be better to have something more explicit; such as creating a sequence, then registering mocks with it, then in the end calling verification methods on the sequence.

VerifyInSequence would be another possibile design, we'd then likely have to record a steadily increasing sequence number (think a tick of a global virtual clock) with every invocation so we know in which order they happened. You'd still have to pass it a sequence object because calling VerifyInSequence obviously needs to keep some state (at least, the clock tick value of the invocation last verified in one isolated test).

I'm personally gravitating to the first of the two designs just mentioned (as it has already been discussed further above and got some feedback, also we wouldn't need new method names).

[weitzhandler]

I'm personally gravitating to the first of the two designs just mentioned

I'm not sure which one you're referring to.

[weitzhandler]

Is there a central place where all setups/invocations are registered or can be intercepted from?

[stakx]

I'm not sure which one you're referring to.

The one that's been discussed in the latter half of this issue, where you register mocks with a sequence object, then call verify on the sequence.

Is there a central place where all setups/invocations are registered [...]

No, setups & invocations are recorded on the mocks to which they belong (setups) / on which they occur (invocations).

[weitzhandler]

No, setups & invocations are recorded on the mocks to which they belong (setups) / on which they occur (invocations).

I'm wondering if there is a way to get a notification when Mock.Invocations updates, any chance to turn this collection into an INotifyCollectionChanged or similar, or introduce a Mock.InvocationOccurred event? Adding a timestamp property to Invocation? Any other way to intercept an invocation occurrence? Can we "intercept the interceptors"?

Please consider reopening this issue, or should I post a new issue as my request is different, my request is allowing sequence verification for loose mocks.

[stakx]

The approach that has been discussed in this issue before does not require any events being added to mock.Invocations, nor does it require timestamps. All state would be kept in MockSequence, i.e. you'd only pay (performance wise) for what you need... which is another reason why I personally favor that approach.

[weitzhandler]

But that's only possible for strict setup, did I miss something?

[stakx]

You'd still have to register mocks with the sequence—something along the lines of mock.InSequence(mockSequence)—so that Moq knows to record all of mock's invocations in mockSequence, too.

[weitzhandler]

You'd still have to register mocks with the sequence, so that their invocations can be recorded there.

Given a mock, is there a way to listen and intercept its invocations as they happen? Because if it's possible, is easy to record them with a timestamp or chronologically. What's the magic work to look for, I'm a Castle noob, it'll help me tremendously if you give me a keyword or a reference to look for.

something along the lines of mock.InSequence(mockSequence)—so that Moq knows to record all of mock's invocations in mockSequence, too.

Oops, I see your update now. So, could I write using var seq = new Sequence(mock1, mock2, mock3); instead of mock1.InSequence(seq); mock2.InSequence(seq); mock3.InSequence(seq);? Would you consider that?

Thank you for everything @stakx!

[stakx]

Given a mock, is there a way to listen and intercept its invocations as they happen?

No, but that shouldn't be necessary. It would perhaps be easiest if mocks actively cooperated with sequences and recorded their invocations there.

What's the magic work to look for, I'm a Castle noob, it'll help me tremendously if you give me a keyword or a reference to look for.

It shouldn't be necessary to descend into the parts of Moq that deal with Castle DynamicProxy. Moq builds its own abstractions on top. Moq's method invocation interception pipeline can be found there. I should probably give you a few more details how we should hook it up with MockSequence but until then take a look if you'd like.

One word of caution: it's good to keep in mind that all code we add in the interception pipeline will run for every single method that gets called on mocks. That's why it's somewhat important to keep that code as slim as possible. I'm not saying we need to micro-optimise (though I've been guilty of doing just that in the past :wink:) but just to stay aware of performance in that part of Moq.

using var seq = new Sequence(mock1, mock2, mock3); instead of mock1.InSequence(seq);

(I'm going to assume you really meant MockSequence, not Sequence.)

I haven't given that too much thought yet, TBH. I suppose that, in order to keep in line with the existing API, it would be nice if you could register mocks to an already existing sequence (similar to how you can keep adding setups to a mock), which wouldn't be possible if we did the ctor syntax. Let me get back to you on that point.

[weitzhandler]

No, but that shouldn't be necessary. It would perhaps be easiest if mocks actively cooperated with sequences and recorded their invocations there.

Ok. I guess then it'll be fairly easy to implement my suggestion above (VerifyInSequence).

But do you think the following could also be achievable:

// arrange
var sut = new SystemUnderTest(mock1, mock2, ...);

using var seq = new MockSequence(mock1, mock2, ...); // as soon as we add to sequence, all verifications on this mock 
                                                     // are now intercepted by the sequence too

// act
sut.DoWork();

// assert
mock1.Verify(service1 => service1.DoWork(), Times.Once); // mocks are automatically tracked by sequence
mock2.Verify(service2 => service2.DoWork(), Times.Once);

Is there a way to "intercept" verifications on a mock? Or do you even like this way?

Ideally my goal would be:

• [x] Setup a sequence once with all mocks
• [x] No need for mock setup
• [ ] Regular verification on mock triggers sequence verification, if mock is in sequence One way to achieve this, is having a single/list of sequences in the mock itself, so that that it notifies the sequence about an invocation/verification. Though not a great design I think.