Investigate left over arg matchers

[dtchepak]

Describe the bug Example shown in this post suggests arg matchers not being cleared correctly for DidNotReceiveWithAnyArgs (and potentially with ReceivedWithAnyArgs).

To Reproduce

I have not not been able to reproduce this with NSub 3.1. Here is what I tried:

using Xunit;
using NSubstitute;

namespace NSubTestWorkshop {

    // Based on example here: https://www.elasticmint.com/blog/2018/04/16/fixing-intermittent-test-failures/

    public interface IMapper {
        string Map(string s);
        string AnotherCall();
    }

    public class SampleTestFixture {
        [Fact]
        public void MapperSample() {
            var mapper = Substitute.For<IMapper>();
            mapper.DidNotReceiveWithAnyArgs()
                  .Map(Arg.Any<string>());

            var remainingSpecs = NSubstitute.Core.SubstitutionContext.Current.DequeueAllArgumentSpecifications();
            Assert.Empty(remainingSpecs);
        }
    }
}

This tests passes as hoped.

Additional context Is there other tests that can reproduce the behaviour seen in the blog post? Not sure about what version of NSub is being used there. The post also links to @zvirja's answer and DiagnosticsSubstitutionContext.cs snippet.

[zvirja]

@dtchepak Briefly checking the source code I cannot see any difference between DidNotReceiveWithAnyArgs() and Received() as internally we use the same code. Better ask client for the problem code, otherwise it's really hard to guess 😟

[dtchepak]

@zvirja I've asked the post author. :) I get the feeling it was in real code so may be difficult to get an independent reproduction, but we'll see.

[britebit]

Hi all, I'm sorry, but I was unable to reproduce this problem locally (I am the person who wrote the post the Elastic Mint blog ). I don't have access to the original codebase else I would be able to go back through the git history and pull something out from there. I'll send an email to the devs in the company I was working with to see if they can help. Best Regards, Gordon

[dtchepak]

Thanks @britebit! Appreciate you going to those lengths to follow it up.

[rondelward]

I was reading the blog post linked and saw the comments mentioned there. I ran into a similar issue myself also. Our issue specifically came up because we were using Arg.Any<> in a place where it shouldn't have been used

[SetUp]
        public void SetUp()
        {
            _userFilterRepository = Substitute.For<IMongoUserFilterRepository>();
            _userFilterServiceHelper = new UserFilterServiceHelper(_userFilterRepository);
        }
...
[Test]
        public void GetRequest_DoNotHaveTypeInRequest_ReturnNull()
        {
            var result = _userFilterServiceHelper.GetRequest("", Arg.Any<JObject>()); //bad usage here
            Assert.AreEqual(null, result);
        }

In our case, we were passing Arg.Any into a class that was not generated using NSubstitute(version 3.1.0). The weird part is that the actual broken test was not the one where the error eventually was thrown. It seems to be the next test (from a different test class) that attempted to use NSubstitute.

NSubstitute.Exceptions.AmbiguousArgumentsException : Cannot determine argument specifications to use.Please use specifications for all arguments of the same type.

Running any of the test fixtures in isolation did not produce the error though. Only when running multiple fixtures through my TestRunner did the error show up. Fixing the Arg.Any usage in this fixture resolved the intermittent failure of the unrelated test that was erroring out.

 [Test]
        public void GetRequest_DoNotHaveTypeInRequest_ReturnNull()
        {
            var result = _userFilterServiceHelper.GetRequest("", new JObject());
            Assert.AreEqual(null, result);
        }

Hopefully this helps with reproducing the issue

[zvirja]

@robdmoore Thanks for the scenario. Indeed, that is one of the weak NSubstitute parts and the behavior you discovered is kinda expected (we can do nothing about it, unfortunately 😟).

You might try the NSubstitute Analyzers package which should capture such kind of issues.

In our case, we were passing Arg.Any into a class that was not generated using NSubstitute(version 3.1.0).

Notice, the sample suffers from one more issue. You should use argument specifies (like Arg.Any() or Arg.Do) when configuring a call only or when you test for the received calls (i.e. call is followed by Returns(), is a part of When().. Do() chain or is preceded by Received()). In your case it was neither from those cases. See more detail here.