NUnit Console/Engine v4

[ChrisMaddock]

We've talked casually about several changes we'd like to make in v4 over the years - but there's never yet been an important enough breaking change to justify a new major version. Most of our big projects currently seem to be possible in a backwards-compatible fashion. My view however is that the .NET Core changes are starting to create these, and it might be time to start thinking about the future.

I'd like to start tracking changes we'd like to make in v4 - I've created the v4 milestone to do that. Consider this an open invite to anyone to put in issues against v4! 😄 For now, let's just go with tracking them - in future we might create a v4 branch, but I don't want to introduce the overhead of having to keep two branches in sync until we need to.

For issues under 'v4', I'd like to mainly target them towards breaking changes - new features that could be done in a backwards-compatible way in v3 seem much better not to be delayed, and tracked as normal issues. Ultimately I don't see v4 being a 'big bang' release - more a well-curated group of minor breaking changes, to make our lives easier and allow us to move quicker in future.

Thoughts very welcome. 🙂

[ChrisMaddock]

I'd encourage everyone to create to create an issue-per-change as normal for this, but due to time, here's a quick brain dump of things I see possibly going in to this. I'll create individual issues for these at some later point, with more time.

• Improved .NET Core packaging
• Removal of App domain support
• Possible conversion for self-contained .NET Core Console/Engine, with platform-specific agents?
• Resultant removal of inprocess running
• Agent-level extensibility (e.g. for runners specific to less common platforms.)
• timeout flag renaming
• Restructuring of 'labels' options
• Legacy test-params removal/reworking
• Reworking of code visibility - remove public visibility of all engine methods, and only allow access through the API assembly
• Simplifying of different possible TestPackage formations (through current multiple constructors)
• Review and slimming of NUnit Engine API.

[CharliePoole]

@ChrisMaddock I'd like to add some things which are more like "decide whether to dos" than "todos". Probably using label is:question (for the team) or is:idea depending on how doubtful we feel about the particular item.

As we discussed, I'd also like to create a GitHub Project with a subset of these items aiming toward re-converging the NUnit and TestCentric engines in V4. I'll create that under the TestCentric project, since it's a goal for that project and it will allow pairing up issues in both projects, where one or both may need to make a change in order to converge.

One comment about not wanting this to be a "big bang"...

I think that's desirable and also possible to a certain extent. The API is the issue. Once you release an API, you are stuck unless we carefully define what changes are allowed. I'll continue in the individual issues we create for this.

[CharliePoole]

@ChrisMaddock Re your list of potential issues... could you clarify a few things? (either here or in creating the individual issues):

• What does "Removal of AppDomain support" mean to you? Assuming we continue to support .NET Framework, we have to support AppDomains in some sense. So in what sense would you remove the support?

• Regarding TestPackage are you talking about removing one of the inconsistent constructors or adding new constructors?

[CharliePoole]

@ChrisMaddock Do we have a written definition of "Breaking Change" anywhere? I know we have an unwritten one, which is essentially to try not to break anyone even if they use the code in an unexpected way.

One thing to consider for V4 is to write down a definition and publish it so that we do not have to feel responsible for "breaking" people who rely on implementation details we don't want them to rely on. I wrote a document for the TestCentric GUI, which came out of my experience with NUnit and our changing view of "breaking" over time.

I suspect we might all argue a bit over the details, but I hope that "writing it down" isn't a point of contention. :smile_cat:

[ChrisMaddock]

Sounds good. For v4 - I'd like to enforce public/internal classes as aggressively as possible - so we lose as much of this ambiguity as possible.

Would you like to take the lead on writing something up, and then we'll argue it out? 😇

[ChrisMaddock]

Oh - I only saw the last response at first. On your other points:

• "decide whether to do" totally agree
• "Testcentic/nunit reconciliation project" sounds good. Here or on TC is fine by me, everything we've discussed so far has been desirable NUnit Engine functionality anyway. If there's anything we disagree on - let's cross that bridge when we come to it!
• TestPackage ctors - you've thought into it more than I have already. 😁 My desire with this is to reduce all the complexity we currently have to deal with in the rest of the engine. I want there to be a single standardised package structure that the engine can expect to work with. The details of that are still to be worked out!
• App domain support - I mean remove all user-controllable app domain isolation. v3 has proved to us that we struggle to manage both app domain and process isolation together with all out project functionality - we're on v3.11 and we still don't have this working reliably! 😅 I'd like to simplify the options we provide in terms of test assembly isolation - with process isolation being up front, as the version which will work across all platforms.

[ChrisMaddock]

Been thinking more about this - I'd like to lay out some high-level key objectives for v4, to give us more focus. Here's some of my thoughts, suggestions welcome! I might try and write something up for this, next time I get a little time.

1. Test Anywhere

Since NUnit 3.0 was released, the .NET scene has evolved from being largely Windows focussed, to running across a whole bunch of different platforms. Currently users often need to set up large amounts of boilerplate code to test across these platforms - the next generation engine should provide that functionality directly to the user. The engine should also be suitably extensible, to allow users to run tests on less-common platforms that the project itself isn't resourced to support.

2. Improving the Developer Experience

NUnit has a long, proud history and a big user-base, utilising a range of technologies from the last 20 years of .NET development. This is not without it's cost - we have a large codebase still restricted to building for legacy platforms, and a drawn-out and manual release process, which impacts our ability to move quickly.

For v4, I'd like to see us improve the experience for developers of the engine and console. This would cover things like moving as much code as possible to modern platforms, reducing our reliance on manual testing and release processes, and considering code complexity as a factor against feature value and development.

3. Clarity of supported interfaces, and protection for the future

A big cost of on-going development, particularly in the framework, is handling code which has been exposed to end users, but was never intended to be as supported interface. The team are often in a position of 'guessing' how user's may be using a class to decide on an appropriate change - which delays and confuses external contributions, and often unintentionally breaks end-users' setups.

For v4, I'd like for us to consider locking everything down as far as possible, and only exposing guaranteed interfaces, intended for ongoing support. Docs are great too - but locking things down prevents people making the mistake right from the start.

---

Thoughts and ideas welcome. 🙂

[CharliePoole]

I like the high level goals. They provide a vision that your team (us) can buy into. I'm missing one thing... How about a statement about backward compatibility? Traditionally, NUnit has been at an extreme of backard compatibility as compared to other projects - and especially to Microsot. We've moved somewhat away from that or at least changed what backward compatibility means to us. Shouldn't we say something that clarifies where we are on that now?

As a straw man, I would say "We will continue to support running tests on all the platforms supported by NUnit 3, although we may no longer support running some platforms in process."

[ChrisMaddock]

It's a good point Charlie, and one I've thought a lot about. Your suggestion motivated me to start writing them down, and it turned into a small essay I'm afraid. 😅 Let's discuss this over at https://github.com/nunit/nunit-console/issues/770.

[ChrisMaddock]

I should add the above is specifically relating to platforms, as that was the particular element you focussed on in your straw-man. 🙂 I'd like to also expand our backwards compatibility statement to cover the API's and functionality we intend on supporting - the engine API of course, and then additional particular console behaviour, any generated xml, packaging, and other things we may see fit.

[CharliePoole]

I liked what you wrote!

WRT more general backwards compatibility, is there an existing statement somewhere? If so, I wasn't aware of it. I wrote something for my own work in the GUI because I think people will be OK - or more OK - with changes if you have given them a statement in the first place.

[ChrisMaddock]

No, not yet, at least not that I know of. It sounds like a sensible thing to document.

[CharliePoole]

I think most of what is covered in this issue has been either completed or documented as an individual issue. I'll review it before closing.