Storyboard: Add ability to write and run Tests

[kendrahavens]

Purpose

Users should have clear guidance on how to write and run tests. Adding writing and running tests in try.dot.net will encourage users to test and clarify good testing practices in an easily consumable browser experience.

Experience/Scenarios

General flow. (I'm very open to suggestions.)

• Intro
  • User is able to see one test method, edit that test method (Change an Assert to passing), and run the test.
  • User can then view a product method and a test method covering it in the same window. The user must change the product code so the test passes. (Test is not editable.)
  • The user is given product code and asked to write a test to check the expected behavior. (Ex: this code should never return a negative number. Write a test to check for that.)
  • Tests help visualize behavior changes over time. An overloaded method was added to the above example. Ask the user to add another test for the new method.
• Test structure
  • Explain the different types of tests: unit, integration, manual, performance (stress testing), and more.
  • Outline the basic ideal for unit testing with a 1-1 relationship of test classes to product code classes and test projects to product code projects.
• Good test practices
  • Benefits of TDD. User receives a test and must write code to make it pass.
  • Code coverage doesn't necessarily mean good code. User must fix a test that passes that shouldn't have been passing. Writing the same logic twice is not good testing it's just more work.

Additional topics

• PR flow. If a feature is being added, it is helpful to have a test documenting the expected behavior of that feature. It gives reviewers a good idea of expectations.

Outstanding Questions

1. Experience aside, should this issue focus on designing an API for allowing 2nd party users to extend try.dot.net to use tests?

[kendrahavens]

Ideas copied from @jonsequitur:

• [ ] For test samples, allow the run button to mean "run my tests".
• [ ] Allow tests to be hidden and the implementation editable to teach test-driven problem solving.
• [ ] Allow the tests to be visible and the implementation hidden to teach test-driven ways of exploring existing behaviors.
• [ ] We would want a way to display test results that's a little more interesting than the current console output.

[LadyNaggaga]

Adding @rowanmiller as we build out this scenario

[rowanmiller]

Our scenario is a "Coding challenge". We would show a stubbed out method with some // TODOs to implement or fix the code. It would be great to define a test to verify the output.

We want to be able to give detailed feedback, so either multiple asserts with different messages we could show if they fail, or multiple unit tests.

It would be cool to have the unit test code available in a separate code listing (but not editable).

[rowanmiller]

BTW this would also apply to our current In-Browser Tutorial - currently all the validation is done in JavaScript by string parsing the code and output.

[jonsequitur]

@rowanmiller One approach that would work right now for having the test code available in a separate listing would be a multi-file Gist or a single-file Gist where the editable code is marked by a region directive. (These would be functionally equivalent.) Thoughts?

[rowanmiller]

@jonsequitur sounds promising... you support that now? Gist's wouldn't be ideal, as the Playpen system would be driven by metadata and it would be cumbersome to put everything into Gists... but we could do it as a temporary solution until we can do it without Gists.

[jonsequitur]

The code masking and multi-file support can be used without Gists. Sorry if that was unclear. The two features are usable independently.

[rowanmiller]

Nice! Do you have an example?

[jonsequitur]

On https://try.dot.net/docs take a look at the "Set source code from workspace" example.

[jonsequitur]

I've been getting the t-rex global tool into shape to gather test output for display. Since we haven't had much discussion about custom UI for this scenario, I'll toss this out to generate some.

https://github.com/jonsequitur/t-rex

[rowanmiller]

@jonsequitur once we've used setWorkspace, is there a way to reset the editor? Or send a setWorkspace message that will load up a snippet (without the full console app etc.)? Calling setCode (like we typically do for the snippets) doesn't work because (not surprisingly) it just puts the code in for the current buffer and leaves all the surrounding test harness code in place.

BTW we have the setWorkspace approach working for the test scenario, with a console app that functions as a test-harness and some well-known output that we parse to find the result and pass/fail message. Hacky... but it works 😄.

[SeanKilleen]

Hey all! Not sure where this stands but I was in the process of suggesting it when I found this issue.

I think this change in particular could have wide-reaching advantages across the ecosystem.

This idea occurred to me, for example, while browsing so many spots in the .NET docs where I see a call to Console.WriteLine and then // Will print "xyz".

I think this change has the opportunity to allow the the definition of docs examples in terms of unit tests / assertions, potentially even allowing the tests themselves to live in a repo that effectively regression-tests a lot of the surface area of .NET. It would be fantastic to enable a scenario where certain unit tests for functionality that describe the functionality also become real live docs. A great convergence. :)

I just wanted to offer my support for this as a focus for the group on the roadmap.

Thanks for all the work on this cool thing so far!

[ignatandrei]

As an workaround, I think that I can make a console application that executes dotnet test

However, it will be usefull if dotnet try can execute dotnet directly

[jonsequitur]

@SeanKilleen The scenario you outlined fits very much into our story for dotnet try. We'd love your thoughts. https://devblogs.microsoft.com/dotnet/creating-interactive-net-documentation/

[SeanKilleen]

@jonsequitur sorry it's taken me a while to get back to this. I think I may be missing some understanding of the tool, but after reading the article I'm still a little unclear.

My goal would be to have something along the lines of:

using xunit; 
using FluentAssertions;

public class AdditionTests
{
   [Fact]
   public void AddingTwoNumbersTogether_ReturnsSum()
   {
      var sum = 2+2;
      sum.Should().Be(4);
   }
}

...and have try.net run the xunit test, verifying that it passes.

I'm failing to see a path to this in the current state, but if such a path exists, let me know and I'll see if maybe I can contribute some docs or a tutorial for it. Very excited by the potential still!

[jonsequitur]

Reading back through this, my impression is that the desired goal is to enable the author of a tutorial to verify a learner's code and provide feedback based on whether it met some criteria. The use of a test framework seems like a means to an end but not critical to the experience. In fact, the output of a test run might be too limited for the tutorial author to provide a useful response. For example, you might want access to the actual code submission to make sure they solved a problem in an certain way.

@BillWagner