Making Microsoft.Bot.Builder.Dialogs.Adaptive.Testing GA

[gabog]

Background

The Microsoft.Bot.Builder.Dialogs.Adaptive.Testing is currently labeled as preview and being used by the SDK to generate tests scripts out of transcripts (or manually) and run unit tests on adaptive dialogs.

This issue provides brief descriptions for the different issues and design changes that need to be addressed to be able to take Microsoft.Bot.Builder.Dialogs.Adaptive.Testing out of preview:

1. Reorganize namespaces and files

There are several classes that are grouped inconsistently in the package and we need to review their namespaces and locations and place them in a folders that follows current practices in the rest of the SDK.

A good example is the different mocks we have. There is a Mocks folder that contains several mocks that don't match the folder namespaces and we also have SettingsMocks and HttpRequestMocks that are not childs of Mocks

The action for mocks would be to move everything under the Mocks folder and organize objects based on the namespace of the object they are mocking.

2. Decouple TestScript and TestActions from test execution

Currently TestScript and TestAction combine properties that describe a test and the code needed to execute it (through TestScript.ExecuteAsync() and TestAction.ExecuteAsync()). The execution is also tied to DialogManager, TestAdapter, ResourceExplorer and other mock classes.

Separating the script and the actions from execution will allow us to write tests with the same actions that don't involve dialogs (Functional Tests is a place where we would like to reuse this).

At a high level, the refactoring would involve:

• Remove test execution logic from TestScript and create a new DialogTestRunner that uses TestScript and the adaptive classes and mocks needed to run unit tests on adaptive components (it may be also worth spending time deciding how we should inject mocks so they are more extensive).
• Rename the cleaned up TestScript as DialogTestScript. We would like to be able to use TestAction to create unit tests on dialogs and also to create functional tests on bots, this refactoring will allow us to add a BotTestScript and BotTestRunner that can be used to write functional tests that interact with an entire bot while reusing existing test actions.

3. Make script execution not batch

TestScript currently "builds" the script through calls to TestScript.Send* methods, TestScript.Assert* and runs the script when you call TestScript.ExecuteAsync(). This makes step by step debugging hard and also is hard to compose multiple scripts and code to create complex scenarios.

We can refactor TestScript to preserve the fluid notation and also support step by step debugging over code.

4. Implement unit tests

There is currently no unit testing for Microsoft.Bot.Builder.Dialogs.Adaptive.Testing, we are implicitly testing it by using it in our unit tests but we will need to write formal unit tests for it if we need to make this GA.

5. Formalize and publish schemas for TestAction and TestScripts

There are several schemas in the project, some of them can probably be published it to the SDK repo once we GA to make easier to write json tests in VS code directly with intellisense and validation.

6. Code quality

Need to fix typos in comments and documentation, and also fix style and linting issues.

7. Document usage

There is currently no documentation on how to use the different classes in this package to write unit tests.

8. Add Adaptive Debugging support

It is currently not possible to debug over a TestScript json, we need to make some changes to make this possible to debug test scripts (this will also impact the Microsoft.Bot.Builder.Dialogs.Debugging preview package).

9. Update BF CLI tool to generate test scripts

BF CLI can currently generate test scripts our of emulator transcripts, we will need to update the tool to generate transcript based on the changes above and ensure it can also be extended to create functional tests.

11. Update existing unit tests

The current TestScript is being used in many adaptive unit tests and they will need to be updated to use the GA bits.

12. Review TestUtils

There is a helper TestUtils class in Microsoft.Bot.Builder.Dialogs.Adaptive.Tests that is also linked in the Microsoft.Bot.Builder.AI.Luis.Tests project, we need to review this classes and see if there are things we can promote to Microsoft.Bot.Builder.Dialogs.Adaptive.Testing.