Run tests on self-contained test project without dotnet runtime

[wli3]

Description

similar to https://github.com/microsoft/vstest/issues/775 But we don't want to install dotnet core runtime. In order to run self-cotnained tests. Since it is self-contained, it should no longer required dotnet core runtime.

This could greatly help testing IOT device. Since requiring dotnet core runtime is much harder for such devices with limited internet access.

Steps to reproduce

dotnet new xunit dotnet publish -r linux-arm /p:selfcontained=true /p:UseAppHost=true

Expected behavior

There is an vstest executable to run the tests

Actual behavior

There is only an test dll executable, which is no use.

[efoleymsft]

+1 we would like a self contained vstest.console executable to run test executions cross platform without installing the runtime.

[nohwnd]

Played with this over lunch. One way would be to import the cli package and then invoke the test project direcly, passing on the parameters for test console:

# file Program.cs
using System.IO;
using System.Linq;
using System.Reflection;

namespace self_contained
{
    public static class Program
    {
        public static int Main(string[] args)
        {
            var currentAssembly = typeof(Program).Assembly.Location;
            var assemblyDir = Path.GetDirectoryName(currentAssembly);
            Directory.SetCurrentDirectory(assemblyDir);

            var argsWithPath = args.Concat(new[] { currentAssembly }).ToArray();
            var asm = Assembly.LoadFrom(Path.Combine(assemblyDir, @"vstest.console.dll"));

            var program = asm.GetTypes().Where(t => t.Name == "Program").First();
            var main = program.GetMethods().Where(m => m.Name == "Main").First();

            return (int)main.Invoke(null, new object[] { argsWithPath });
        }
    }
}

# file self-contained.csproj
<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <GenerateProgramFile>false</GenerateProgramFile>
    <TargetFramework>netcoreapp3.1</TargetFramework>
    <RootNamespace>self_contained</RootNamespace>

    <IsPackable>false</IsPackable>

    <StartupObject></StartupObject>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="16.5.0" />
    <PackageReference Include="Microsoft.TestPlatform.CLI" Version="16.5.0" />
    <PackageReference Include="xunit" Version="2.4.0" />
    <PackageReference Include="xunit.runner.visualstudio" Version="2.4.0" />
    <PackageReference Include="coverlet.collector" Version="1.2.0" />
  </ItemGroup>
</Project>

# file UnitTest1.cs
using System;
using Xunit;

namespace self_contained
{
    public class UnitTest1
    {
        [Fact]
        public void Test1()
        {
            throw new Exception("faaaaail");
        }
    }
}

Notice <GenerateProgramFile>false</GenerateProgramFile> which prevents test.sdk from emitting defualt entrypoint.

Then I just did

dotnet publish -r linux-x64 /p:selfcontained=true /p:UseAppHost=true .\self-contained.csproj --output publish

The runtime is different than yours, but I was able to run it in WSL2 as:

/mnt/c/projects/temp/self-contained$ publish/self-contained
Microsoft (R) Test Execution Command Line Tool Version 16.5.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...

A total of 1 test files matched the specified pattern.
[xUnit.net 00:00:01.80]     self_contained.UnitTest1.Test1 [FAIL]
  X self_contained.UnitTest1.Test1 [10ms]
  Error Message:
   System.Exception : faaaaail
  Stack Trace:
     at self_contained.UnitTest1.Test1() in C:\projects\temp\self-contained\UnitTest1.cs:line 11

Test Run Failed.
Total tests: 1
     Failed: 1
 Total time: 3.4949 Seconds

I am not sure what savings this will bring, because it does not seem much smaller than installing the appropriate runtime on the target machine and running vstest console and test host using that. This approach will also require you to copy all the dlls to the target machine everytime, while if you install on the target machine you can use it for many runs.

[insightcoder]

@nohwnd I did indeed try your workaround, but when trying to run on a machine that doesn't have dotnet installed I do also get the Could not find 'dotnet' host error that you linked in #2337.

Does this mean that test projects cannot actually be packaged up to be truly self-contained?

[nohwnd]

I did one more prototype somewhere. What we would need to do would be:

1. do the thing above
2. add a condition that would do the same for testhost.dll when "testhost" option is passed

This way we have 1 executable that we build from the test project that refers both to vstest.console, and testhost and also contains the tests.

3. change the testhost provider code in vstest.console to be able to detect this mode of execution and calls the current executable passing all the parameters testhost would get + the additional --testhost parameter.

This way when we build the test project as self-contained we will have have the runtime and the executable for the target system in the publish folder. This executable would then be run in place of vstest.console, and testhost, and would also provide the tests. Say it's called MyCode.Tests.exe (exe is here just to show that this is executable, as opposed to a dll, this is of course OS dependent):

MyCode.Tests.exe -> loads vstest.console.dll and runs the Main it in. Telling it to run tests in MyCode.Tests.dll -> This would then go on till the point where testhost.exe would normally be called, failing with the error you see above because we are missing dotnet executable. But instead we would call: MyCode.Tests.exe --testhost -> this would load testhost.dll and run the Main from it. Passing MyCode.Tests.dll as the DLL that has the tests.

It's difficult to explain, sorry.

One more option would be to have InProcess test execution for .Net Core assemblies. But this would only work when running with single .dll, because we would not be able to multiplex to multiple testhosts as we do normally. Then only the steps above would need to be done.

[insightcoder]

@nohwnd Thanks for the reply and the thought. For my particular use case, we have decided to use either a console app or a service to run some high-level end-to-end tests outside of the xUnit framework. Not being able to run in self-contained mode was just one of the many factors that contributed to that. I still think that being able to run tests in self-contained mode would be extremely helpful. Thanks again!

[MartyIX]

Would fixing this issue help with https://github.com/xamarin/xamarin-macios/issues/13319?

[nohwnd]

This is a great suggestion, but we won't be implementing this into TestPlatform any time soon, sorry.