[Feature]: Playwright.Xunit should have a V3 variant

[alexaka1]

🚀 Feature Request

With XUnit v3 released (https://xunit.net/releases/v3/1.0.0), Playwright should support it the same way it supports NUnit, MSTest etc.

Example

No response

Motivation

It will allow Playwright to utilize the new TestContext of XUnit v3, such as TestContext.Current.TestStatus for enabling tracing on failed tests (currently not possible with xunit v2).

[mxschmitt]

Linking https://github.com/xunit/xunit/issues/3146 and https://github.com/microsoft/playwright-dotnet/pull/3097. Looks like this requires architectural changes.

[omni-htg]

Should proposals be set up over a new PR, or continue over what #3097 has achieved so far? Thanks!

[alexaka1]

Sorry guys, I kept meaning to post my workaround but completely forgot it. This is what I went with, works perfectly for my project:

using System.Collections.Concurrent;
using System.Globalization;
using System.Text.Json;
using System.Text.Json.Serialization;
using Microsoft.Playwright.TestAdapter;
using Xunit;

// ReSharper disable once CheckNamespace
namespace Microsoft.Playwright.Xunit;

public class PageTest : ContextTest
{
    public IPage Page { get; private set; } = null!;

    public override async ValueTask InitializeAsync()
    {
        await base.InitializeAsync().ConfigureAwait(false);
        Page = await Context.NewPageAsync().ConfigureAwait(false);
    }
}

public class ContextTest : BrowserTest
{
    public IBrowserContext Context { get; private set; } = null!;

    public override async ValueTask InitializeAsync()
    {
        await base.InitializeAsync().ConfigureAwait(false);
        Context = await NewContext(ContextOptions()).ConfigureAwait(false);
    }

    public virtual BrowserNewContextOptions ContextOptions()
    {
        return new BrowserNewContextOptions
        {
            Locale = "en-US",
            ColorScheme = ColorScheme.Light,
        };
    }
}

public class BrowserTest : PlaywrightTest
{
    private readonly List<IBrowserContext> _contexts = new();
    public IBrowser Browser { get; internal set; } = null!;

    /// <summary>
    ///     Creates a new context and adds it to the list of contexts to be disposed.
    /// </summary>
    /// <param name="options"></param>
    /// <returns></returns>
    public async Task<IBrowserContext> NewContext(BrowserNewContextOptions? options = null)
    {
        var context = await Browser.NewContextAsync(options).ConfigureAwait(false);
        _contexts.Add(context);
        return context;
    }

    public override async ValueTask InitializeAsync()
    {
        await base.InitializeAsync().ConfigureAwait(false);
        var service = await BrowserService.Register(this, BrowserType).ConfigureAwait(false);
        Browser = service.Browser;
    }

    public override async ValueTask DisposeAsync()
    {
        if (TestOk)
        {
            foreach (var context in _contexts)
            {
                await context.CloseAsync().ConfigureAwait(false);
            }
        }

        _contexts.Clear();
        Browser = null!;
        await base.DisposeAsync().ConfigureAwait(false);
    }
}

public class PlaywrightTest : WorkerAwareTest
{
    public string BrowserName { get; internal set; } = null!;

    public IPlaywright Playwright { get; private set; } = null!;
    public IBrowserType BrowserType { get; private set; } = null!;

    private static readonly Task<IPlaywright> _playwrightTask = Microsoft.Playwright.Playwright.CreateAsync();

    public override async ValueTask InitializeAsync()
    {
        await base.InitializeAsync().ConfigureAwait(false);
        Playwright = await _playwrightTask.ConfigureAwait(false);
        BrowserName = PlaywrightSettingsProvider.BrowserName;
        BrowserType = Playwright[BrowserName];
        Playwright.Selectors.SetTestIdAttribute("data-testid");
    }

    public static void SetDefaultExpectTimeout(float timeout)
    {
        Assertions.SetDefaultExpectTimeout(timeout);
    }

    public ILocatorAssertions Expect(ILocator locator)
    {
        return Assertions.Expect(locator);
    }

    public IPageAssertions Expect(IPage page)
    {
        return Assertions.Expect(page);
    }

    public IAPIResponseAssertions Expect(IAPIResponse response)
    {
        return Assertions.Expect(response);
    }
}

public class WorkerAwareTest : ExceptionCapturer
{
    private Worker _currentWorker = null!;

    public int WorkerIndex { get; internal set; }
    private static readonly ConcurrentStack<Worker> _allWorkers = new();

    public async Task<T> RegisterService<T>(string name, Func<Task<T>> factory) where T : class, IWorkerService
    {
        if (!_currentWorker.Services.ContainsKey(name))
        {
            _currentWorker.Services[name] = await factory().ConfigureAwait(false);
        }

        return (_currentWorker.Services[name] as T)!;
    }

    public override async ValueTask InitializeAsync()
    {
        await base.InitializeAsync().ConfigureAwait(false);
        if (!_allWorkers.TryPop(out _currentWorker!))
        {
            _currentWorker = new Worker();
        }

        WorkerIndex = _currentWorker.WorkerIndex;
        if (PlaywrightSettingsProvider.ExpectTimeout.HasValue)
        {
            Assertions.SetDefaultExpectTimeout(PlaywrightSettingsProvider.ExpectTimeout.Value);
        }
    }

    public override async ValueTask DisposeAsync()
    {
        if (TestOk)
        {
            foreach (var kv in _currentWorker.Services)
            {
                await kv.Value.ResetAsync().ConfigureAwait(false);
            }

            _allWorkers.Push(_currentWorker);
        }
        else
        {
            foreach (var kv in _currentWorker.Services)
            {
                await kv.Value.DisposeAsync().ConfigureAwait(false);
            }

            _currentWorker.Services.Clear();
        }

        await base.DisposeAsync().ConfigureAwait(false);
    }

    internal class Worker
    {
        public Dictionary<string, IWorkerService> Services = [];
        public int WorkerIndex = Interlocked.Increment(ref _lastWorkedIndex);
        private static int _lastWorkedIndex;
    }
}

public interface IWorkerService
{
    public Task ResetAsync();
    public Task DisposeAsync();
}

/// <summary>
///     ExceptionCapturer is a best-effort way of detecting if a test did pass or fail in xUnit.
///     This class uses the AppDomain's FirstChanceException event to set a flag indicating
///     whether an exception has occurred during the test execution.
///     Note: There is no way of getting the test status in xUnit in the dispose method.
///     For more information, see: https://stackoverflow.com/questions/28895448/current-test-status-in-xunit-net
/// </summary>
public class ExceptionCapturer : IAsyncLifetime
{
    protected static bool TestOk { get => TestContext.Current.TestState?.Result is not TestResult.Failed; }

    public virtual ValueTask DisposeAsync()
    {
        return ValueTask.CompletedTask;
    }

    public virtual ValueTask InitializeAsync()
    {
        return ValueTask.CompletedTask;
    }
}

internal class BrowserService : IWorkerService
{
    public IBrowser Browser { get; }

    private BrowserService(IBrowser browser)
    {
        Browser = browser;
    }

    public Task DisposeAsync()
    {
        return Browser.CloseAsync();
    }

    public Task ResetAsync()
    {
        return Task.CompletedTask;
    }

    public static Task<BrowserService> Register(WorkerAwareTest test, IBrowserType browserType)
    {
        return test.RegisterService("Browser",
            async () => new BrowserService(await CreateBrowser(browserType).ConfigureAwait(false)));
    }

    private static async Task<IBrowser> CreateBrowser(IBrowserType browserType)
    {
        var accessToken = Environment.GetEnvironmentVariable("PLAYWRIGHT_SERVICE_ACCESS_TOKEN");
        var serviceUrl = Environment.GetEnvironmentVariable("PLAYWRIGHT_SERVICE_URL");

        if (string.IsNullOrEmpty(accessToken) || string.IsNullOrEmpty(serviceUrl))
        {
            return await browserType.LaunchAsync(PlaywrightSettingsProvider.LaunchOptions).ConfigureAwait(false);
        }

        var exposeNetwork = Environment.GetEnvironmentVariable("PLAYWRIGHT_SERVICE_EXPOSE_NETWORK") ?? "<loopback>";
        var os = Uri.EscapeDataString(Environment.GetEnvironmentVariable("PLAYWRIGHT_SERVICE_OS") ?? "linux");
        var runId = Uri.EscapeDataString(Environment.GetEnvironmentVariable("PLAYWRIGHT_SERVICE_RUN_ID") ??
#pragma warning disable RS0030
                                         DateTime.Now.ToUniversalTime().ToString("yyyy-MM-ddTHH:mm:ss",
#pragma warning restore RS0030
                                             CultureInfo.InvariantCulture));
        var apiVersion = "2023-10-01-preview";
        var wsEndpoint = $"{serviceUrl}?os={os}&runId={runId}&api-version={apiVersion}";
        var connectOptions = new BrowserTypeConnectOptions
        {
            Timeout = 3 * 60 * 1000,
            ExposeNetwork = exposeNetwork,
            Headers = new Dictionary<string, string>
            {
                ["Authorization"] = $"Bearer {accessToken}",
                ["x-playwright-launch-options"] = JsonSerializer.Serialize(PlaywrightSettingsProvider.LaunchOptions,
                    new JsonSerializerOptions
                    {
                        DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull,
                    }),
            },
        };

        return await browserType.ConnectAsync(wsEndpoint, connectOptions).ConfigureAwait(false);
    }
}

This is pretty much 1:1 the Xunit v2 version, except that the EceptionCapturer is no longer needed, but I kept it because I was lazy to redo the inheritance. And TestContext now has some data that can be used instead, like for TestOk. I just used Microsoft.Playwright.TestAdapter and xunit.v3.extensibility.core in a class lib, to share this with all my test projects.

[julealgon]

Is this being actively worked on? We are starting to use Playwright in our solution, and we've been using xunit v3 for all of our other tests thus far. I'd like to keep using that same version for the new UI tests as well but was surprised to find that there was no XunitV3 package for Playwright.

I see the v1.52 label there @mxschmitt . Is there an ETA for that release?

Thanks for posting your workaround @alexaka1 . We'll likely go with something like that as well while we wait for official support. I don't want to maintain 2 distinct xunit pipelines in our solution.

[kohestanimahdi]

Do you have any updates on this one?

[timbussmann]

would be really nice to get an update on this since it looks like the v1.53 milestone was just removed without replacement. xUnit 3 support would be extremely helpful.

[b4n4n4j03]

Hi,

Do you have any updates about xUnit 3 support?

[mxschmitt]

We published a pre-release: https://www.nuget.org/packages/Microsoft.playwright.xunit.v3

[julealgon]

We published a pre-release: https://www.nuget.org/packages/Microsoft.playwright.xunit.v3

Nice! Do you guys have an ETA for the non-preview release @mxschmitt ?

[mxschmitt]

1-3 weeks I assume.