playwright-dotnet

playwright-dotnet copied to clipboard

• Playwright Version: 1.23
• Operating System: Windows
• .NET version: 6
• Browser: Chromium

I'm getting Flaky test when running two similar test in parallel, using [Parallelizable(ParallelScope.All)]. The test always pass when running with [Parallelizable(ParallelScope.Self)], which runs one at time. I

Code Snippet

[Parallelizable(ParallelScope.All)]
public class UnionReportTests : PageTest
{

    public override BrowserNewContextOptions ContextOptions()
    {
        return new BrowserNewContextOptions()
        {
            StorageStatePath = "./state.json",
        };
    }

    // Testing Admin union reports view
    [Test]
    public async Task Unions_Reports_View()
    {
        // TEST PASS IN PARALLEL WHEN I CREATE A NEW BROWSER CONTEXT
        // Create a new context with the saved storage state.
        //var context = await Browser.NewContextAsync(new BrowserNewContextOptions
        //{
        //    StorageStatePath = "state.json"
        //});


        //var Page = await context.NewPageAsync();

        var unionPage = new UnionReportPage(Page);

        await unionPage.NavigateToUnionPage();
        // User should be on/Reports/Unions
        StringAssert.Contains("/Reports/Unions", Page.Url);

        // Table should be available
        await Expect(Page.Locator(".p-datatable#unionsReportTable")).ToBeVisibleAsync();
        await Page.WaitForSelectorAsync(".p-datatable-tbody > tr");
        int rowCount = await Page.Locator(".p-datatable-tbody > tr").CountAsync();

        Assert.True(rowCount > 0, "The union table should have more than one row");

        unionPage.searchTable("Boilermakers Local 92");

        rowCount = await Page.Locator(".p-datatable-tbody > tr").CountAsync();

        // the table should only have one record
        Assert.True(rowCount == 1, "The union table should have one row");
    }
   
     //THIS IS THE SAME TEST WITH A DIFFERENT NAME
    // Testing Admin union reports view
    [Test]
    public async Task Unions_Reports_Create()
    {
        TEST PASS IN PARALLEL WHEN I CREATE A NEW BROWSER CONTEXT
        // Create a new context with the saved storage state.
        //var context = await Browser.NewContextAsync(new BrowserNewContextOptions
        //{
        //    StorageStatePath = "state.json"
        //});


        //var Page = await context.NewPageAsync();

        var unionPage = new UnionReportPage(Page);

        await unionPage.NavigateToUnionPage();
        // User should be on /Reports/Unions
        StringAssert.Contains("/Reports/Unions", Page.Url);

        // Table should be available
        await Expect(Page.Locator(".p-datatable#unionsReportTable")).ToBeVisibleAsync();
        await Page.WaitForSelectorAsync(".p-datatable-tbody > tr");
        int rowCount = await Page.Locator(".p-datatable-tbody > tr").CountAsync();

        Assert.True(rowCount > 0, "The union table should have more than one row");

        unionPage.searchTable("Boilermakers Local 92");

        rowCount = await Page.Locator(".p-datatable-tbody > tr").CountAsync();

        // the table should only have one record
        Assert.True(rowCount == 1, "The union table should have one row");
    }

Error #1: Closes the browser because of a corrupted state, seems to be conflicting with the override??

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
  Failed Unions_Reports_View [894 ms]
  Error Message:
   Microsoft.Playwright.PlaywrightException : Target page, context or browser has been closed
  Stack Trace:
     at Microsoft.Playwright.Transport.Connection.InnerSendMessageToServerAsync[T](String guid, String method, Object args) in /_/src/Playwright/Transport/Connection.cs:line 167
   at Microsoft.Playwright.Transport.Connection.WrapApiCallAsync[T](Func`1 action, Boolean isInternal)
   at Microsoft.Playwright.Core.Browser.NewContextAsync(BrowserNewContextOptions options) in /_/src/Playwright/Core/Browser.cs:line 99
   at Microsoft.Playwright.NUnit.BrowserTest.NewContext(BrowserNewContextOptions options) in /_/src/Playwright.NUnit/BrowserTest.cs:line 38
   at Microsoft.Playwright.NUnit.ContextTest.ContextSetup() in /_/src/Playwright.NUnit/ContextTest.cs:line 42
   at NUnit.Framework.Internal.TaskAwaitAdapter.GenericAdapter`1.BlockUntilCompleted()
   at NUnit.Framework.Internal.MessagePumpStrategy.NoMessagePumpStrategy.WaitForCompletion(AwaitAdapter awaiter)
   at NUnit.Framework.Internal.AsyncToSyncAdapter.Await(Func`1 invoke)
   at NUnit.Framework.Internal.Commands.SetUpTearDownItem.RunSetUpOrTearDownMethod(TestExecutionContext context, IMethodInfo method)
   at NUnit.Framework.Internal.Commands.SetUpTearDownItem.RunSetUp(TestExecutionContext context)
   at NUnit.Framework.Internal.Commands.SetUpTearDownCommand.<>c__DisplayClass0_0.<.ctor>b__0(TestExecutionContext context)
   at NUnit.Framework.Internal.Commands.BeforeAndAfterTestCommand.<>c__DisplayClass1_0.<Execute>b__0()
   at NUnit.Framework.Internal.Commands.DelegatingTestCommand.RunTestMethodInThreadAbortSafeZone(TestExecutionContext context, Action action)
  Failed Unions_Reports_Create [934 ms]
  Error Message:
   System.InvalidOperationException : Operations that change non-concurrent collections must have exclusive access. A concurrent update was performed on this collection and corrupted its state. The collection's state is no longer correct.
  Stack Trace:
     at System.Collections.Generic.Dictionary`2.TryInsert(TKey key, TValue value, InsertionBehavior behavior)
   at System.Collections.Generic.Dictionary`2.set_Item(TKey key, TValue value)
   at Microsoft.Playwright.NUnit.WorkerAwareTest.RegisterService[T](String name, Func`1 factory) in /_/src/Playwright.NUnit/WorkerAwareTest.cs:line 53
   at Microsoft.Playwright.NUnit.BrowserTest.BrowserSetup() in /_/src/Playwright.NUnit/BrowserTest.cs:line 46
   at NUnit.Framework.Internal.TaskAwaitAdapter.GenericAdapter`1.BlockUntilCompleted()
   at NUnit.Framework.Internal.MessagePumpStrategy.NoMessagePumpStrategy.WaitForCompletion(AwaitAdapter awaiter)
   at NUnit.Framework.Internal.AsyncToSyncAdapter.Await(Func`1 invoke)
   at NUnit.Framework.Internal.Commands.SetUpTearDownItem.RunSetUpOrTearDownMethod(TestExecutionContext context, IMethodInfo method)
   at NUnit.Framework.Internal.Commands.SetUpTearDownItem.RunSetUp(TestExecutionContext context)
   at NUnit.Framework.Internal.Commands.SetUpTearDownCommand.<>c__DisplayClass0_0.<.ctor>b__0(TestExecutionContext context)
   at NUnit.Framework.Internal.Commands.BeforeAndAfterTestCommand.<>c__DisplayClass1_0.<Execute>b__0()
   at NUnit.Framework.Internal.Commands.DelegatingTestCommand.RunTestMethodInThreadAbortSafeZone(TestExecutionContext context, Action action)

Error #2: Not sure how localhost failed since both test were executed at the same time. One passed and one fail, both test were identical

Message: 
Microsoft.Playwright.PlaywrightException : net::ERR_ABORTED at https://localhost:63302/
=========================== logs ===========================
navigating to "https://localhost:63302/", waiting until "load"
============================================================

Sorry for the late response! Doing [Parallelizable(ParallelScope.All/Fixtures)] is not supported as of today, you need to run the class in serial mode, with e.g. [Parallelizable(ParallelScope.Self)].

I'll keep this issue open so we can either document or or throw if the wrong Parallelizable mode us used.

@mxschmitt Thanks for getting back to me! I mean it works well if I don't use PageTest and generate my own BrowserContext. Also Azure pipelines has many issues with using [Parallelizable(ParallelScope.All/Fixtures)] on lengthy browser test, so I just resorted to use [Parallelizable(ParallelScope.Self)]. Thanks for the clarification though.

We merged the docs change, so I think we can close this one! https://github.com/microsoft/playwright/pull/16471