WebApi icon indicating copy to clipboard operation
WebApi copied to clipboard

Some tests fail when run in full test suite (succeed individually)

Open lisicase opened this issue 2 years ago • 1 comments

When running all of the tests in Microsoft.Test.E2E.AspNet.OData in conjunction, some tests fail on the first try. (If run again individually or as a separate group, they then succeed.)

Assemblies affected

Observed on the latest main branch of OData WebApi (lib v7.5.17).

Reproduce steps

Run all tests in Microsoft.Test.E2E.AspNet.OData.

Expected result

All tests succeed.

Actual result

The following four tests fail in Microsoft.Test.E2E.AspNet.OData.ModelBoundQuerySettings.CombinedTest.CombinedTest:

  • DefaultQuerySettings
    • DefaultQuerySettings(url: "{0}/enablequery/Orders?$top=3", error: "top")
    • DefaultQuerySettings(url: "{0}/modelboundapi/Orders?$top=4", error: "top")
  • QuerySettingsOnPropertyNegative
    • QuerySettingsOnPropertyNegative(entitySetUrl: "{0}/enablequery/Orders", url: "?$expand=Customers2($expand=Order($top=3))", error: "top")
    • QuerySettingsOnPropertyNegative(entitySetUrl: "{0}/modelboundapi/Orders", url: "?$expand=Customers2($expand=Order($top=3))", error: "top")

Failure message (same for all failing tests): Assert.Equal() Failure Expected: BadRequest Actual: OK

Additional detail

If the entire E2E.AspNet.OData test suite is run a second time, these tests continue to fail. However, if the failed tests are run individually or in a smaller test set (such as ModelBoundQuerySettings.CombinedTest), they then succeed.

This could be impacted by potential interdependency between tests.

lisicase avatar Oct 12 '22 19:10 lisicase

Repros on commit cda7e833b7e86453fdf41666d1d2e593a70e8176

corranrogue9 avatar Oct 13 '22 19:10 corranrogue9