dotnet-docs-samples
dotnet-docs-samples copied to clipboard
GoogleCloudPlatform
Retail: Some tests are flaky
There were these failures, possibly because the same product is being modified by all the tests and you need to create a product specifically for these tests, with maybe a random name or similar (again, remember that tests run at the same time in different environments, i.e. Windows and Linux, etc.).
Failed RetailProducts.Samples.Tests.AddFulfillmentPlacesTest.TestAddFulfillmentPlaces [1 m 36 s]
Error Message:
Assert.Contains() Failure
Not found: store2
In value: RepeatedField<String> ["store0", "store1"]
Stack Trace:
at RetailProducts.Samples.Tests.AddFulfillmentPlacesTest.TestAddFulfillmentPlaces() in T:\src\github\dotnet-docs-samples\retail\interactive-tutorial\RetailProducts.Samples.Tests\AddFulfillmentPlacesTest.cs:line 39
Failed RetailProducts.Samples.Tests.SetInventoryTest.TestSetInventory [1 m 36 s]
Error Message:
Assert.Contains() Failure
Not found: store2
In value: RepeatedField<String> ["store0", "store1"]
Stack Trace:
at RetailProducts.Samples.Tests.SetInventoryTest.TestSetInventory() in T:\src\github\dotnet-docs-samples\retail\interactive-tutorial\RetailProducts.Samples.Tests\SetInventoryTest.cs:line 40
Failed RetailProducts.Samples.Tests.RemoveFulfillmentInfoTest.TestRemoveFulfillmentPlaces [1 m 35 s]
Error Message:
Assert.DoesNotContain() Failure
Found: store0
In value: RepeatedField<String> ["store0", "store1"]
Stack Trace:
at RetailProducts.Samples.Tests.RemoveFulfillmentInfoTest.TestRemoveFulfillmentPlaces() in T:\src\github\dotnet-docs-samples\retail\interactive-tutorial\RetailProducts.Samples.Tests\RemoveFulfillmentInfoTest.cs:line 43
Originally posted by @amanda-tarafa in https://github.com/GoogleCloudPlatform/dotnet-docs-samples/issues/1837#issuecomment-1201312614
I cannot take a look myself until after Wednesday, but there's something going on, I just double check and the same tests have been flaky on nightly CI. Sometimes just one of them, sometimes all, etc. And sometimes it is all green. Definetely seems like some are stepping over the other, or maybe that changes take a while to propagate, this last bit you may be able to confirm with the API team. I can't remember, are these tests the long running operations? If they are, then I might know what is going on.
And in case they are stepping on each other: Are the IDs timed based? Up to what precission? Can you use a GUID instead or something like that?
Yes, they are long running operations. Product id: Id = Guid.NewGuid().ToString("N").Substring(0, 14)
Closing, there was a known issue with these long running operaitons, the fix has been rolling out for the past week, hence the flakiness. Roll out should have now completed so we shouldn't see failures any more. I'll keep an eye out and I'll reopen if they fail again.
Reopening as this has happened again twice in #1846.
I've checked the code and so has @mkoshurnikov , it does look OK, we are both pretty certain that the tests are not overwriting each other, etc. Maybe the previous issues with these operations has not been fully resolved yet.
This issue should've been resolved by recent backend fix. Feel free to reopen if the issue still persists.
This has happened again today, reopening:
- https://source.cloud.google.com/results/invocations/2a5ccd52-3476-4404-bc6a-56eab3dff362/log
- https://source.cloud.google.com/results/invocations/0e469764-3d17-4512-92cc-439e667ec655/log
And again: https://source.cloud.google.com/results/invocations/c248e0be-7607-4e4a-973c-6dea2ff388db/log
This is expected to be fixed by https://github.com/GoogleCloudPlatform/dotnet-docs-samples/pull/1892