mcp icon indicating copy to clipboard operation
mcp copied to clipboard
verified publisher icon microsoft

SQL FirewallRuleDeleteCommandTests test fails on MacOS in CI

Open joshfree opened this issue 1 month ago • 1 comments 

2025-11-10T16:05:20.1260180Z [xUnit.net 00:00:10.10]       Assert.Equal() Failure: Values differ
2025-11-10T16:05:20.1752000Z [xUnit.net 00:00:10.10]       Expected: BadRequest
2025-11-10T16:05:20.1930840Z [xUnit.net 00:00:10.10]       Actual:   OK
2025-11-10T16:05:20.2045450Z [xUnit.net 00:00:10.10]       Stack Trace:
2025-11-10T16:05:20.2913600Z [xUnit.net 00:00:10.11]         /Users/runner/work/1/s/tools/Azure.Mcp.Tools.Sql/tests/Azure.Mcp.Tools.Sql.UnitTests/FirewallRule/FirewallRuleDeleteCommandTests.cs(86,0): at Azure.Mcp.Tools.Sql.UnitTests.FirewallRule.FirewallRuleDeleteCommandTests.ExecuteAsync_ValidatesInputCorrectly(String args, Boolean shouldSucceed)
2025-11-10T16:05:20.3274430Z [xUnit.net 00:00:10.11]         --- End of stack trace from previous location ---
2025-11-10T16:05:20.3545480Z   Passed Azure.Mcp.Tools.Sql.UnitTests.Database.DatabaseUpdateCommandTests.ExecuteAsync_ValidatesRequiredParameters(commandArgs: "--subscription sub --resource-group rg", shouldSucceed: False, expectedError: "Missing required options") [1 ms]
2025-11-10T16:05:20.3708790Z   Passed Azure.Mcp.Tools.Sql.UnitTests.Database.DatabaseUpdateCommandTests.ExecuteAsync_ValidatesRequiredParameters(commandArgs: "--subscription sub", shouldSucceed: False, expectedError: "Missing required options") [1 ms]
2025-11-10T16:05:20.4030090Z   Passed Azure.Mcp.Tools.Sql.UnitTests.Database.DatabaseUpdateCommandTests.ExecuteAsync_ValidatesRequiredParameters(commandArgs: "--resource-group rg --server server1 --database te"···, shouldSucceed: False, expectedError: "Missing required options") [< 1 ms]
2025-11-10T16:05:20.4205030Z   Passed Azure.Mcp.Tools.Sql.UnitTests.Database.DatabaseUpdateCommandTests.ExecuteAsync_ValidatesRequiredParameters(commandArgs: "--subscription sub --resource-group rg --database "···, shouldSucceed: False, expectedError: "Missing required options") [1 ms]
2025-11-10T16:05:20.4380780Z   Passed Azure.Mcp.Tools.Sql.UnitTests.Database.DatabaseUpdateCommandTests.ExecuteAsync_ValidatesRequiredParameters(commandArgs: "--subscription sub --resource-group rg --server se"···, shouldSucceed: True, expectedError: null) [< 1 ms]
2025-11-10T16:05:20.4504670Z   Passed Azure.Mcp.Tools.Sql.UnitTests.Database.DatabaseUpdateCommandTests.ExecuteAsync_ValidatesRequiredParameters(commandArgs: "--subscription sub --resource-group rg --server se"···, shouldSucceed: False, expectedError: "Missing required options") [< 1 ms]
2025-11-10T16:05:20.4641110Z   Passed Azure.Mcp.Tools.Sql.UnitTests.Database.DatabaseUpdateCommandTests.ExecuteAsync_ValidatesRequiredParameters(commandArgs: "", shouldSucceed: False, expectedError: "Missing required options") [< 1 ms]
2025-11-10T16:05:20.4899370Z   Failed Azure.Mcp.Tools.Sql.UnitTests.FirewallRule.FirewallRuleDeleteCommandTests.ExecuteAsync_ValidatesInputCorrectly(args: "--resource-group rg --server server --firewall-rul"···, shouldSucceed: False) [10 ms]
2025-11-10T16:05:20.5023840Z   Error Message:
2025-11-10T16:05:20.5173580Z    Assert.Equal() Failure: Values differ
2025-11-10T16:05:20.5246270Z Expected: BadRequest
2025-11-10T16:05:20.5363470Z Actual:   OK
2025-11-10T16:05:20.5532820Z   Stack Trace:
2025-11-10T16:05:20.5747980Z      at Azure.Mcp.Tools.Sql.UnitTests.FirewallRule.FirewallRuleDeleteCommandTests.ExecuteAsync_ValidatesInputCorrectly(String args, Boolean shouldSucceed) in /Users/runner/work/1/s/tools/Azure.Mcp.Tools.Sql/tests/Azure.Mcp.Tools.Sql.UnitTests/FirewallRule/FirewallRuleDeleteCommandTests.cs:line 86
2025-11-10T16:05:20.5850200Z --- End of stack trace from previous location ---
2025-11-10T16:05:20.6000560Z   Passed Azure.Mcp.Tools.Sql.UnitTests.FirewallRule.FirewallRuleDeleteCommandTests.ExecuteAsync_Handles403Error [3 ms]
2025-11-10T16:05:20.6113150Z   Passed Azure.Mcp.Tools.Sql.UnitTests.Database.DatabaseUpdateCommandTests.ExecuteAsync_UpdateFromBasicToPremium_Succeeds [52 ms]
2025-11-10T16:05:20.6265900Z   Passed Azure.Mcp.Tools.Sql.UnitTests.FirewallRule.FirewallRuleDeleteCommandTests.ExecuteAsync_HandlesArgumentException [34 ms]
2025-11-10T16:05:20.6441560Z   Passed Azure.Mcp.Tools.Sql.UnitTests.FirewallRule.FirewallRuleDeleteCommandTests.Command_HasCorrectMetadata [< 1 ms]
2025-11-10T16:05:20.7211830Z   Passed Azure.Mcp.Tools.Sql.UnitTests.Database.DatabaseUpdateCommandTests.ExecuteAsync_HandlesAuthorizationFailure [1 ms]
2025-11-10T16:05:20.7610790Z   Passed Azure.Mcp.Tools.Sql.UnitTests.FirewallRule.FirewallRuleDeleteCommandTests.ExecuteAsync_VerifiesResultContainsExpectedData [1 ms]
2025-11-10T16:05:20.7853810Z   Passed Azure.Mcp.Tools.Sql.UnitTests.Database.DatabaseUpdateCommandTests.ExecuteAsync_WithValidParameters_UpdatesDatabase [2 ms]
2025-11-10T16:05:20.8109400Z   Passed Azure.Mcp.Tools.Sql.UnitTests.FirewallRule.FirewallRuleDeleteCommandTests.ExecuteAsync_Handles404Error [2 ms]
2025-11-10T16:05:20.8335710Z [xUnit.net 00:00:10.16]   Finished:    Azure.Mcp.Tools.Sql.UnitTests
2025-11-10T16:05:20.8447470Z Starting test execution, please wait...
2025-11-10T16:05:20.8707350Z   Passed Azure.Mcp.Tools.Sql.UnitTests.Database.DatabaseUpdateCommandTests.ExecuteAsync_UpdateFromStandardToBasic_Succeeds [2 ms]
2025-11-10T16:05:20.8805320Z   Passed Azure.Mcp.Tools.Sql.UnitTests.FirewallRule.FirewallRuleDeleteCommandTests.ExecuteAsync_CallsServiceWithCorrectParameters [1 ms]
2025-11-10T16:05:20.9427750Z   Passed Azure.Mcp.Tools.Sql.UnitTests.Database.DatabaseUpdateCommandTests.Constructor_InitializesCommandCorrectly [< 1 ms]
2025-11-10T16:05:20.9695020Z   Passed Azure.Mcp.Tools.Sql.UnitTests.Database.DatabaseUpdateCommandTests.ExecuteAsync_UpdateFromBasicToStandard_Succeeds [2 ms]
2025-11-10T16:05:20.9900630Z A total of 1 test files matched the specified pattern.
2025-11-10T16:05:21.0078810Z /Users/runner/work/1/s/tools/Azure.Mcp.Tools.AzureBestPractices/tests/Azure.Mcp.Tools.AzureBestPractices.UnitTests/bin/Debug/net9.0/Azure.Mcp.Tools.AzureBestPractices.UnitTests.dll
2025-11-10T16:05:21.0168510Z Test Run Failed.
2025-11-10T16:05:21.0230330Z Results File: /Users/runner/work/1/a/testResults/_sjc20-cw758-2ba54ffd-943c-4f37-8871-66484c8274fb-925BAAA4243C_2025-11-10_16_05_11[1].trx
2025-11-10T16:05:21.0345560Z 
2025-11-10T16:05:21.0404800Z Total tests: 221
2025-11-10T16:05:21.0597770Z      Passed: 220
2025-11-10T16:05:21.0849830Z      Failed: 1
2025-11-10T16:05:21.1239720Z  Total time: 13.3865 Seconds

joshfree avatar Nov 10 '25 16:11 joshfree

Hey @ericshape is there anything we can do to improve the reliability of this test?

Would switching to a static resource instead of dynamically provisioning the resource help or is that orthogonal to the issue here?

joshfree avatar Nov 13 '25 16:11 joshfree