mcp icon indicating copy to clipboard operation
mcp copied to clipboard

[BUG] Managed Lustre live tests fail with message: "Failed to create AML file system"

Open vcolin7 opened this issue 1 month ago • 2 comments

Describe the bug

Recently, Managed Lustre live tests have been failing with the following error message:

  Failed Azure.Mcp.Tools.ManagedLustre.LiveTests.ManagedLustreCommandTests.Should_create_azure_managed_lustre_no_blob_no_cmk [8 s]
  Error Message:
   Property 'fileSystem' not found. Full element: '{"message":"Failed to create AML file system \u0027amlfs-fb5caf58\u0027: Operation results in exceeding quota limits of resource type AmlFilesystem. Maximum allowed: 4, Current in use: 4, Additional requested: 1.\nStatus: 409 (Conflict)\nErrorCode: OperationNotAllowed\n\nContent:\n{\r\n  \u0022error\u0022: {\r\n    \u0022code\u0022: \u0022OperationNotAllowed\u0022,\r\n    \u0022message\u0022: \u0022Operation results in exceeding quota limits of resource type AmlFilesystem. Maximum allowed: 4, Current in use: 4, Additional requested: 1.\u0022\r\n  }\r\n}\n\nHeaders:\nCache-Control: no-cache\nPragma: no-cache\nStrict-Transport-Security: REDACTED\nx-ms-served-by: REDACTED\nx-ms-request-id: 320d7588-6a6a-429a-9d57-4a1d478ec6ab\nx-ms-client-request-id: fe27d6af-776f-4fe0-b43c-8da56d67a995\nx-ms-operation-identifier: REDACTED\nx-ms-ratelimit-remaining-subscription-writes: REDACTED\nx-ms-ratelimit-remaining-subscription-global-writes: REDACTED\nx-ms-correlation-request-id: REDACTED\nx-ms-routing-request-id: REDACTED\nX-Content-Type-Options: REDACTED\nX-Cache: REDACTED\nX-MSEdge-Ref: REDACTED\nDate: Thu, 13 Nov 2025 18:55:49 GMT\nContent-Length: 217\nContent-Type: application/json; charset=utf-8\nExpires: -1\n","stackTrace":"   at Azure.Mcp.Tools.ManagedLustre.Services.ManagedLustreService.CreateFileSystemAsync(String subscription, String resourceGroup, String name, String location, String sku, Int32 sizeTiB, String subnetId, String zone, String maintenanceDay, String maintenanceTime, String hsmContainer, String hsmLogContainer, String importPrefix, String rootSquashMode, String noSquashNidLists, Nullable\u00601 squashUid, Nullable\u00601 squashGid, Boolean enableCustomEncryption, String keyUrl, String sourceVaultId, String userAssignedIdentityId, String tenant, RetryPolicyOptions retryPolicy) in /mnt/vss/_work/1/s/tools/Azure.Mcp.Tools.ManagedLustre/src/Services/ManagedLustreService.cs:line 381\n   at Azure.Mcp.Tools.ManagedLustre.Commands.FileSystem.FileSystemCreateCommand.ExecuteAsync(CommandContext context, ParseResult parseResult, CancellationToken cancellationToken) in /mnt/vss/_work/1/s/tools/Azure.Mcp.Tools.ManagedLustre/src/Commands/FileSystem/FileSystemCreateCommand.cs:line 113","type":"Exception"}'
  Stack Trace:
     at Azure.Mcp.Tests.TestExtensions.AssertProperty(JsonElement element, String propertyName) in /mnt/vss/_work/1/s/core/Azure.Mcp.Core/tests/Azure.Mcp.Tests/TestExtensions.cs:line 38
   at Azure.Mcp.Tests.TestExtensions.AssertProperty(Nullable`1 element, String propertyName) in /mnt/vss/_work/1/s/core/Azure.Mcp.Core/tests/Azure.Mcp.Tests/TestExtensions.cs:line 33
   at Azure.Mcp.Tools.ManagedLustre.LiveTests.ManagedLustreCommandTests.Should_create_azure_managed_lustre_no_blob_no_cmk() in /mnt/vss/_work/1/s/tools/Azure.Mcp.Tools.ManagedLustre/tests/Azure.Mcp.Tools.ManagedLustre.LiveTests/ManagedLustreCommandTests.cs:line 157
--- End of stack trace from previous location ---
  Standard Output Messages:
 Attempting to start MCP Client
 MCP client initialized successfully
 request: {"command":"managedlustre_fs_create","parameters":{"subscription":"4d042dc6-fe17-4698-a23f-ec6a8d1e98f4","resource-group":"SSS3PT_rg-***","location":"westus","name":"amlfs-fb5caf58","sku":"AMLFS-Durable-Premium-500","size":4,"zone":1,"subnet-id":"***","hsm-container":"***","hsm-log-container":"***","custom-encryption":true,"key-url":"***","source-vault":"***","user-assigned-identity-id":"***","maintenance-day":"Monday","maintenance-time":"01:00","root-squash-mode":"All","no-squash-nid-list":"10.0.0.4","squash-uid":1000,"squash-gid":1000}}
 response: {
   "status": 500,
   "message": "Failed to create AML file system \u0027amlfs-fb5caf58\u0027: Operation results in exceeding quota limits of resource type AmlFilesystem. Maximum allowed: 4, Current in use: 4, Additional requested: 1.\nStatus: 409 (Conflict)\nErrorCode: OperationNotAllowed\n\nContent:\n{\r\n  \u0022error\u0022: {\r\n    \u0022code\u0022: \u0022OperationNotAllowed\u0022,\r\n    \u0022message\u0022: \u0022Operation results in exceeding quota limits of resource type AmlFilesystem. Maximum allowed: 4, Current in use: 4, Additional requested: 1.\u0022\r\n  }\r\n}\n\nHeaders:\nCache-Control: no-cache\nPragma: no-cache\nStrict-Transport-Security: REDACTED\nx-ms-served-by: REDACTED\nx-ms-request-id: 320d7588-6a6a-429a-9d57-4a1d478ec6ab\nx-ms-client-request-id: fe27d6af-776f-4fe0-b43c-8da56d67a995\nx-ms-operation-identifier: REDACTED\nx-ms-ratelimit-remaining-subscription-writes: REDACTED\nx-ms-ratelimit-remaining-subscription-global-writes: REDACTED\nx-ms-correlation-request-id: REDACTED\nx-ms-routing-request-id: REDACTED\nX-Content-Type-Options: REDACTED\nX-Cache: REDACTED\nX-MSEdge-Ref: REDACTED\nDate: Thu, 13 Nov 2025 18:55:49 GMT\nContent-Length: 217\nContent-Type: application/json; charset=utf-8\nExpires: -1\n. To mitigate this issue, please refer to the troubleshooting guidelines here at [https://aka.ms/azmcp/troubleshooting.",](https://aka.ms/azmcp/troubleshooting.%22,)
   "results": {
     "message": "Failed to create AML file system \u0027amlfs-fb5caf58\u0027: Operation results in exceeding quota limits of resource type AmlFilesystem. Maximum allowed: 4, Current in use: 4, Additional requested: 1.\nStatus: 409 (Conflict)\nErrorCode: OperationNotAllowed\n\nContent:\n{\r\n  \u0022error\u0022: {\r\n    \u0022code\u0022: \u0022OperationNotAllowed\u0022,\r\n    \u0022message\u0022: \u0022Operation results in exceeding quota limits of resource type AmlFilesystem. Maximum allowed: 4, Current in use: 4, Additional requested: 1.\u0022\r\n  }\r\n}\n\nHeaders:\nCache-Control: no-cache\nPragma: no-cache\nStrict-Transport-Security: REDACTED\nx-ms-served-by: REDACTED\nx-ms-request-id: 320d7588-6a6a-429a-9d57-4a1d478ec6ab\nx-ms-client-request-id: fe27d6af-776f-4fe0-b43c-8da56d67a995\nx-ms-operation-identifier: REDACTED\nx-ms-ratelimit-remaining-subscription-writes: REDACTED\nx-ms-ratelimit-remaining-subscription-global-writes: REDACTED\nx-ms-correlation-request-id: REDACTED\nx-ms-routing-request-id: REDACTED\nX-Content-Type-Options: REDACTED\nX-Cache: REDACTED\nX-MSEdge-Ref: REDACTED\nDate: Thu, 13 Nov 2025 18:55:49 GMT\nContent-Length: 217\nContent-Type: application/json; charset=utf-8\nExpires: -1\n",
     "stackTrace": "   at Azure.Mcp.Tools.ManagedLustre.Services.ManagedLustreService.CreateFileSystemAsync(String subscription, String resourceGroup, String name, String location, String sku, Int32 sizeTiB, String subnetId, String zone, String maintenanceDay, String maintenanceTime, String hsmContainer, String hsmLogContainer, String importPrefix, String rootSquashMode, String noSquashNidLists, Nullable\u00601 squashUid, Nullable\u00601 squashGid, Boolean enableCustomEncryption, String keyUrl, String sourceVaultId, String userAssignedIdentityId, String tenant, RetryPolicyOptions retryPolicy) in /mnt/vss/_work/1/s/tools/Azure.Mcp.Tools.ManagedLustre/src/Services/ManagedLustreService.cs:line 381\n   at Azure.Mcp.Tools.ManagedLustre.Commands.FileSystem.FileSystemCreateCommand.ExecuteAsync(CommandContext context, ParseResult parseResult, CancellationToken cancellationToken) in /mnt/vss/_work/1/s/tools/Azure.Mcp.Tools.ManagedLustre/src/Commands/FileSystem/FileSystemCreateCommand.cs:line 113",
     "type": "Exception"
   },
   "duration": 0
 }

Expected behavior

The live test Azure.Mcp.Tools.ManagedLustre.LiveTests.ManagedLustreCommandTests.Should_create_azure_managed_lustre_no_blob_no_cmk should succeed.

Actual behavior

The test fails with the error mentioned above.

Reproduction Steps

Run live tests for all tools using this pipeline with PublishTarget set to internal.

Environment

No response

vcolin7 avatar Nov 13 '25 19:11 vcolin7

Closing as a request for an increased limit was submitted and approved.

alzimmermsft avatar Nov 14 '25 17:11 alzimmermsft

Re-opening as we saw this happen again for another live test run.

vcolin7 avatar Nov 20 '25 23:11 vcolin7