How to deal with Invoke-AzAksRunCommand failing with "Long running operation failed with status 'Failed'" with no additional details?

[fschmied]

Description

I'm calling Invoke-AzAksRunCommand with a command that can take a long time to complete. (It's kubectl drain which can potentially run forever, so I'm giving it a timeout of 1h.) After some time, Invoke-AzAksRunCommand will fail with the error message "Long running operation failed with status 'Failed'." with no additional details. (I'm currently using version 7.5.0 of the AZ module.)

I'd like to know:

• How long is the default timeout allowed by Invoke-AzAksRunCommand? I could tweak the timeout of my inner command to match it.
• Can I change the timeout somehow? There doesn't seem to be a cmdlet parameter, but maybe there's some other way?
• On error, can I somehow obtain the usual output that Invoke-AzAksRunCommand would normally produce, specifically stdout and stderr? That would help a lot with diagnosing why the command in question didn't complete.

Script or Debug output

No response

Environment data

Name                           Value
----                           -----
PSVersion                      7.2.0
PSEdition                      Core
GitCommitId                    7.2.0
OS                             Microsoft Windows 10.0.19044
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

Module versions

ModuleType Version    PreRelease Name                                ExportedCommands
---------- -------    ---------- ----                                ----------------
Script     7.5.0                 Az
Script     2.7.6                 Az.Accounts                         {Add-AzEnvironment, Clear-AzContext, Clear-AzDefault, Connect-AzAccount…}
Script     1.1.2                 Az.Advisor                          {Disable-AzAdvisorRecommendation, Enable-AzAdvisorRecommendation, Get-AzAdvisorConfiguration, Get-AzAdvisorRecomme… 
Script     3.1.3                 Az.Aks                              {Disable-AzAksAddOn, Enable-AzAksAddOn, Get-AzAksCluster, Get-AzAksNodePool…}
Script     1.1.4                 Az.AnalysisServices                 {Add-AzAnalysisServicesAccount, Export-AzAnalysisServicesInstanceLog, Get-AzAnalysisServicesServer, New-AzAnalysis… 
Script     2.3.2                 Az.ApiManagement                    {Add-AzApiManagementApiToGateway, Add-AzApiManagementApiToProduct, Add-AzApiManagementProductToGroup, Add-AzApiMan… 
Script     1.1.0                 Az.AppConfiguration                 {Get-AzAppConfigurationStore, Get-AzAppConfigurationStoreKey, New-AzAppConfigurationStore, New-AzAppConfigurationS… 
Script     1.3.1                 Az.ApplicationInsights              {Get-AzApplicationInsights, Get-AzApplicationInsightsApiKey, Get-AzApplicationInsightsContinuousExport, Get-AzAppl… 
Script     1.0.0                 Az.Attestation                      {Add-AzAttestationPolicySigner, Get-AzAttestation, Get-AzAttestationPolicy, Get-AzAttestationPolicySigners…}        
Script     1.7.3                 Az.Automation                       {Export-AzAutomationDscConfiguration, Export-AzAutomationDscNodeReportContent, Export-AzAutomationRunbook, Get-AzA… 
Script     3.2.0                 Az.Batch                            {Disable-AzBatchAutoScale, Disable-AzBatchComputeNodeScheduling, Disable-AzBatchJob, Disable-AzBatchJobSchedule…}   
Script     2.0.0                 Az.Billing                          {Get-AzBillingAccount, Get-AzBillingInvoice, Get-AzBillingPeriod, Get-AzBillingProfile…}
Script     1.8.3                 Az.Cdn                              {Confirm-AzCdnEndpointProbeURL, Disable-AzCdnCustomDomain, Disable-AzCdnCustomDomainHttps, Enable-AzCdnCustomDomai… 
Script     1.1.0                 Az.CloudService                     {Get-AzCloudService, Get-AzCloudServiceInstanceView, Get-AzCloudServiceNetworkInterface, Get-AzCloudServiceOSFamil… 
Script     1.11.0                Az.CognitiveServices                {Add-AzCognitiveServicesAccountNetworkRule, Get-AzCognitiveServicesAccount, Get-AzCognitiveServicesAccountCommitme… 
Script     4.26.0                Az.Compute                          {Add-AzImageDataDisk, Add-AzVhd, Add-AzVMAdditionalUnattendContent, Add-AzVMDataDisk…}
Script     3.1.0                 Az.ContainerInstance                {Add-AzContainerInstanceOutput, Get-AzContainerGroup, Get-AzContainerInstanceCachedImage, Get-AzContainerInstanceC… 
Script     2.2.3                 Az.ContainerRegistry                {Connect-AzContainerRegistry, Get-AzContainerRegistry, Get-AzContainerRegistryCredential, Get-AzContainerRegistryM…
Script     1.7.0                 Az.CosmosDB                         {Get-AzCosmosDBAccount, Get-AzCosmosDBAccountKey, Get-AzCosmosDBCassandraKeyspace, Get-AzCosmosDBCassandraKeyspace… 
Script     1.1.0                 Az.DataBoxEdge                      {Get-AzDataBoxEdgeBandwidthSchedule, Get-AzDataBoxEdgeDevice, Get-AzDataBoxEdgeJob, Get-AzDataBoxEdgeOrder…}        
Script     1.2.0                 Az.Databricks                       {Get-AzDatabricksOutboundNetworkDependenciesEndpoint, Get-AzDatabricksVNetPeering, Get-AzDatabricksWorkspace, New-… 
Script     1.16.6                Az.DataFactory                      {Add-AzDataFactoryV2DataFlowDebugSessionPackage, Add-AzDataFactoryV2TriggerSubscription, Get-AzDataFactory, Get-Az… 
Script     1.0.2                 Az.DataLakeAnalytics                {Add-AzDataLakeAnalyticsDataSource, Add-AzDataLakeAnalyticsFirewallRule, Get-AzDataLakeAnalyticsAccount, Get-AzDat… 
Script     1.3.0                 Az.DataLakeStore                    {Add-AzDataLakeStoreFirewallRule, Add-AzDataLakeStoreItemContent, Add-AzDataLakeStoreTrustedIdProvider, Add-AzData… 
Script     1.0.1                 Az.DataShare                        {Get-AzDataShare, Get-AzDataShareAccount, Get-AzDataShareDataSet, Get-AzDataShareDataSetMapping…}
Script     1.1.0                 Az.DeploymentManager                {Get-AzDeploymentManagerArtifactSource, Get-AzDeploymentManagerRollout, Get-AzDeploymentManagerService, Get-AzDepl… 
Script     3.1.0                 Az.DesktopVirtualization            {Disconnect-AzWvdUserSession, Expand-AzWvdMsixImage, Get-AzWvdApplication, Get-AzWvdApplicationGroup…}
Script     1.0.2                 Az.DevTestLabs                      {Get-AzDtlAllowedVMSizesPolicy, Get-AzDtlAutoShutdownPolicy, Get-AzDtlAutoStartPolicy, Get-AzDtlVMsPerLabPolicy…}   
Script     1.1.2                 Az.Dns                              {Add-AzDnsRecordConfig, Get-AzDnsRecordSet, Get-AzDnsZone, New-AzDnsRecordConfig…}
Script     1.3.0                 Az.EventGrid                        {Get-AzEventGridDomain, Get-AzEventGridDomainKey, Get-AzEventGridDomainTopic, Get-AzEventGridSubscription…}
Script     1.11.1                Az.EventHub                         {Add-AzEventHubIPRule, Add-AzEventHubVirtualNetworkRule, Get-AzEventHub, Get-AzEventHubAuthorizationRule…}
Script     1.9.0                 Az.FrontDoor                        {Disable-AzFrontDoorCustomDomainHttps, Enable-AzFrontDoorCustomDomainHttps, Get-AzFrontDoor, Get-AzFrontDoorFronte… 
Script     4.0.2                 Az.Functions                        {Get-AzFunctionApp, Get-AzFunctionAppAvailableLocation, Get-AzFunctionAppPlan, Get-AzFunctionAppSetting…}
Script     5.0.1                 Az.HDInsight                        {Add-AzHDInsightClusterIdentity, Add-AzHDInsightComponentVersion, Add-AzHDInsightConfigValue, Add-AzHDInsightMetas… 
Script     1.3.2                 Az.HealthcareApis                   {Get-AzHealthcareApisService, New-AzHealthcareApisService, Remove-AzHealthcareApisService, Set-AzHealthcareApisSer… 
Script     2.7.4                 Az.IotHub                           {Add-AzIotHubCertificate, Add-AzIotHubConfiguration, Add-AzIotHubDeployment, Add-AzIotHubDevice…}
Script     4.4.0                 Az.KeyVault                         {Add-AzKeyVaultCertificate, Add-AzKeyVaultCertificateContact, Add-AzKeyVaultKey, Add-AzKeyVaultManagedStorageAccou… 
Script     2.1.0                 Az.Kusto                            {Add-AzKustoClusterLanguageExtension, Add-AzKustoDatabasePrincipal, Get-AzKustoAttachedDatabaseConfiguration, Get-… 
Script     1.5.0                 Az.LogicApp                         {Get-AzIntegrationAccount, Get-AzIntegrationAccountAgreement, Get-AzIntegrationAccountAssembly, Get-AzIntegrationA… 
Script     1.1.3                 Az.MachineLearning                  {Add-AzMlWebServiceRegionalProperty, Export-AzMlWebService, Get-AzMlCommitmentAssociation, Get-AzMlCommitmentPlan…} 
Script     1.2.0                 Az.Maintenance                      {Get-AzApplyUpdate, Get-AzConfigurationAssignment, Get-AzMaintenanceConfiguration, Get-AzMaintenancePublicConfigur… 
Script     3.0.0                 Az.ManagedServices                  {Get-AzManagedServicesAssignment, Get-AzManagedServicesDefinition, Get-AzManagedServicesMarketplaceDefinition, New… 
Script     1.0.2                 Az.MarketplaceOrdering              {Get-AzMarketplaceTerms, Set-AzMarketplaceTerms}
Script     1.1.1                 Az.Media                            {Get-AzMediaService, Get-AzMediaServiceKey, Get-AzMediaServiceNameAvailability, New-AzMediaService…}
Script     1.1.2                 Az.Migrate                          {Get-AzMigrateDiscoveredServer, Get-AzMigrateJob, Get-AzMigrateProject, Get-AzMigrateReplicationEligibilityResult…} 
Script     3.0.1                 Az.Monitor                          {Add-AzAutoscaleSetting, Add-AzLogProfile, Add-AzMetricAlertRule, Add-AzMetricAlertRuleV2…}
Script     1.0.0                 Az.MySql                            {Get-AzMySqlConfiguration, Get-AzMySqlConnectionString, Get-AzMySqlFirewallRule, Get-AzMySqlFlexibleServer…}        
Script     4.16.1                Az.Network                          {Add-AzApplicationGatewayAuthenticationCertificate, Add-AzApplicationGatewayBackendAddressPool, Add-AzApplicationG…
Script     1.1.1                 Az.NotificationHubs                 {Get-AzNotificationHub, Get-AzNotificationHubAuthorizationRule, Get-AzNotificationHubListKey, Get-AzNotificationHu… 
Script     3.1.0                 Az.OperationalInsights              {Disable-AzOperationalInsightsIISLogCollection, Disable-AzOperationalInsightsLinuxCustomLogCollection, Disable-AzO… 
Script     1.5.0                 Az.PolicyInsights                   {Get-AzPolicyEvent, Get-AzPolicyMetadata, Get-AzPolicyRemediation, Get-AzPolicyState…}
Script     1.1.0                 Az.PostgreSql                       {Get-AzPostgreSqlConfiguration, Get-AzPostgreSqlConnectionString, Get-AzPostgreSqlFirewallRule, Get-AzPostgreSqlFl… 
Script     1.1.2                 Az.PowerBIEmbedded                  {Get-AzPowerBIEmbeddedCapacity, Get-AzPowerBIWorkspace, Get-AzPowerBIWorkspaceCollection, Get-AzPowerBIWorkspaceCo… 
Script     1.0.3                 Az.PrivateDns                       {Add-AzPrivateDnsRecordConfig, Get-AzPrivateDnsRecordSet, Get-AzPrivateDnsVirtualNetworkLink, Get-AzPrivateDnsZone… 
Script     5.3.1                 Az.RecoveryServices                 {Add-AzRecoveryServicesAsrReplicationProtectedItemDisk, Backup-AzRecoveryServicesBackupItem, Copy-AzRecoveryServic… 
Script     1.6.0                 Az.RedisCache                       {Export-AzRedisCache, Get-AzRedisCache, Get-AzRedisCacheFirewallRule, Get-AzRedisCacheKey…}
Script     1.0.0                 Az.RedisEnterpriseCache             {Export-AzRedisEnterpriseCache, Get-AzRedisEnterpriseCache, Get-AzRedisEnterpriseCacheDatabase, Get-AzRedisEnterpr… 
Script     1.0.3                 Az.Relay                            {Get-AzRelayAuthorizationRule, Get-AzRelayHybridConnection, Get-AzRelayKey, Get-AzRelayNamespace…}
Script     1.1.0                 Az.ResourceMover                    {Add-AzResourceMoverMoveResource, Get-AzResourceMoverMoveCollection, Get-AzResourceMoverMoveResource, Get-AzResour… 
Script     5.6.0                 Az.Resources                        {Export-AzResourceGroup, Export-AzTemplateSpec, Get-AzDenyAssignment, Get-AzDeployment…}
Script     1.2.0                 Az.Security                         {Add-AzSecurityAdaptiveNetworkHardening, Add-AzSecuritySqlVulnerabilityAssessmentBaseline, Confirm-AzSecurityAutom… 
Script     1.1.0                 Az.SecurityInsights                 {Get-AzSentinelAlertRule, Get-AzSentinelAlertRuleAction, Get-AzSentinelAlertRuleTemplate, Get-AzSentinelBookmark…}  
Script     1.8.1                 Az.ServiceBus                       {Add-AzServiceBusIPRule, Add-AzServiceBusVirtualNetworkRule, Complete-AzServiceBusMigration, Get-AzServiceBusAutho… 
Script     3.0.2                 Az.ServiceFabric                    {Add-AzServiceFabricClientCertificate, Add-AzServiceFabricManagedClusterClientCertificate, Add-AzServiceFabricMana… 
Script     1.4.1                 Az.SignalR                          {Get-AzSignalR, Get-AzSignalRKey, Get-AzSignalRUsage, New-AzSignalR…}
Script     3.8.0                 Az.Sql                              {Add-AzSqlDatabaseToFailoverGroup, Add-AzSqlElasticJobStep, Add-AzSqlElasticJobTarget, Add-AzSqlInstanceKeyVaultKe… 
Script     1.1.0                 Az.SqlVirtualMachine                {Get-AzAvailabilityGroupListener, Get-AzSqlVM, Get-AzSqlVMGroup, New-AzAvailabilityGroupListener…}
Script     1.1.1                 Az.StackHCI                         {Add-AzStackHCIVMAttestation, Disable-AzStackHCIAttestation, Disable-AzStackHCIRemoteSupport, Enable-AzStackHCIAtt… 
Script     4.5.0                 Az.Storage                          {Add-AzRmStorageContainerLegalHold, Add-AzStorageAccountManagementPolicyAction, Add-AzStorageAccountNetworkRule, C… 
Script     1.7.0                 Az.StorageSync                      {Get-AzStorageSyncCloudEndpoint, Get-AzStorageSyncGroup, Get-AzStorageSyncServer, Get-AzStorageSyncServerEndpoint…} 
Script     2.0.0                 Az.StreamAnalytics                  {Get-AzStreamAnalyticsCluster, Get-AzStreamAnalyticsClusterStreamingJob, Get-AzStreamAnalyticsDefaultFunctionDefin… 
Script     1.0.0                 Az.Support                          {Get-AzSupportProblemClassification, Get-AzSupportService, Get-AzSupportTicket, Get-AzSupportTicketCommunication…}  
Script     1.3.0                 Az.Synapse                          {Add-AzSynapseDataFlowDebugSessionPackage, Add-AzSynapseTriggerSubscription, Clear-AzSynapseSqlPoolVulnerabilityAs… 
Script     1.1.0                 Az.TrafficManager                   {Add-AzTrafficManagerCustomHeaderToEndpoint, Add-AzTrafficManagerCustomHeaderToProfile, Add-AzTrafficManagerEndpoi… 
Script     2.11.1                Az.Websites                         {Add-AzWebAppAccessRestrictionRule, Add-AzWebAppTrafficRouting, Edit-AzWebAppBackupConfiguration, Enter-AzWebAppCo…

Error output

Invoke-AzAksRunCommand: C:\...\Invoke-CommandOnAksCluster.ps1:36:21
Line |
  36 |  … te:result = Invoke-AzAksRunCommand -ResourceGroupName $ResourceGroupN …
     |                ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | Long running operation failed with status 'Failed'.

[dingmeng-xue]

@wyunchi-ms , could you check this cmdlet? We need to provide more information based on cmdlet logic.

[wyunchi-ms]

Hi @fschmied , the timeout of Invoke-AzAksRunCommand is controlled by service side and cannot set by client side. The current value is 600s.

[fschmied]

@wyunchi-ms Okay, thanks for the information, then I will just ensure my own script finishes (sufficiently) before 600 seconds have passed.

In general, is there any way to obtain the log output in case the operation is aborted due to a timeout?