azure-powershell icon indicating copy to clipboard operation
azure-powershell copied to clipboard

Remove-AzConnectedMachineExtension -asjob not working as expected

Open pejuborg opened this issue 3 years ago • 5 comments

Description

When running these commands everything works as expected. Get-AzConnectedMachine -ResourceGroupName Arc | % { Get-AzConnectedMachineExtension -ResourceGroupName $_.ResourceGroupName -MachineName $_.Name | Remove-AzConnectedMachineExtension }

When trying to invoke the same, but adding the switch -asjob to the Remove-AzConnectedMachineExtension command Get-AzConnectedMachine -ResourceGroupName Arc | % { Get-AzConnectedMachineExtension -ResourceGroupName $_.ResourceGroupName -MachineName $_.Name | Remove-AzConnectedMachineExtension -AsJob }

The correct number of jobs are started but all of them fail, giving me the message: NullReferenceException: Object reference not set to an instance of an object.

Is this a bug or by design?

If by design, how do I work around it?

Issue script & Debug output

Get-AzConnectedMachine -ResourceGroupName Arc -Name DC3-AV71 | % { Get-AzConnectedMachineExtension -ResourceGroupName $_.ResourceGroupName -MachineName $_.Name | Remove-AzConnectedMachineExtension -AsJob }
DEBUG: 13:10:43 - GetAzureRMContextCommand end processing.
DEBUG: [CmdletBeginProcessing]: Starting command
DEBUG: CmdletBeginProcessing:
DEBUG: CmdletProcessRecordStart:
DEBUG: CmdletGetPipeline:
DEBUG: CmdletBeforeAPICall:
DEBUG: URLCreated: /subscriptions/49248c33-1747-4f1d-8666-84cebd2e5859/resourceGroups/Arc/providers/Microsoft.HybridCompute/machines/DC3-AV71?api-version=2022-03-10
DEBUG: RequestCreated: /subscriptions/49248c33-1747-4f1d-8666-84cebd2e5859/resourceGroups/Arc/providers/Microsoft.HybridCompute/machines/DC3-AV71?api-version=2022-03-10
DEBUG: HeaderParametersAdded:
DEBUG: ============================ HTTP REQUEST ============================

HTTP Method:
GET

Absolute Uri:
https://management.azure.com/subscriptions/49248c33-1747-4f1d-8666-84cebd2e5859/resourceGroups/Arc/providers/Microsoft.HybridCompute/machines/DC3-AV71?api-version=2022-03-10

Headers:
x-ms-unique-id                : 52
x-ms-client-request-id        : a18ff713-fed4-4f3e-82b9-68b39c90ea0b
CommandName                   : Get-AzConnectedMachine
FullCommandName               : Get-AzConnectedMachine_Get
ParameterSetName              : __AllParameterSets
User-Agent                    : AzurePowershell/v8.2.0,PSVersion/v7.2.6,Az.ConnectedMachine/0.4.0

Body:



DEBUG: BeforeCall:
DEBUG: ============================ HTTP RESPONSE ============================

Status Code:
OK

Headers:
Cache-Control                 : no-cache
Pragma                        : no-cache
x-ms-ratelimit-remaining-subscription-reads: 11999
Strict-Transport-Security     : max-age=31536000; includeSubDomains
x-ms-request-id               : 16a78f3b-7992-464a-be2d-33e94f0f3552
Server                        : Kestrel
x-ms-correlation-request-id   : e2f83b0b-1aa2-4409-99f7-985d72230890
x-ms-routing-request-id       : UKSOUTH:20220905T111039Z:e2f83b0b-1aa2-4409-99f7-985d72230890
X-Content-Type-Options        : nosniff
Date                          : Mon, 05 Sep 2022 11:10:39 GMT

Body:
{
  "id": "/subscriptions/49248c33-1747-4f1d-8666-84cebd2e5859/resourceGroups/Arc/providers/Microsoft.HybridCompute/machines/DC3-AV71",
  "name": "DC3-AV71",
  "location": "westeurope",
  "tags": {
    "City": "Skogas",
    "CountryOrRegion": "Sweden",
    "Datacenter": "AV71",
    "StateOrDistrict": "Stockholm"
  },
  "identity": {
    "type": "SystemAssigned",
    "principalId": "6896d3fd-51da-46c8-985f-008bcdf2568a",
    "tenantId": "abfbff15-0c3b-4e9c-93d1-a3c38d1acbe4"
  },
  "type": "Microsoft.HybridCompute/machines",
  "properties": {
    "provisioningState": "Succeeded",
    "agentVersion": "1.21.02043.412",
    "status": "Connected",
    "lastStatusChange": "2022-09-05T10:43:03.3753641Z",
    "errorDetails": [],
    "displayName": "DC3-AV71",
    "machineFqdn": "DC3-AV71",
    "osName": "windows",
    "osVersion": "10.0.19042.1889",
    "osType": "windows",
    "osProfile": {
      "computerName": "DC3-AV71"
    },
    "vmId": "7aedbaa8-f72e-4de2-b20b-8d3c5f0fea72",
    "vmUuid": "EE35695D-518E-4C49-BE7A-07A0422384EA",
    "clientPublicKey": "MIIBCgKCAQEA3dEJ8MZlfMbJefJbdRLFQt8mPDFWwOlIRdEhV04ur0cCypPFN4vLfD4jIJCRlj39+Vy+eaZsujm0r+tl5hM3EjRJf1ehLKVxK/e7s3qC446aHEODOj6XJUw+EmpFX0A/54qoGlCBWcsSFEU/Ctqjibg9Roi5f116+efCiiLMEdvhUbDHiqBTAtBRBkGUFzCFNSqhSsUEevAeOt4tllQBrb59r+MdaYD3kysBsN9PvwznZ04tojM/blVilMh2V/HY7Z056HPfDm+8UnkAnmRYdH4vb+n87pBESib74jo3b4m1LEKw+Ey781HXWQ1ckuIxd9+ohg6fRQr+r+grja9P1wIDAQAB",
    "osSku": "Windows Server Standard",
    "domainName": "internal.juborg.nu",
    "adFqdn": "DC3-AV71.internal.juborg.nu",
    "dnsFqdn": "internal.juborg.nu.DC3-AV71",
    "mssqlDiscovered": "false",
    "cloudMetadata": {
      "provider": "N/A"
    },
    "detectedProperties": {
      "cloudprovider": "N/A",
      "logicalCoreCount": "2",
      "manufacturer": "Microsoft Corporation",
      "model": "Virtual Machine",
      "mssqldiscovered": "false"
    },
    "agentConfiguration": {
      "proxyUrl": "",
      "incomingConnectionsPorts": [],
      "extensionsAllowList": [],
      "extensionsBlockList": [],
      "proxyBypass": [],
      "extensionsEnabled": "true",
      "guestConfigurationEnabled": "true"
    },
    "serviceStatuses": {
      "extensionService": {
        "status": "running",
        "startupType": "automatic"
      },
      "guestConfigurationService": {
        "status": "running",
        "startupType": "automatic"
      }
    }
  },
  "resources": [
    {
      "id": "/subscriptions/49248c33-1747-4f1d-8666-84cebd2e5859/resourceGroups/Arc/providers/Microsoft.HybridCompute/machines/DC3-AV71/extensions/AzureMonitorWindowsAgent",
      "name": "AzureMonitorWindowsAgent",
      "type": "Microsoft.HybridCompute/machines/extensions",
      "location": "westeurope",
      "properties": {
        "publisher": "Microsoft.Azure.Monitor",
        "type": "AzureMonitorWindowsAgent",
        "typeHandlerVersion": "1.1.5.1",
        "autoUpgradeMinorVersion": false,
        "enableAutomaticUpgrade": true,
        "provisioningState": "Succeeded",
        "instanceView": {
          "name": "AzureMonitorWindowsAgent",
          "type": "AzureMonitorWindowsAgent",
          "typeHandlerVersion": "1.1.5.1",
          "status": {
            "code": "success",
            "level": "Information",
            "message": "Extension Message: ExtensionOperation:enable. Status:Success"
          }
        }
      }
    },
    {
      "id": "/subscriptions/49248c33-1747-4f1d-8666-84cebd2e5859/resourceGroups/Arc/providers/Microsoft.HybridCompute/machines/DC3-AV71/extensions/MDE.Windows",
      "name": "MDE.Windows",
      "type": "Microsoft.HybridCompute/machines/extensions",
      "location": "westeurope",
      "properties": {
        "publisher": "Microsoft.Azure.AzureDefenderForServers",
        "type": "MDE.Windows",
        "typeHandlerVersion": "1.0.7.0",
        "autoUpgradeMinorVersion": false,
        "enableAutomaticUpgrade": true,
        "settings": {
          "azureResourceId": "/subscriptions/49248c33-1747-4f1d-8666-84cebd2e5859/resourceGroups/Arc/providers/Microsoft.HybridCompute/machines/DC3-AV71",
          "forceReOnboarding": false,
          "vNextEnabled": true
        },
        "provisioningState": "Failed",
        "instanceView": {
          "name": "MDE.Windows",
          "type": "MDE.Windows",
          "typeHandlerVersion": "1.0.7.0",
          "status": {
            "code": "error",
            "level": "Error",
            "message": "Extension Message: Failed to configure Microsoft Defender for Endpoint: Onboarding script failed with error code 15, executionlog: [2022-09-05 08:52:30Z][Information] OS Name: Microsoft Windows Server Standard\r\n[2022-09-05 08:52:30Z][Information] Product type: 2\r\n[2022-09-05 08:52:30Z][Information] OperatingSystem SKU: 146\r\n[2022-09-05 08:52:30Z][Information] MDE onboarding package applicability: 2\r\n[2022-09-05 08:52:30Z][Information] Decoding onboarding script from base64 string\r\n[2022-09-05 08:52:30Z][Information] Running onboarding package\r\n[2022-09-05 08:52:31Z][Information] Successfully started process, waiting to finish with timeout\r\n[2022-09-05 08:53:03Z][Error] Onboarding script failed with error code 15 \r\n[2022-09-05 08:53:03Z][Error] Failed to configure Microsoft Defender for Endpoint: Onboarding script failed with error code 15\r\n[2022-09-05 08:53:04Z][Information] Set handler status (C:\\Packages\\Plugins\\Microsoft.Azure.AzureDefenderForServers.MDE.Windows\\1.0.7.0\\status\\0.status), Status=error, Code=1, Message='Failed to configure Microsoft Defender for Endpoint: Onboarding script failed with error code 15'"
          }
        }
      }
    }
  ]
}


DEBUG: ResponseCreated:
DEBUG: BeforeResponseDispatch:
DEBUG: 13:10:43 - GetAzureRMContextCommand end processing.
DEBUG: [CmdletBeginProcessing]: Starting command
DEBUG: CmdletBeginProcessing:
DEBUG: [CmdletBeginProcessing]: Starting command
DEBUG: CmdletBeginProcessing:
DEBUG: CmdletProcessRecordStart:
DEBUG: CmdletGetPipeline:
DEBUG: CmdletBeforeAPICall:
DEBUG: URLCreated: /subscriptions/49248c33-1747-4f1d-8666-84cebd2e5859/resourceGroups/Arc/providers/Microsoft.HybridCompute/machines/DC3-AV71/extensions?api-version=2022-03-10
DEBUG: RequestCreated: /subscriptions/49248c33-1747-4f1d-8666-84cebd2e5859/resourceGroups/Arc/providers/Microsoft.HybridCompute/machines/DC3-AV71/extensions?api-version=2022-03-10
DEBUG: HeaderParametersAdded:
DEBUG: ============================ HTTP REQUEST ============================

HTTP Method:
GET

Absolute Uri:
https://management.azure.com/subscriptions/49248c33-1747-4f1d-8666-84cebd2e5859/resourceGroups/Arc/providers/Microsoft.HybridCompute/machines/DC3-AV71/extensions?api-version=2022-03-10

Headers:
x-ms-unique-id                : 53
x-ms-client-request-id        : 456d2907-868d-42a6-bffa-62352d8df7f9
CommandName                   : Get-AzConnectedMachineExtension
FullCommandName               : Get-AzConnectedMachineExtension_List
ParameterSetName              : __AllParameterSets
User-Agent                    : AzurePowershell/v8.2.0,PSVersion/v7.2.6,Az.ConnectedMachine/0.4.0

Body:



DEBUG: BeforeCall:
DEBUG: ============================ HTTP RESPONSE ============================

Status Code:
OK

Headers:
Cache-Control                 : no-cache
Pragma                        : no-cache
x-ms-ratelimit-remaining-subscription-reads: 11998
Strict-Transport-Security     : max-age=31536000; includeSubDomains
x-ms-request-id               : b20f8dd4-9521-43e0-86dc-76dd04c8c7f3
Server                        : Kestrel
x-ms-correlation-request-id   : 395cf18c-8319-489a-be53-ba613f04e634
x-ms-routing-request-id       : UKSOUTH:20220905T111039Z:395cf18c-8319-489a-be53-ba613f04e634
X-Content-Type-Options        : nosniff
Date                          : Mon, 05 Sep 2022 11:10:39 GMT

Body:
{
  "value": [
    {
      "id": "/subscriptions/49248c33-1747-4f1d-8666-84cebd2e5859/resourceGroups/Arc/providers/Microsoft.HybridCompute/machines/DC3-AV71/extensions/AzureMonitorWindowsAgent",
      "name": "AzureMonitorWindowsAgent",
      "location": "westeurope",
      "type": "Microsoft.HybridCompute/machines/extensions",
      "properties": {
        "publisher": "Microsoft.Azure.Monitor",
        "type": "AzureMonitorWindowsAgent",
        "typeHandlerVersion": "1.1.5.1",
        "autoUpgradeMinorVersion": false,
        "enableAutomaticUpgrade": true,
        "provisioningState": "Succeeded",
        "instanceView": {
          "name": "AzureMonitorWindowsAgent",
          "type": "AzureMonitorWindowsAgent",
          "typeHandlerVersion": "1.1.5.1",
          "status": {
            "code": "success",
            "level": "Information",
            "message": "Extension Message: ExtensionOperation:enable. Status:Success"
          }
        }
      }
    },
    {
      "id": "/subscriptions/49248c33-1747-4f1d-8666-84cebd2e5859/resourceGroups/Arc/providers/Microsoft.HybridCompute/machines/DC3-AV71/extensions/MDE.Windows",
      "name": "MDE.Windows",
      "location": "westeurope",
      "type": "Microsoft.HybridCompute/machines/extensions",
      "properties": {
        "publisher": "Microsoft.Azure.AzureDefenderForServers",
        "type": "MDE.Windows",
        "typeHandlerVersion": "1.0.7.0",
        "autoUpgradeMinorVersion": false,
        "enableAutomaticUpgrade": true,
        "settings": {
          "azureResourceId": "/subscriptions/49248c33-1747-4f1d-8666-84cebd2e5859/resourceGroups/Arc/providers/Microsoft.HybridCompute/machines/DC3-AV71",
          "forceReOnboarding": false,
          "vNextEnabled": true
        },
        "provisioningState": "Failed",
        "instanceView": {
          "name": "MDE.Windows",
          "type": "MDE.Windows",
          "typeHandlerVersion": "1.0.7.0",
          "status": {
            "code": "error",
            "level": "Error",
            "message": "Extension Message: Failed to configure Microsoft Defender for Endpoint: Onboarding script failed with error code 15, executionlog: [2022-09-05 08:52:30Z][Information] OS Name: Microsoft Windows Server Standard\r\n[2022-09-05 08:52:30Z][Information] Product type: 2\r\n[2022-09-05 08:52:30Z][Information] OperatingSystem SKU: 146\r\n[2022-09-05 08:52:30Z][Information] MDE onboarding package applicability: 2\r\n[2022-09-05 08:52:30Z][Information] Decoding onboarding script from base64 string\r\n[2022-09-05 08:52:30Z][Information] Running onboarding package\r\n[2022-09-05 08:52:31Z][Information] Successfully started process, waiting to finish with timeout\r\n[2022-09-05 08:53:03Z][Error] Onboarding script failed with error code 15 \r\n[2022-09-05 08:53:03Z][Error] Failed to configure Microsoft Defender for Endpoint: Onboarding script failed with error code 15\r\n[2022-09-05 08:53:04Z][Information] Set handler status (C:\\Packages\\Plugins\\Microsoft.Azure.AzureDefenderForServers.MDE.Windows\\1.0.7.0\\status\\0.status), Status=error, Code=1, Message='Failed to configure Microsoft Defender for Endpoint: Onboarding script failed with error code 15'"
          }
        }
      }
    }
  ]
}


DEBUG: ResponseCreated:
DEBUG: BeforeResponseDispatch:
DEBUG: CmdletProcessRecordStart:

Id     Name            PSJobTypeName   State         HasMoreData     Location             Command
--     ----            -------------   -----         -----------     --------             -------
10     Remove-AzConne…                 Running       True            localhost            Get-AzConnectedMachine -…
DEBUG: CmdletProcessRecordEnd:
DEBUG: CmdletProcessRecordStart:
11     Remove-AzConne…                 Failed        True            localhost            Get-AzConnectedMachine -…
DEBUG: CmdletProcessRecordEnd:
DEBUG: Finally:
DEBUG: CmdletAfterAPICall:
DEBUG: [CmdletProcessRecordAsyncEnd]: Finish HTTP process
DEBUG: CmdletProcessRecordAsyncEnd:
DEBUG: CmdletProcessRecordEnd:
DEBUG: Finally:
DEBUG: CmdletAfterAPICall:
DEBUG: [CmdletProcessRecordAsyncEnd]: Finish HTTP process
DEBUG: CmdletProcessRecordAsyncEnd:
DEBUG: CmdletProcessRecordEnd:

PS C:\> get-job 11 | Receive-Job
NullReferenceException: Object reference not set to an instance of an object.
DEBUG: CmdletGetPipeline:
DEBUG: CmdletBeforeAPICall:
DEBUG: [CmdletProcessRecordAsyncEnd]: Finish HTTP process
DEBUG: CmdletProcessRecordAsyncEnd:

Environment data

Name                           Value
----                           -----
PSVersion                      7.2.6
PSEdition                      Core
GitCommitId                    7.2.6
OS                             Microsoft Windows 10.0.22000
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

Module versions

Version    PreRelease Name
-------    ---------- ----
2.9.1                 Az.Accounts
0.4.0                 Az.ConnectedMachine

Error output

HistoryId:

Message        : Object reference not set to an instance of an object.
StackTrace     :    at Microsoft.Azure.PowerShell.Cmdlets.ConnectedMachine.Cmdlets.RemoveAzConnectedMachineExtension_De
                 leteViaIdentity.ProcessRecordAsync()
                    at Microsoft.Azure.PowerShell.Cmdlets.ConnectedMachine.Cmdlets.RemoveAzConnectedMachineExtension_De
                 leteViaIdentity.ProcessRecordAsync()
Exception      : System.NullReferenceException
InvocationInfo : {}
Line           :
Position       :
HistoryId      :

Message        : Object reference not set to an instance of an object.
StackTrace     :    at Microsoft.Azure.PowerShell.Cmdlets.ConnectedMachine.Cmdlets.RemoveAzConnectedMachineExtension_De
                 leteViaIdentity.ProcessRecordAsync()
                    at Microsoft.Azure.PowerShell.Cmdlets.ConnectedMachine.Cmdlets.RemoveAzConnectedMachineExtension_De
                 leteViaIdentity.ProcessRecordAsync()
Exception      : System.NullReferenceException
InvocationInfo : {}
Line           :
Position       :
HistoryId      :


   HistoryId: 6

Message        : The 'Get-AzConfidentialLedger' command was found in the module 'Az.ConfidentialLedger', but the module
                  could not be loaded. For more information, run 'Import-Module Az.ConfidentialLedger'.
StackTrace     :    at System.Management.Automation.ExceptionHandlingOps.CheckActionPreference(FunctionContext funcCont
                 ext, Exception exception)
                    at System.Management.Automation.Interpreter.ActionCallInstruction`2.Run(InterpretedFrame frame)
                    at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame fr
                 ame)
                    at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame fr
                 ame)
                    at System.Management.Automation.Interpreter.Interpreter.Run(InterpretedFrame frame)
                    at System.Management.Automation.Interpreter.LightLambda.RunVoid1[T0](T0 arg0)
                    at System.Management.Automation.DlrScriptCommandProcessor.RunClause(Action`1 clause, Object dollarU
                 nderbar, Object inputToProcess)
Exception      : System.Management.Automation.CommandNotFoundException
InvocationInfo : {}
Line           : Get-AzConfidentialLedger
Position       : At line:1 char:1
                 + Get-AzConfidentialLedger
                 + ~~~~~~~~~~~~~~~~~~~~~~~~
HistoryId      : 6

pejuborg avatar Sep 05 '22 11:09 pejuborg

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @rpsqrd, @edyoung.

Can you upgrade to the latest release (0.6.0) and see if this issue is still there?

yaotongms avatar Jan 02 '24 23:01 yaotongms

I have now retried the procedure, and I still get the same result.

pejuborg avatar Jan 29 '24 08:01 pejuborg

I couldn't reproduce this error. Remove-AzConnectedMachineExtension works fine for me for removing a CustomScriptExtension that I just created:

Remove-AzConnectedMachineExtension -MachineName testmachine -Name CustomScriptExtension -ResourceGroupName az-sdk-test -AsJob

image

In your error output, it seems like you are getting Null Reference Exception, and also some error related to Get-AzConfidentialLedger:

  • For the null reference error, make sure you only call Remove-AzConnectedMachineExtension with the machine name and extension name, so you are not deleting an extension that doesn't exist - although the latest command will always ask you to input those two arguments
  • For the error related to calling Get-AzConfidentialLedger that doesn't exist, make sure you don't call other commands while removing an extension, so that we can know which command is causing the error

yaotongms avatar Feb 05 '24 22:02 yaotongms

Can you also share the result when running the latest module in debug mode?

yaotongms avatar Feb 06 '24 21:02 yaotongms

@pejuborg - Please try with 0.8.0 of the Module and reactivate if you still see the issue: https://www.powershellgallery.com/packages/Az.ConnectedMachine/0.8.0

raghushantha avatar Jun 10 '24 20:06 raghushantha