DscConfigurationClient.GetContent is broken since `2020-01-13-preview/automation` API version

[dz-sourced]

Bug Report

Summary: When trying to read back a DSC Configuration content using func (client DscConfigurationClient) GetContent an error happens in the version of the API 2020-01-13-preview/automation and later while trying to unmarshall DSC Configuration content from JSON. DSC Configurations are not valid JSON objects and thus unmarshalling fails. This issue was not happening in 2018-06-30-preview/automation and earlier, where DSC Configuration content was treated as a plain string value. I have confirmed the issue on the main branch.

Details: In the 2020-01-13-preview/automation the actual failure seems to happen when trying to unmarshall DSC Configuration content from JSON: https://github.com/Azure/azure-sdk-for-go/blob/94316bad34922dee81518b17ddb1454d7a5f89c8/services/preview/automation/mgmt/2020-01-13-preview/automation/dscconfiguration.go#L384

The error that I get when trying to read back a sample DSC Configuration like Configuration test {} is:

automation.DscConfigurationClient#GetContent: Failure responding to request: StatusCode=200 -- Original Error: Error occurred unmarshalling JSON - Error = 'invalid character 'C' looking for beginning of value' JSON = 'Configuration test{}'

This was not happening in the previous version of the API 2018-06-30-preview/automation and the content value was just passed as is: https://github.com/Azure/azure-sdk-for-go/blob/94316bad34922dee81518b17ddb1454d7a5f89c8/services/preview/automation/mgmt/2018-06-30-preview/automation/dscconfiguration.go#L383

Finally, the root cause can be traced to the change in the Swagger API definitions. The latest version of the GetContent API defines the output as a string type, which I assume is tried to be de-serialized from JSON by default: https://github.com/Azure/azure-rest-api-specs/blob/3034ada77d465b317cda51250a92454824cca06b/specification/automation/resource-manager/Microsoft.Automation/stable/2019-06-01/dscConfiguration.json#L309

In the earlier version of the API spec the output type is set to file: https://github.com/Azure/azure-rest-api-specs/blob/3026119ab41bbce77275cfa3a1afbabf43af5aea/specification/automation/resource-manager/Microsoft.Automation/stable/2015-10-31/dscConfiguration.json#L309

I don't know whether this was intentional API spec change, or whether there is a need to provide further type modifiers in the Swagger API or whether it's an issue with the SDK codegen tool.

Here is a small script that demostrates the problem https://gist.github.com/dz-sourced/11654206078fcc159c54564de6758659 It creates a resource group, an automation account, a sample empty DSC Configuration and the tries to read it using older and new API versions. You should be able to see that read is succesful in 2018-06-30-preview/automation and fails in the 2020-01-13-preview/ version.

[ArcturusZhang]

Hi @dz-sourced thank you for this issue!

By looking at the error message, it is very likely to be an issue that the service is returning something uncompatible with its swagger. But could you please paste the request log here so that we could confirm where does this issue come from? You could get the request logs by exporting an env var AZURE_GO_SDK_LOG_FILE=DEBUG

[dz-sourced]

Hey @ArcturusZhang, do I need to initialize the logger in any way? I set then env variable as you asked, but I don't see any debug logs. You can see the code I'm using here https://gist.github.com/dz-sourced/11654206078fcc159c54564de6758659

[dz-sourced]

Actually, I have figured it out, need to set AZURE_SDK_GO_LOG_LEVEL=DEBUG first. @ArcturusZhang you can find the log here https://gist.github.com/dz-sourced/050d6b14a80c231a13840171dcfd7a6b I have redacted the subscription id.

[msftbot[bot]]

Thank you for your feedback. This has been routed to the support team for assistance.

[ArcturusZhang]

Hey @dz-sourced based on the information you shared, it is pretty sure that this issue was introduced by the swagger changes - therefore I added some labels to loop the correct persons to answer your questions

[msftbot[bot]]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @zjalexander.

Issue Details

---

Bug Report

Summary: When trying to read back a DSC Configuration content using func (client DscConfigurationClient) GetContent an error happens in the version of the API 2020-01-13-preview/automation and later while trying to unmarshall DSC Configuration content from JSON. DSC Configurations are not valid JSON objects and thus unmarshalling fails. This issue was not happening in 2018-06-30-preview/automation and earlier, where DSC Configuration content was treated as a plain string value. I have confirmed the issue on the main branch.

Details: In the 2020-01-13-preview/automation the actual failure seems to happen when trying to unmarshall DSC Configuration content from JSON: https://github.com/Azure/azure-sdk-for-go/blob/94316bad34922dee81518b17ddb1454d7a5f89c8/services/preview/automation/mgmt/2020-01-13-preview/automation/dscconfiguration.go#L384

The error that I get when trying to read back a sample DSC Configuration like Configuration test {} is:

automation.DscConfigurationClient#GetContent: Failure responding to request: StatusCode=200 -- Original Error: Error occurred unmarshalling JSON - Error = 'invalid character 'C' looking for beginning of value' JSON = 'Configuration test{}'

This was not happening in the previous version of the API 2018-06-30-preview/automation and the content value was just passed as is: https://github.com/Azure/azure-sdk-for-go/blob/94316bad34922dee81518b17ddb1454d7a5f89c8/services/preview/automation/mgmt/2018-06-30-preview/automation/dscconfiguration.go#L383

Finally, the root cause can be traced to the change in the Swagger API definitions. The latest version of the GetContent API defines the output as a string type, which I assume is tried to be de-serialized from JSON by default: https://github.com/Azure/azure-rest-api-specs/blob/3034ada77d465b317cda51250a92454824cca06b/specification/automation/resource-manager/Microsoft.Automation/stable/2019-06-01/dscConfiguration.json#L309

In the earlier version of the API spec the output type is set to file: https://github.com/Azure/azure-rest-api-specs/blob/3026119ab41bbce77275cfa3a1afbabf43af5aea/specification/automation/resource-manager/Microsoft.Automation/stable/2015-10-31/dscConfiguration.json#L309

I don't know whether this was intentional API spec change, or whether there is a need to provide further type modifiers in the Swagger API or whether it's an issue with the SDK codegen tool.

Here is a small script that demostrates the problem https://gist.github.com/dz-sourced/11654206078fcc159c54564de6758659 It creates a resource group, an automation account, a sample empty DSC Configuration and the tries to read it using older and new API versions. You should be able to see that read is succesful in 2018-06-30-preview/automation and fails in the 2020-01-13-preview/ version.

Author:	dz-sourced
Assignees:	ArcturusZhang
Labels:	bug, Automation, Service Attention, Mgmt, customer-reported, needs-team-attention
Milestone:	-

[tombuildsstuff]

ping @zjalexander

[zjalexander]

I'm not the owner, but I've reached out to the responsible team to 1) get a response to this issue and 2) get my name taken off the owner list for this service.

[ralfwanninkhof]

@zjalexander Did you got any response? We would love to have this issue fixed, is there anything we can do to help?

[zjalexander]

Thank you for the ping! I did get a response, but it needs some follow up on the correct owner's side. I have reached out to them again to get action taken.

[wuxu92]

@zjalexander hi is there any update on this issue? looking forward to having this issue fixed asap.

[zjalexander]

I have forwarded your comment to the Automation service owners.

[wuxu92]

there is another similar break change in the runbook.GetCotent from 2018-06-30 to 2019-06-01. could you please forward this to the service team too? @zjalexander or should I raise another issue to track this

https://github.com/Azure/azure-rest-api-specs/blob/55458b642a7cdc8dd9b2928942d13c6f076399bc/specification/automation/resource-manager/Microsoft.Automation/stable/2018-06-30/runbook.json#L82

        "responses": {
          "200": {
            "description": "OK",
            "schema": {
              "type": "file"  <-- file
            }

==>

https://github.com/Azure/azure-rest-api-specs/blob/55458b642a7cdc8dd9b2928942d13c6f076399bc/specification/automation/resource-manager/Microsoft.Automation/stable/2019-06-01/runbook.json#L82

        "responses": {
          "200": {
            "description": "OK",
            "schema": {
              "type": "string"  <-- string
            }
          },

[tadelesh]

@dz-sourced We've depreacated services/preview/automation/mgmt. A replacement package is available github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/automation/armautomation. See Migration Guide for guidance on upgrading. And in the new version, the response is desterilized correctly.

[wuxu92]

but we can not migrate to the new SDK package in the short term(or even in the long term)

[lirenhe]

Please consider the migration and we could help if you have any specific asks. We already stopped regular release of old version of the SDKs. cc @xboxeer

[tombuildsstuff]

@wuxu92 since this is related to a Terraform issue, this method is now available in hashicorp/go-azure-sdk - so it'd be worth confirming if that works for this issue.

[wuxu92]

@tombuildsstuff hashicorp/go-azure-sdk generated the same code by autorest, so the line autorest.ByUnmarshallingJSON(&result.Model), will always cause an error.