azure-rest-api-specs icon indicating copy to clipboard operation
azure-rest-api-specs copied to clipboard

Azure Databricks - Managed Disk Encryption - Critical API Version Cha…

Open scottmacchia-msft opened this issue 2 years ago • 8 comments

ARM API Information (Control Plane)

MSFT employees can try out our new experience at OpenAPI Hub - one location for using our validation tools and finding your workflow. Azure 1st Party Service can try out the Shift Left experience to initiate API design review from ADO code repo. If you are interested, may request engineering support by filling in with the form https://aka.ms/ShiftLeftSupportForm.

Changelog

Add a changelog entry for this PR by answering the following questions:

  1. What's the purpose of the update?
    • [ ] new service onboarding
    • [ ] new API version
    • [ ] update existing version for new feature
    • [X] update existing version to fix swagger quality issue in s360
    • [ ] Other, please clarify
  2. When are you targeting to deploy the new service/feature to public regions? Please provide the date or, if the date is not yet available, the month.

Code changes for new feature has already been deployed to public regions earlier this week (October 24, 2022). Our team believed that these changes were previously merged to the public repo but they have not been. Additionally, we already have one major customer using the feature.

  1. When do you expect to publish the swagger? Please provide date or, the the date is not yet available, the month.

October 2022

  1. By default, Azure SDKs of all languages (.NET/Python/Java/JavaScript for both management-plane SDK and data-plane SDK, Go for management-plane SDK only ) MUST be refreshed with/after swagger of new version is published. If you prefer NOT to refresh any specific SDK language upon swagger updates in the current PR, please leave details with justification here.

Contribution checklist (MS Employees Only):

If any further question about AME onboarding or validation tools, please view the FAQ.

ARM API Review Checklist

Applicability: :warning:

If your changes encompass only the following scenarios, you should SKIP this section, as these scenarios do not require ARM review.

  • Change to data plane APIs
  • Adding new properties
  • All removals

Otherwise your PR may be subject to ARM review requirements. Complete the following:

  • [] Check this box if any of the following apply to the PR so that the label "ARMReview" and "WaitForARMFeedback" will be added by bot to kick off ARM API Review. Missing to check this box in the following scenario may result in delays to the ARM manifest review and deployment.

    • Adding a new service
    • Adding new API(s)
    • Adding a new API version -[ ] To review changes efficiently, ensure you copy the existing version into the new directory structure for first commit and then push new changes, including version updates, in separate commits. You can use OpenAPIHub to initialize the PR for adding a new version. For more details refer to the wiki.
  • [X] Ensure you've reviewed following guidelines including ARM resource provider contract and REST guidelines. Estimated time (4 hours). This is required before you can request review from ARM API Review board.

  • [ ] If you are blocked on ARM review and want to get the PR merged with urgency, please get the ARM oncall for reviews (RP Manifest Approvers team under Azure Resource Manager service) from IcM and reach out to them.

Breaking Change Review Checklist

If you have any breaking changes as defined in the Breaking Change Policy, request approval from the Breaking Change Review Board.

Action: to initiate an evaluation of the breaking change, create a new intake using the template for breaking changes. Additional details on the process and office hours are on the Breaking Change Wiki.

NOTE: To update API(s) in public preview for over 1 year (refer to Retirement of Previews)

Please follow the link to find more details on PR review process.

scottmacchia-msft avatar Oct 25 '22 19:10 scottmacchia-msft

Hi, @scottmacchia-msft Thanks for your PR. I am workflow bot for review process. Here are some small tips.

  • Please ensure to do self-check against checklists in first PR comment.
  • PR assignee is the person auto-assigned and responsible for your current PR reviewing and merging.
  • For specs comparison cross API versions, Use API Specs Comparison Report Generator
  • If there is CI failure(s), to fix CI error(s) is mandatory for PR merging; or you need to provide justification in PR comment for explanation. How to fix?
  • Any feedback about review process or workflow bot, pls contact swagger and tools team. [email protected]

    Swagger Validation Report

    ️❌BreakingChange: 5 Errors, 0 Warnings failed [Detail]
    compared swaggers (via Oad v0.10.1)] new version base version
    databricks.json 2022-04-01-preview(54f5eab) 2022-04-01-preview(main)
    Rule Message
    1041 - AddedPropertyInResponse The new version has a new property 'managedDisk' in response that was not found in the old version.
    New: Microsoft.Databricks/preview/2022-04-01-preview/databricks.json#L1188:7
    Old: Microsoft.Databricks/preview/2022-04-01-preview/databricks.json#L1168:7
    1041 - AddedPropertyInResponse The new version has a new property 'managedDiskIdentity' in response that was not found in the old version.
    New: Microsoft.Databricks/preview/2022-04-01-preview/databricks.json#L741:7
    Old: Microsoft.Databricks/preview/2022-04-01-preview/databricks.json#L730:7
    1041 - AddedPropertyInResponse The new version has a new property 'diskEncryptionSetId' in response that was not found in the old version.
    New: Microsoft.Databricks/preview/2022-04-01-preview/databricks.json#L741:7
    Old: Microsoft.Databricks/preview/2022-04-01-preview/databricks.json#L730:7
    1045 - AddedOptionalProperty The new version has a new optional property 'managedDiskIdentity' that was not found in the old version.
    New: Microsoft.Databricks/preview/2022-04-01-preview/databricks.json#L741:7
    Old: Microsoft.Databricks/preview/2022-04-01-preview/databricks.json#L730:7
    1045 - AddedOptionalProperty The new version has a new optional property 'diskEncryptionSetId' that was not found in the old version.
    New: Microsoft.Databricks/preview/2022-04-01-preview/databricks.json#L741:7
    Old: Microsoft.Databricks/preview/2022-04-01-preview/databricks.json#L730:7
    ️️✔️Breaking Change(Cross-Version) succeeded [Detail] [Expand]
    There are no breaking changes.
    ️️✔️CredScan succeeded [Detail] [Expand]
    There is no credential detected.
    ️⚠️LintDiff: 1 Warnings warning [Detail]
    compared tags (via openapi-validator v1.13.0) new version base version
    package-2022-04-01-preview package-2022-04-01-preview(54f5eab) package-2022-04-01-preview(main)

    [must fix]The following errors/warnings are introduced by current PR:

    Rule Message Related RPC [For API reviewers]
    :warning: R3018 - EnumInsteadOfBoolean Booleans are not descriptive and make them hard to use. Consider using string enums with allowed set of values defined. Property: rotationToLatestKeyVersionEnabled
    Location: Microsoft.Databricks/preview/2022-04-01-preview/databricks.json#L1280


    The following errors/warnings exist before current PR submission:

    Only 30 items are listed, please refer to log for more details.

    Rule Message
    R4018 - OperationsApiResponseSchema The response schema of operations API '/providers/Microsoft.Databricks/operations' does not match the ARM specification. Please standardize the schema.
    Location: Microsoft.Databricks/preview/2022-04-01-preview/databricks.json#L337
    R4037 - MissingTypeObject The schema 'Workspace' is considered an object but without a 'type:object', please add the missing 'type:object'.
    Location: Microsoft.Databricks/preview/2022-04-01-preview/databricks.json#L712
    R4037 - MissingTypeObject The schema 'Workspace' is considered an object but without a 'type:object', please add the missing 'type:object'.
    Location: Microsoft.Databricks/preview/2022-04-01-preview/databricks.json#L712
    R4037 - MissingTypeObject The schema 'WorkspaceProperties' is considered an object but without a 'type:object', please add the missing 'type:object'.
    Location: Microsoft.Databricks/preview/2022-04-01-preview/databricks.json#L740
    R4037 - MissingTypeObject The schema 'WorkspaceProperties' is considered an object but without a 'type:object', please add the missing 'type:object'.
    Location: Microsoft.Databricks/preview/2022-04-01-preview/databricks.json#L740
    R4037 - MissingTypeObject The schema 'TrackedResource' is considered an object but without a 'type:object', please add the missing 'type:object'.
    Location: Microsoft.Databricks/preview/2022-04-01-preview/databricks.json#L853
    R4037 - MissingTypeObject The schema 'TrackedResource' is considered an object but without a 'type:object', please add the missing 'type:object'.
    Location: Microsoft.Databricks/preview/2022-04-01-preview/databricks.json#L853
    R4037 - MissingTypeObject The schema 'Resource' is considered an object but without a 'type:object', please add the missing 'type:object'.
    Location: Microsoft.Databricks/preview/2022-04-01-preview/databricks.json#L886
    R4037 - MissingTypeObject The schema 'Resource' is considered an object but without a 'type:object', please add the missing 'type:object'.
    Location: Microsoft.Databricks/preview/2022-04-01-preview/databricks.json#L886
    R4037 - MissingTypeObject The schema 'Sku' is considered an object but without a 'type:object', please add the missing 'type:object'.
    Location: Microsoft.Databricks/preview/2022-04-01-preview/databricks.json#L907
    R4037 - MissingTypeObject The schema 'Sku' is considered an object but without a 'type:object', please add the missing 'type:object'.
    Location: Microsoft.Databricks/preview/2022-04-01-preview/databricks.json#L907
    R4037 - MissingTypeObject The schema 'ManagedIdentityConfiguration' is considered an object but without a 'type:object', please add the missing 'type:object'.
    Location: Microsoft.Databricks/preview/2022-04-01-preview/databricks.json#L960
    R4037 - MissingTypeObject The schema 'ManagedIdentityConfiguration' is considered an object but without a 'type:object', please add the missing 'type:object'.
    Location: Microsoft.Databricks/preview/2022-04-01-preview/databricks.json#L960
    R4037 - MissingTypeObject The schema 'WorkspaceCustomBooleanParameter' is considered an object but without a 'type:object', please add the missing 'type:object'.
    Location: Microsoft.Databricks/preview/2022-04-01-preview/databricks.json#L996
    R4037 - MissingTypeObject The schema 'WorkspaceCustomBooleanParameter' is considered an object but without a 'type:object', please add the missing 'type:object'.
    Location: Microsoft.Databricks/preview/2022-04-01-preview/databricks.json#L996
    R4037 - MissingTypeObject The schema 'WorkspaceCustomObjectParameter' is considered an object but without a 'type:object', please add the missing 'type:object'.
    Location: Microsoft.Databricks/preview/2022-04-01-preview/databricks.json#L1012
    R4037 - MissingTypeObject The schema 'WorkspaceCustomObjectParameter' is considered an object but without a 'type:object', please add the missing 'type:object'.
    Location: Microsoft.Databricks/preview/2022-04-01-preview/databricks.json#L1012
    R4037 - MissingTypeObject The schema 'WorkspaceCustomStringParameter' is considered an object but without a 'type:object', please add the missing 'type:object'.
    Location: Microsoft.Databricks/preview/2022-04-01-preview/databricks.json#L1028
    R4037 - MissingTypeObject The schema 'WorkspaceCustomStringParameter' is considered an object but without a 'type:object', please add the missing 'type:object'.
    Location: Microsoft.Databricks/preview/2022-04-01-preview/databricks.json#L1028
    R4037 - MissingTypeObject The schema 'WorkspaceCustomParameters' is considered an object but without a 'type:object', please add the missing 'type:object'.
    Location: Microsoft.Databricks/preview/2022-04-01-preview/databricks.json#L1044
    R4037 - MissingTypeObject The schema 'WorkspaceCustomParameters' is considered an object but without a 'type:object', please add the missing 'type:object'.
    Location: Microsoft.Databricks/preview/2022-04-01-preview/databricks.json#L1044
    R4037 - MissingTypeObject The schema 'CreatedBy' is considered an object but without a 'type:object', please add the missing 'type:object'.
    Location: Microsoft.Databricks/preview/2022-04-01-preview/databricks.json#L1120
    R4037 - MissingTypeObject The schema 'CreatedBy' is considered an object but without a 'type:object', please add the missing 'type:object'.
    Location: Microsoft.Databricks/preview/2022-04-01-preview/databricks.json#L1120
    R4037 - MissingTypeObject The schema 'WorkspaceEncryptionParameter' is considered an object but without a 'type:object', please add the missing 'type:object'.
    Location: Microsoft.Databricks/preview/2022-04-01-preview/databricks.json#L1142
    R4037 - MissingTypeObject The schema 'WorkspaceEncryptionParameter' is considered an object but without a 'type:object', please add the missing 'type:object'.
    Location: Microsoft.Databricks/preview/2022-04-01-preview/databricks.json#L1142
    R4037 - MissingTypeObject The schema 'Encryption' is considered an object but without a 'type:object', please add the missing 'type:object'.
    Location: Microsoft.Databricks/preview/2022-04-01-preview/databricks.json#L1155
    R4037 - MissingTypeObject The schema 'Encryption' is considered an object but without a 'type:object', please add the missing 'type:object'.
    Location: Microsoft.Databricks/preview/2022-04-01-preview/databricks.json#L1155
    R4037 - MissingTypeObject The schema 'EncryptionEntitiesDefinition' is considered an object but without a 'type:object', please add the missing 'type:object'.
    Location: Microsoft.Databricks/preview/2022-04-01-preview/databricks.json#L1187
    R4037 - MissingTypeObject The schema 'EncryptionEntitiesDefinition' is considered an object but without a 'type:object', please add the missing 'type:object'.
    Location: Microsoft.Databricks/preview/2022-04-01-preview/databricks.json#L1187
    R4037 - MissingTypeObject The schema 'EncryptionV2' is considered an object but without a 'type:object', please add the missing 'type:object'.
    Location: Microsoft.Databricks/preview/2022-04-01-preview/databricks.json#L1202
    ️️✔️Avocado succeeded [Detail] [Expand]
    Validation passes for Avocado.
    ️️✔️ApiReadinessCheck succeeded [Detail] [Expand]
    ️⚠️~[Staging] ServiceAPIReadinessTest: 0 Warnings warning [Detail]

    API Test is not triggered due to precheck failure. Check pipeline log for details.

    ️️✔️ModelValidation succeeded [Detail] [Expand]
    Validation passes for ModelValidation.
    ️️✔️SemanticValidation succeeded [Detail] [Expand]
    Validation passes for SemanticValidation.
    ️️✔️PoliCheck succeeded [Detail] [Expand]
    Validation passed for PoliCheck.
    ️️✔️PrettierCheck succeeded [Detail] [Expand]
    Validation passes for PrettierCheck.
    ️️✔️SpellCheck succeeded [Detail] [Expand]
    Validation passes for SpellCheck.
    ️️✔️Lint(RPaaS) succeeded [Detail] [Expand]
    Validation passes for Lint(RPaaS).
    ️️✔️CadlValidation succeeded [Detail] [Expand]
    Validation passes for CadlValidation.
    ️️✔️PR Summary succeeded [Detail] [Expand]
    Validation passes for Summary.
    Posted by Swagger Pipeline | How to fix these errors?

    Swagger Generation Artifacts

    ️️✔️ApiDocPreview succeeded [Detail] [Expand]
     Please click here to preview with your @microsoft account. 
    ️️✔️SDK Breaking Change Tracking succeeded [Detail] [Expand]

    Breaking Changes Tracking


    ️⚠️ azure-sdk-for-python-track2 warning [Detail]
    • ⚠️Warning [Logs]Release - Generate from aafb0944f7ab936e8cfbad8969bd5eb32263fb4f. SDK Automation 14.0.0
      command	sh scripts/automation_init.sh ../azure-sdk-for-python_tmp/initInput.json ../azure-sdk-for-python_tmp/initOutput.json
      cmderr	[automation_init.sh] WARNING: Skipping azure-nspkg as it is not installed.
      command	sh scripts/automation_generate.sh ../azure-sdk-for-python_tmp/generateInput.json ../azure-sdk-for-python_tmp/generateOutput.json
    • ️✔️track2_azure-mgmt-databricks [View full logs]  [Release SDK Changes]
      info	[Changelog] ### Features Added
      info	[Changelog]
      info	[Changelog]   - Added operation group AccessConnectorsOperations
      info	[Changelog]   - Added operation group OutboundNetworkDependenciesEndpointsOperations
      info	[Changelog]   - Model EncryptionEntitiesDefinition has a new parameter managed_disk
      info	[Changelog]   - Model Workspace has a new parameter disk_encryption_set_id
      info	[Changelog]   - Model Workspace has a new parameter managed_disk_identity
    ️️✔️ azure-sdk-for-js succeeded [Detail] [Expand]
    • ️✔️Succeeded [Logs]Release - Generate from aafb0944f7ab936e8cfbad8969bd5eb32263fb4f. SDK Automation 14.0.0
      command	sh .scripts/automation_init.sh ../azure-sdk-for-js_tmp/initInput.json ../azure-sdk-for-js_tmp/initOutput.json
      warn	File azure-sdk-for-js_tmp/initOutput.json not found to read
      command	sh .scripts/automation_generate.sh ../azure-sdk-for-js_tmp/generateInput.json ../azure-sdk-for-js_tmp/generateOutput.json
    • ️✔️@azure/arm-databricks [View full logs]  [Release SDK Changes]
      info	[Changelog]
      error	breakingChangeTracking is enabled, but version or changelogItem is not found in output.
    ️⚠️ azure-resource-manager-schemas warning [Detail]
    • ⚠️Warning [Logs]Release - Generate from aafb0944f7ab936e8cfbad8969bd5eb32263fb4f. Schema Automation 14.0.0
      command	.sdkauto/initScript.sh ../azure-resource-manager-schemas_tmp/initInput.json ../azure-resource-manager-schemas_tmp/initOutput.json
      cmderr	[initScript.sh]  old lockfile
      cmderr	[initScript.sh] npm WARN old lockfile The package-lock.json file was created with an old version of npm,
      cmderr	[initScript.sh] npm WARN old lockfile so supplemental metadata must be fetched from the registry.
      cmderr	[initScript.sh] npm WARN old lockfile
      cmderr	[initScript.sh] npm WARN old lockfile This is a one-time fix-up, please be patient...
      cmderr	[initScript.sh] npm WARN old lockfile
      warn	File azure-resource-manager-schemas_tmp/initOutput.json not found to read
      command	.sdkauto/generateScript.sh ../azure-resource-manager-schemas_tmp/generateInput.json ../azure-resource-manager-schemas_tmp/generateOutput.json
    • ️✔️databricks [View full logs]  [Release Schema Changes]
    ️️✔️ azure-powershell succeeded [Detail] [Expand]
    • ️✔️Succeeded [Logs]Release - Generate from aafb0944f7ab936e8cfbad8969bd5eb32263fb4f. SDK Automation 14.0.0
      command	sh ./tools/SwaggerCI/init.sh ../azure-powershell_tmp/initInput.json ../azure-powershell_tmp/initOutput.json
      command	pwsh ./tools/SwaggerCI/psci.ps1 ../azure-powershell_tmp/generateInput.json ../azure-powershell_tmp/generateOutput.json
    • ️✔️Az.Databricks [View full logs]  [Release SDK Changes]
    Posted by Swagger Pipeline | How to fix these errors?

    Generated ApiView

    Language Package Name ApiView Link
    Python track2_azure-mgmt-databricks Create ApiView failed. Please ensure your github account in Azure/Microsoft is public and add a comment "/azp run" to re-trigger the CI.
    JavaScript @azure/arm-databricks Create ApiView failed. Please ensure your github account in Azure/Microsoft is public and add a comment "/azp run" to re-trigger the CI.

    Hi @scottmacchia-msft, one or multiple breaking change(s) is detected in your PR. Please check out the breaking change(s), and provide business justification in the PR comment and @ PR assignee why you must have these change(s), and how external customer impact can be mitigated. Please ensure to follow breaking change policy to request breaking change review and approval before proceeding swagger PR review. Action: To initiate an evaluation of the breaking change, create a new intake using the template for breaking changes. Addition details on the process and office hours are on the Breaking change Wiki. If you want to know the production traffic statistic, please see ARM Traffic statistic. If you think it is false positive breaking change, please provide the reasons in the PR comment, report to Swagger Tooling Team via https://aka.ms/swaggerfeedback. Note: To avoid breaking change, you can refer to Shift Left Solution for detecting breaking change in early phase at your service code repository.

    Hi @scottmacchia-msft, Your PR has some issues. Please fix the CI sequentially by following the order of Avocado, semantic validation, model validation, breaking change, lintDiff. If you have any questions, please post your questions in this channel https://aka.ms/swaggersupport.

    TaskHow to fixPriority
    AvocadoFix-AvocadoHigh
    Semantic validationFix-SemanticValidation-ErrorHigh
    Model validationFix-ModelValidation-ErrorHigh
    LintDiffFix-LintDiffhigh
    If you need further help, please feedback via swagger feedback.

    NewApiVersionRequired reason: A service’s API is a contract with customers and is represented by using the api-version query parameter. Changes such as adding an optional property to a request/response or introducing a new operation is a change to the service’s contract and therefore requires a new api-version value. This is critically important for documentation, client libraries, and customer support. EXAMPLE: if a customer calls a service in the public cloud using api-version=2020-07-27, the new property or operation may exist but if they call the service in a government cloud, air-gapped cloud, or Azure Stack Hub cloud using the same api-version, the property or operation may not exist. Because there is no clear relationship between the service api-version and the new property/operation, customers can’t trust the documentation and Azure customer have difficulty helping customers diagnose issues. In addition, each client library version documents the service version it supports. When an optional property or new operation is added to a service and its Swagger, new client libraries must be produced to expose this functionality to customers. Without updating the api-version, it is unclear to customers which version of a client library supports these new features.

    Appears that you are adding missing properties that should be documented in swagger in this api-version.

    But I will still need an approval from breaking changes for this. Let me know if you ready to do that.

    weidongxu-microsoft avatar Oct 27 '22 02:10 weidongxu-microsoft

    ping me to merge, when you are ready.

    weidongxu-microsoft avatar Oct 28 '22 02:10 weidongxu-microsoft