Adding cadl sample project

[raych1]

added a cadl sample project for DPG scenario with shared library. referenced this sample project in the cadl-structure-guidelines document.

[openapi-workflow-bot[bot]]

Hi, @raych1 Thanks for your PR. I am workflow bot for review process. Here are some small tips.

Please ensure to do self-check against checklists in first PR comment.

PR assignee is the person auto-assigned and responsible for your current PR reviewing and merging.

For specs comparison cross API versions, Use API Specs Comparison Report Generator

If there is CI failure(s), to fix CI error(s) is mandatory for PR merging; or you need to provide justification in PR comment for explanation. How to fix?

Any feedback about review process or workflow bot, pls contact swagger and tools team. [email protected]

[openapi-pipeline-app[bot]]

Swagger Validation Report

️️✔️BreakingChange succeeded [Detail] [Expand]

There are no breaking changes.

️️✔️Breaking Change(Cross-Version) succeeded [Detail] [Expand]

There are no breaking changes.

️️✔️CredScan succeeded [Detail] [Expand]

There is no credential detected.

️️✔️LintDiff succeeded [Detail] [Expand]

Validation passes for LintDiff.

️❌Avocado: 2 Errors, 0 Warnings failed [Detail]

Rule	Message
❌ MISSING_README	Can not find readme.md in the folder. If no readme.md file, it will block SDK generation. folder: Azure.Contoso.WidgetManager/preview/2022-11-01-preview/examples
❌ MISSING_README	Can not find readme.md in the folder. If no readme.md file, it will block SDK generation. folder: data-plane/Azure.Contoso.WidgetManager/preview/2022-11-01-preview

️️✔️ApiReadinessCheck succeeded [Detail] [Expand]

️❌~[Staging] ServiceAPIReadinessTest: 0 Errors, 0 Warnings failed [Detail]

️️✔️~[Staging] SwaggerAPIView succeeded [Detail] [Expand]

️️✔️ModelValidation succeeded [Detail] [Expand]

Validation passes for ModelValidation.

️️✔️SemanticValidation succeeded [Detail] [Expand]

Validation passes for SemanticValidation.

️️✔️PoliCheck succeeded [Detail] [Expand]

Validation passed for PoliCheck.

️️✔️PrettierCheck succeeded [Detail] [Expand]

Validation passes for PrettierCheck.

️️✔️SpellCheck succeeded [Detail] [Expand]

Validation passes for SpellCheck.

️️✔️Lint(RPaaS) succeeded [Detail] [Expand]

Validation passes for Lint(RPaaS).

️️✔️CadlValidation succeeded [Detail] [Expand]

Validation passes for CadlValidation.

️️✔️PR Summary succeeded [Detail] [Expand]

Validation passes for Summary.

_{Posted by Swagger Pipeline | How to fix these errors?}

[openapi-pipeline-app[bot]]

Swagger Generation Artifacts

️️✔️ApiDocPreview succeeded [Detail] [Expand]

 Please click here to preview with your @microsoft account. 

️️✔️SDK Breaking Change Tracking succeeded [Detail] [Expand]

Breaking Changes Tracking

️⚠️ azure-sdk-for-python warning [Detail]

• ⚠️Warning [Logs]Release - Generate from 6c192122507728c8ad9eb999e7970262f45673c3. SDK Automation 14.0.0

  command	sh scripts/automation_init.sh ../azure-sdk-for-python_tmp/initInput.json ../azure-sdk-for-python_tmp/initOutput.json
  cmderr	[automation_init.sh] WARNING: Skipping azure-nspkg as it is not installed.
  command	sh scripts/automation_generate.sh ../azure-sdk-for-python_tmp/generateInput.json ../azure-sdk-for-python_tmp/generateOutput.json
  cmderr	[automation_generate.sh] npm notice
  cmderr	[automation_generate.sh] npm notice New major version of npm available! 8.19.2 -> 9.2.0
  cmderr	[automation_generate.sh] npm notice Changelog: <https://github.com/npm/cli/releases/tag/v9.2.0>
  cmderr	[automation_generate.sh] npm notice Run `npm install -g [email protected]` to update!
  cmderr	[automation_generate.sh] npm notice
  warn	No file changes detected after generation
  warn	No package detected after generation

️❌ azure-sdk-for-net-track2 failed [Detail]

• ❌Code Generator Failed [Logs]Release - Generate from 6c192122507728c8ad9eb999e7970262f45673c3. SDK Automation 14.0.0

  command	pwsh ./eng/scripts/Automation-Sdk-Init.ps1 ../azure-sdk-for-net_tmp/initInput.json ../azure-sdk-for-net_tmp/initOutput.json
  command	pwsh ./eng/scripts/Invoke-GenerateAndBuildV2.ps1 ../azure-sdk-for-net_tmp/generateInput.json ../azure-sdk-for-net_tmp/generateOutput.json
  cmderr	[Invoke-GenerateAndBuildV2.ps1] [91mInvalidOperation: [0m/mnt/vss/_work/1/s/azure-sdk-for-net/eng/scripts/Invoke-GenerateAndBuildV2.ps1:122
  cmderr	[Invoke-GenerateAndBuildV2.ps1] [96mLine |
  cmderr	[Invoke-GenerateAndBuildV2.ps1] [96m 122 | [0m         [96m$sdkFolder = $yml["emitters"]["@azure-tools/cadl-csharp"]["sd[0m …
  cmderr	[Invoke-GenerateAndBuildV2.ps1] [96m     | [91m         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  cmderr	[Invoke-GenerateAndBuildV2.ps1] [91m[96m     | [91mCannot index into a null array.
  cmderr	[Invoke-GenerateAndBuildV2.ps1] [0m
  cmderr	[Invoke-GenerateAndBuildV2.ps1] [91mException: [0m/mnt/vss/_work/1/s/azure-sdk-for-net/eng/scripts/automation/GenerateAndBuildLib.ps1:452
  cmderr	[Invoke-GenerateAndBuildV2.ps1] [96mLine |
  cmderr	[Invoke-GenerateAndBuildV2.ps1] [96m 452 | [0m             [96mThrow "Error: invalid namespace name."[0m
  cmderr	[Invoke-GenerateAndBuildV2.ps1] [96m     | [91m             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  cmderr	[Invoke-GenerateAndBuildV2.ps1] [91m[96m     | [91mError: invalid namespace name.
  cmderr	[Invoke-GenerateAndBuildV2.ps1] [0m
  error	Script return with result [failed] code [1] signal [null] cwd [azure-sdk-for-net]: pwsh ./eng/scripts/Invoke-GenerateAndBuildV2.ps1
  warn	Skip package processing as generation is failed

️❌ azure-sdk-for-js failed [Detail]

• ❌Code Generator Failed [Logs]Release - Generate from 6c192122507728c8ad9eb999e7970262f45673c3. SDK Automation 14.0.0

  command	sh .scripts/automation_init.sh ../azure-sdk-for-js_tmp/initInput.json ../azure-sdk-for-js_tmp/initOutput.json
  warn	File azure-sdk-for-js_tmp/initOutput.json not found to read
  command	sh .scripts/automation_generate.sh ../azure-sdk-for-js_tmp/generateInput.json ../azure-sdk-for-js_tmp/generateOutput.json
  cmderr	[automation_generate.sh] [ERROR] Command failed: npx cadl compile . --emit @azure-tools/cadl-typescript
  error	Script return with result [failed] code [1] signal [null] cwd [azure-sdk-for-js]: sh .scripts/automation_generate.sh
  warn	Skip package processing as generation is failed

️❌ azure-sdk-for-java failed [Detail]

• ❌Code Generator Failed [Logs]Release - Generate from 6c192122507728c8ad9eb999e7970262f45673c3. SDK Automation 14.0.0

  command	./eng/mgmt/automation/init.sh ../azure-sdk-for-java_tmp/initInput.json ../azure-sdk-for-java_tmp/initOutput.json
  cmderr	[init.sh] [notice] A new release of pip available: 22.3 -> 22.3.1
  cmderr	[init.sh] [notice] To update, run: pip install --upgrade pip
  cmderr	[init.sh] [notice] A new release of pip available: 22.3 -> 22.3.1
  cmderr	[init.sh] [notice] To update, run: pip install --upgrade pip
  command	./eng/mgmt/automation/generate.py ../azure-sdk-for-java_tmp/generateInput.json ../azure-sdk-for-java_tmp/generateOutput.json
  cmderr	[generate.py] Traceback (most recent call last):
  cmderr	[generate.py]   File "/mnt/vss/_work/1/s/azure-sdk-for-java/./eng/mgmt/automation/generate.py", line 274, in <module>
  cmderr	[generate.py]     main()
  cmderr	[generate.py]   File "/mnt/vss/_work/1/s/azure-sdk-for-java/./eng/mgmt/automation/generate.py", line 204, in main
  cmderr	[generate.py]     return sdk_automation(args['config'][0], args['config'][1])
  cmderr	[generate.py]   File "/mnt/vss/_work/1/s/azure-sdk-for-java/./eng/mgmt/automation/generate.py", line 97, in sdk_automation
  cmderr	[generate.py]     packages = sdk_automation_cadl(config)
  cmderr	[generate.py]   File "/mnt/vss/_work/1/s/azure-sdk-for-java/eng/mgmt/automation/generate_data.py", line 42, in sdk_automation_cadl
  cmderr	[generate.py]     sdk_folder = get_cadl_sdk_folder(os.path.join(cadl_dir, 'cadl-project.yaml'))
  cmderr	[generate.py]   File "/mnt/vss/_work/1/s/azure-sdk-for-java/eng/mgmt/automation/generate_data.py", line 118, in get_cadl_sdk_folder
  cmderr	[generate.py]     sdk_folder = project_yaml['emitters']['@azure-tools/cadl-java']['sdk-folder']
  cmderr	[generate.py] KeyError: 'emitters'
  error	Script return with result [failed] code [1] signal [null] cwd [azure-sdk-for-java]: ./eng/mgmt/automation/generate.py
  warn	Skip package processing as generation is failed

_{Posted by Swagger Pipeline | How to fix these errors?}

[openapi-pipeline-app[bot]]

Swagger pipeline restarted successfully. If there is ApiView generated, it will be updated in this comment.

[openapi-workflow-bot[bot]]

Hi @raych1, Your PR has some issues. Please fix the CI sequentially by following the order of Avocado, semantic validation, model validation, breaking change, lintDiff. If you have any questions, please post your questions in this channel https://aka.ms/swaggersupport.

Task	How to fix	Priority
Avocado	Fix-Avocado	High
Semantic validation	Fix-SemanticValidation-Error	High
Model validation	Fix-ModelValidation-Error	High
LintDiff	Fix-LintDiff	high

If you need further help, please feedback via swagger feedback.

[allenjzhang]

Proposal to discuss:

rest-api-spec repo

• cadl-project.yaml

  1. autorest emitter configuration should be there. With pending new cadl config features, this should be a static file (no modification)
  2. suggest NOT contain any SDK emitter specific configuration. these should come from SDKRepo/service folder.

• package.json

  1. should not contain any SDK related packages info.

Repo will enforce a single set of cadl package version via "npm install". Example in step 2 below.

SDK repo

• cadl-project.yaml or quevelant configuration file to store langauge specific emitter settings

• configuration file with link to a spec folder with commit id info

• Pipeline logic

  1. sparse-checkout folder that include all folders
  2. with existing package.json to create a workable version by updating the package versions to a fixed SDK profile with " npm install [email protected] SDK-@?? xxx@??
  3. Combin/append SDK emitter config to cadl-project.yaml downloaded from rest-api-spec to get a working yaml
  4. Build to regen code.

[dw511214992]

Proposal to discuss:

rest-api-spec repo

• cadl-project.yaml

  1. autorest emitter configuration should be there. With pending new cadl config features, this should be a static file (no modification)
  2. suggest NOT contain any SDK emitter specific configuration. these should come from SDKRepo/service folder.

• package.json

  1. should not contain any SDK related packages info.

Repo will enforce a single set of cadl package version via "npm install". Example in step 2 below.

SDK repo

• cadl-project.yaml or quevelant configuration file to store langauge specific emitter settings

• configuration file with link to a spec folder with commit id info

• Pipeline logic

  1. sparse-checkout folder that include all folders
  2. with existing package.json to create a workable version by updating the package versions to a fixed SDK profile with " npm install [email protected] SDK-@?? xxx@??
  3. Combin/append SDK emitter config to cadl-project.yaml downloaded from rest-api-spec to get a working yaml
  4. Build to regen code.

@allenjzhang @weshaggard From my side, I have some questions after looking into the proposal:

1. If the emitter's dependencies' versions are different from package.json’s dependencies, there will be version conflicts.
2. For new package, there is no cadl-project.yaml in sdk folder, and then we cannot generate sdk. (The solution of swagger DPG maybe an option, will we use the solution in cadl too?)
3. For existing package, when there is a spec PR, we cannot find which cadl-project.yaml in sdk folder is related to the changed files in the spec PR. (does the link in sdk repo’s cadl-project.yaml contains enough information to map the spec PR’s changed files to sdk repo’s cadl-project.yaml?)
4. How do we determine which language of sdk will be generated? In swagger readme.md, there is a section “swagger-to-sdk” to define which language of sdk will be generated. In previous cadl project folder structure, I parse the cadl-project.yaml and see which sdk emitter is defined in it, and then get which language of sdk will be generated. If the cadl-project.yaml in spec repo doesn’t include any sdk related information, is there a place to store such kind of information? Or by default we will generate all languages of sdk?

[weshaggard]

Just to clarify some stuff in @allenjzhang's proposal. I do think we want to have a cadl-project.yaml file in the spec repo and it should contain general configuration such as package name and namespace for a given emitter. If there is a need for language specific configurations outside of the basic naming then those properties can come from the sdk repo. I don't think the sdk repo should have a cadl-project.yaml but instead we should have some other metadata file that will contain things like the spec version and such, something like cadl-location.yaml (we could rename it if necessary). In that file we could add additional properties/config if they are needed for the emitter.

To help answer @dw511214992's questions:

1. The goal will be to not have any emitter versions in the package.json file in the specs repo. If there are versions in there we should either update them to be consistent. We might need to update all the package versions for every package dependency of the emitter we are trying to run.
2. For new sdk packages we should generate a basic cadl-location.yaml file (not cadl-project.yaml), see my earlier comment. If one doesn't exist then we should generate one from the template just like we will have to generate a project as well.
3. Yes I would propose that this be the cadl-location.yaml file and it should have the pointers we need to where the spec is contained and it should be updated when we are updating the spec.
4. Good question that I didn't consider yet. I think the list of emitters should still remain in the cald-project.yaml file in the specs repo. This as well as naming configuration is a good reason to keep them in there.

[allenjzhang]

Just to clarify some stuff in @allenjzhang's proposal. I do think we want to have a cadl-project.yaml file in the spec repo and it should contain general configuration such as package name and namespace for a given emitter. If there is a need for language specific configurations outside of the basic naming then those properties can come from the sdk repo. I don't think the sdk repo should have a cadl-project.yaml but instead we should have some other metadata file that will contain things like the spec version and such, something like cadl-location.yaml (we could rename it if necessary). In that file we could add additional properties/config if they are needed for the emitter.

Yup

To help answer @dw511214992's questions:

1. The goal will be to not have any emitter versions in the package.json file in the specs repo. If there are versions in there we should either update them to be consistent. We might need to update all the package versions for every package dependency of the emitter we are trying to run.

2. For new sdk packages we should generate a basic cadl-location.yaml file (not cadl-project.yaml), see my earlier comment. If one doesn't exist then we should generate one from the template just like we will have to generate a project as well.

1, 2 Agreed.

[allenjzhang]

Regarding #4, my think is spec repo will a only contain Cadl-autorest emitter entry. For each language SDK repo, the emitter name will be injected either from command line or manipulation result of Cadl-project.yaml. Most likely the former.

For spec ci pipeline where everything need to be generated, I assume all lang emitter will be passed on CLI.

[weshaggard]

Regarding https://github.com/Azure/azure-rest-api-specs/pull/4, my think is spec repo will a only contain Cadl-autorest emitter entry. For each language SDK repo, the emitter name will be injected either from command line or manipulation result of Cadl-project.yaml. Most likely the former.

For spec ci pipeline where everything need to be generated, I assume all lang emitter will be passed on CLI.

I agree the project.json should only contain the swagger emitter entry but I think we need all the supported emitters and their configurations listed in the cadl-project.yaml file. They will need configurations and they should be listed on each emitter specifically.

[m-nash]

Regarding #4, my think is spec repo will a only contain Cadl-autorest emitter entry. For each language SDK repo, the emitter name will be injected either from command line or manipulation result of Cadl-project.yaml. Most likely the former.

For spec ci pipeline where everything need to be generated, I assume all lang emitter will be passed on CLI.

I agree the project.json should only contain the swagger emitter entry but I think we need all the supported emitters and their configurations listed in the cadl-project.yaml file. They will need configurations and they should be listed on each emitter specifically.

I also lean towards keeping all emitter configurations in the cadl-project.yaml in the spec repo and not having one in the sdk folder. There are no technical blockers from doing this that I am aware of right now and it fits with our long term goals of being able to fully generate out of the spec repo (even if its just a script that wraps generating from all the language repos).

[allenjzhang]

Depending on how you want to use the samples here as a demonstrator or a receipt for a user to follow. If later, we will also need to introduce two different flavors Data-plane and ARM to ensure a good out-of-box user experience, where:

• DP should include cadl-dpg in the package.config
• include generate model flag true by default,
• having a separate client.cadl file with some basic client configurations.

My thinking is the former. It's a demostration project and for us to validate and test. In the official getting started step, we will ask the user to use cadl init azure to select from a list of templates that we can refine later without holding up this PR.

[raych1]

Depending on how you want to use the samples here as a demonstrator or a receipt for a user to follow. If later, we will also need to introduce two different flavors Data-plane and ARM to ensure a good out-of-box user experience, where:

• DP should include cadl-dpg in the package.config
• include generate model flag true by default,
• having a separate client.cadl file with some basic client configurations.

My thinking is the former. It's a demostration project and for us to validate and test. In the official getting started step, we will ask the user to use cadl init azure to select from a list of templates that we can refine later without holding up this PR.

Agree, we can add data plane and management plane templates later to cadl init azure later.

[raych1]

Depending on how you want to use the samples here as a demonstrator or a receipt for a user to follow. If later, we will also need to introduce two different flavors Data-plane and ARM to ensure a good out-of-box user experience, where:

• DP should include cadl-dpg in the package.config
• include generate model flag true by default,
• having a separate client.cadl file with some basic client configurations.

My thinking is the former. It's a demostration project and for us to validate and test. In the official getting started step, we will ask the user to use cadl init azure to select from a list of templates that we can refine later without holding up this PR.

Agree, we can add data plane and management plane templates later to cadl init azure later.

Discussed with @allenjzhang offline, we will create template base on this sample project then integrate into cadl init.

• DP should include cadl-dpg in the package.config -- sample template won't include this as it's only used in SDK generation
• include generate model flag true by default,
• having a separate client.cadl file with some basic client configurations. -- it will be common to add this client.cadl including a barebone with comments and samples