CI: Reuse check schema

[akihironitta]

What does this PR do?

Part of https://github.com/Lightning-AI/lightning/issues/12545.

Same as https://github.com/Lightning-AI/lightning/pull/14469.

Before submitting

• [x] Was this discussed/approved via a Github issue? (no need for typos and docs improvements)
• [x] Did you read the contributor guideline, Pull Request section?
• [x] Did you make sure your PR does only one thing, instead of bundling different changes together?
• [n/a] Did you make sure to update the documentation with your changes?
• [n/a] Did you write any new necessary tests? [not needed for typos/docs]
• [n/a] Did you verify new and existing tests pass locally with your changes?
• [n/a] If you made a notable change (that affects users), did you update the CHANGELOG?

PR review

• [x] Is this pull request ready for review? (if not, please submit in draft mode)

Anyone in the community is free to review the PR once the tests have passed. If we didn't discuss your PR in Github issues there's a high chance it will not be merged.

Did you have fun?

Make sure you had fun coding 🙃

[akihironitta]

@akihironitta - Just curious, what does this do? Sorry, I don't have much context. Approving as I trust you on this ;)

Haha, thanks for asking 😆

The actual workflow file sits in another repository: https://github.com/Lightning-AI/utilities/blob/215a98333e5fe8906156a290ad5f577e176572d1/.github/workflows/check-schema.yml#L25-L37 , and basically, it just checks if yml config files (such as azure and GHA workflow files) are in the correct structure, e.g. indentation, typo, etc.

For example, here's the azure schema file that defines what the correct structure is for azure pipelines' config files.

cc @Borda just in case you have something to add here.

[krshrimali]

Thanks, @akihironitta for the context. This helps! So does this mean that we'll have to rename .azure-pipelines folder to .azure_pipelines? Check shchema job is failing in this PR:

File "/home/runner/.local/lib/python3.8/site-packages/identify/identify.py", line [44](https://github.com/Lightning-AI/lightning-flash/runs/8242093713?check_suite_focus=true#step:7:45), in tags_from_path
    raise ValueError(f'{path} does not exist.')
ValueError: .azure_pipelines//*.yml does not exist.

We have .azure-pipelines folder, so maybe that's not expected?

[akihironitta]

Thank you @krshrimali for pointing it out! I have it fixed now :)

[krshrimali]

Awesome, thank you so much! 🔥 Let's merge once all relevant tests are green.

[akihironitta]

It seems the job added here is failing. Looking into it

[codecov[bot]]

Codecov Report

Base: 84.39% // Head: 92.36% // Increases project coverage by +7.96% :tada:

Coverage data is based on head (7a3622d) compared to base (6e95997). Patch has no changes to coverable lines.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1455      +/-   ##
==========================================
+ Coverage   84.39%   92.36%   +7.96%     
==========================================
  Files         292      292              
  Lines       13158    13158              
==========================================
+ Hits        11105    12153    +1048     
+ Misses       2053     1005    -1048     

Flag	Coverage Δ
pytest	13.03% <ø> (ø)
tpu	13.03% <ø> (ø)
unittests	92.89% <ø> (+8.01%)	:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
flash/core/data/io/input_transform.py	91.30% <0.00%> (+0.96%)	:arrow_up:
flash/core/data/utilities/classification.py	98.34% <0.00%> (+1.10%)	:arrow_up:
flash/core/adapter.py	87.17% <0.00%> (+1.28%)	:arrow_up:
flash/image/segmentation/data.py	100.00% <0.00%> (+2.00%)	:arrow_up:
flash/text/question_answering/input.py	95.06% <0.00%> (+2.46%)	:arrow_up:
flash/image/classification/data.py	98.65% <0.00%> (+2.68%)	:arrow_up:
flash/image/classification/model.py	80.70% <0.00%> (+3.50%)	:arrow_up:
flash/image/detection/model.py	97.29% <0.00%> (+5.40%)	:arrow_up:
flash/core/data/utilities/collate.py	100.00% <0.00%> (+5.55%)	:arrow_up:
flash/text/question_answering/collate.py	97.64% <0.00%> (+7.05%)	:arrow_up:
... and 56 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

[Borda]

@akihironitta could be that some inputs shall be a string, but they are passed as numbers...

[akihironitta]

It was indeed due to the incorrect schema provided by MSFT. true and false are valid values in the Azure Pipelines service but are considered invalid with the schema file as reported in the linked threads:

• schema definition: https://github.com/microsoft/azure-pipelines-vscode/blob/2a607113c4c0fb844d28ca0582a50920448c0e3e/service-schema.json#L4008
• https://github.com/microsoft/azure-pipelines-vscode/issues/446
• https://developercommunity.visualstudio.com/t/Azure-Pipelines-schema-inconsistently-li/1668070

In short, what we can do now is to quote all true and false occurrences so that we have "true" and "false" 😄

---

numbers, too, as you pointed out!

---

I see another issue around PublishCodeCoverageResults@1, and it turned out that this is something @\Borda reported to their repos quite a while ago 😄

• https://github.com/microsoft/azure-pipelines-vscode/issues/412
• https://github.com/python-jsonschema/check-jsonschema/issues/10

[akihironitta]

@Borda Would you say it's worth removing PublishCodeCoverageResults@1 to have the schema check pass?