Ability to score quality of async api specification file

[Ferror]

Reason/Context

Many optional yet valuable parameters in the Async API specification can improve the developer experience. Like example, description, format, etc. When working with many specifications, I was asked, "How good is our documentation?". How do I measure if my documentation is good, mediocre, insufficient, or 1-100?

I propose a new command like score or evaluate that will reuse existing warnings and errors from the validate command. The command will extend the list of best practices and generate some final/aggregated scores. Hints that will increase the score should be provided.

A couple of examples

1. if the specification has servers, then it increases the score
2. if the specification has descriptions, then it increases the score
3. if the specification has examples, then it increases the score
4. if the specification has a format, then it increases the score
5. if the specification has channels, then it increases the score
6. if the specification has operations, then it increases the score
7. if the specification has a license, then it increases the score

[github-actions[bot]]

Welcome to AsyncAPI. Thanks a lot for reporting your first issue. Please check out our contributors guide and the instructions about a basic recommended setup useful for opening a pull request.
Keep in mind there are also other channels you can use to interact with AsyncAPI community. For more details check out this issue.

[AayushSaini101]

@Amzani I want to work on this issue:) I have worked similar kind of issue for Jenkins https://github.com/jenkins-infra/plugin-health-scoring that provide scoring for each pluing on basis of some parameters

[Amzani]

Sure @AayushSaini101 please go ahead. That's a nice feature to have as well in studio

[AayushSaini101]

Sure @AayushSaini101 please go ahead. That's a nice feature to have as well in studio

@Amzani @derberg @Ferror We need to decide how much score we need to increase for particular parameter. Which parameter will be more proritise as compared to others, Otherwise it will create bias issue, What do you think guys ?

[Ferror]

I think we could start with something simple in the form of MVP.

1. The abstraction for scorecards/metrics (What to score)
2. Each metric may have a scale/weight
3. Deciding on an algorithm to calculate the sum/mediana/average

Feel free to propose something. We could also start with linear calculation (without weights) 😄

[AayushSaini101]

I think we could start with something simple in the form of MVP.

1. The abstraction for scorecards/metrics (What to score)
2. Each metric may have a scale/weight
3. Deciding on an algorithm to calculate the sum/mediana/average

Feel free to propose something. We could also start with linear calculation (without weights) 😄

@Amzani @Ferror If we used the same scoring mechanism as Jenkins used on the 1800+ Plugins, this would be the structure: Reference: https://docs.google.com/document/d/1XT5O99-QEZadFZsMuXhzahvEmeL7-1H7HWpsnPLuMIk/edit#heading=h.cw0t6u8yvrcm here my proposal

• New command structure: asyncapi evaluate

• there will be coefficient and score parameters, Coefficient will tell the weight of the particular parameter and score will be [0,1] the final score will be coefficient*score.

Example:

Health Score	Asyncapi Document 1	Asyncapi Document 2	Asyncapi Document 3
Score	(4.7*100)/totalParameters=24	(6.8*100)/totalParameters=35	(9*100)/totalParameters=47

Health Scoring Categories for Percentage: 

Level 1 [ Low Health ]: 0-25% 

Level 2 [ Medium Health]: 25-75%

Level 3 [ Good Health]: >75%

We can add lots of parameters like check whether the latest asyncapi version is used or not etc. We can create a poll to take the approval of the coeffiicent from the TSC Members using the voting that we created in community repo. :)

[Ferror]

TotalParameters will be based on async API schema, so it should be const. Quite simple and efficient ❤️

I'm not sure if the voting process would be IMO necessary, but feel free to create it 😄

[AayushSaini101]

TotalParameters will be based on async API schema, so it should be const. Quite simple and efficient ❤️

I'm not sure if the voting process would be IMO necessary, but feel free to create it 😄

@Ferror Regarding the coefiicient value of different parameters, i guess we need suggestion from members, that's why i am thinking of creating vote

[Amzani]

@AayushSaini101 Great proposal ❤️ voting from TSC members is not necessary, only maintainers of this repo. Could you run your proposal with some AsyncAPI files to see how it will look from user perspective?

[AayushSaini101]

@AayushSaini101 Great proposal ❤️ voting from TSC members is not necessary, only maintainers of this repo. Could you run your proposal with some AsyncAPI files to see how it will look from user perspective?

Sure, I will create a draft PR tomorrow :)

[asyncapi-bot]

:tada: This issue has been resolved in version 2.2.0 :tada:

The release is available on:

• npm package (@latest dist-tag)
• GitHub release

Your semantic-release bot :package::rocket: