asyncapi
Make sure new parser releases are not introducing any regression
Reason/Context
Parser has many tests but not all cases are covered. https://github.com/asyncapi/tck/ was created to hold all the possible valid and invalid cases. We should make sure that with new releases the percentage of the parser compatibility doesn't decrease.
Description
Maybe we can add an action that runs tck on a PR and validates the numbers and block merge? tck outputs JSON with results so this should be relatively easy
This issue has been automatically marked as stale because it has not had recent activity :sleeping: It will be closed in 30 days if no further activity occurs. To unstale this issue, add a comment with detailed explanation. Thank you for your contributions :heart:
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
This issue has been automatically marked as stale because it has not had recent activity :sleeping: It will be closed in 30 days if no further activity occurs. To unstale this issue, add a comment with detailed explanation. Thank you for your contributions :heart:
This issue has been automatically marked as stale because it has not had recent activity :sleeping: It will be closed in 60 days if no further activity occurs. To unstale this issue, add a comment with detailed explanation. Thank you for your contributions :heart:
@derberg I want to work on this. I think tck is not available as an npm repository. If you make it available then we can install it, generate the results and write a test to validate the result? or something? If you could show me the way(or some similar thing that has been done) I can start working on it.
The trick here is to get tck and run against parser from PR and not the one specified in dependencies. We didn't do it anywhere yet. This is what needs to be figured out and I don't have a clear recommendation there as I haven't look into it that deeply, I'm not an author of tck. The options we have here:
- tck is not only a set of test documents but also a project where a number of parsers are integrated and there are runners for those parsers that generate overall reports. Easiest would be to reuse the runner and use parser, but not sure if it is possible without some improvement in the runner
- most probably we need a custom runner (which I don't think is super bad) specially designed for testing on PRs. Then on NPM we would only publish this directory https://github.com/asyncapi/tck/tree/master/tests/asyncapi-2.0. The advantage of publishing only asyncapi files is that we could also reuse the package for running snapshot testing for templates. Where to put this custom runner? 🤷🏼 this is open topic, to decide what is the best. Disadvantage of custom runner is that we would reinvent again the percentage calculation that we already have in the current runner
So as you can see there are 2 different solutions and you would have to first dig into the tck (not much code there really) and see what is better, improve existing code or actually write something new. Would be awesome if you decide working on that, that would help us a lot to improve testing. Once you decide working on it please first start with drafting a proposal on what would be the solution in your opinion
This issue has been automatically marked as stale because it has not had recent activity :sleeping: It will be closed in 60 days if no further activity occurs. To unstale this issue, add a comment with detailed explanation. Thank you for your contributions :heart:
This issue has been automatically marked as stale because it has not had recent activity :sleeping: It will be closed in 60 days if no further activity occurs. To unstale this issue, add a comment with detailed explanation. Thank you for your contributions :heart:
This issue has been automatically marked as stale because it has not had recent activity :sleeping:
It will be closed in 120 days if no further activity occurs. To unstale this issue, add a comment with a detailed explanation.
There can be many reasons why some specific issue has no activity. The most probable cause is lack of time, not lack of interest. AsyncAPI Initiative is a Linux Foundation project not owned by a single for-profit company. It is a community-driven initiative ruled under open governance model.
Let us figure out together how to push this issue forward. Connect with us through one of many communication channels we established here.
Thank you for your patience :heart:
This issue has been automatically marked as stale because it has not had recent activity :sleeping:
It will be closed in 120 days if no further activity occurs. To unstale this issue, add a comment with a detailed explanation.
There can be many reasons why some specific issue has no activity. The most probable cause is lack of time, not lack of interest. AsyncAPI Initiative is a Linux Foundation project not owned by a single for-profit company. It is a community-driven initiative ruled under open governance model.
Let us figure out together how to push this issue forward. Connect with us through one of many communication channels we established here.
Thank you for your patience :heart:
This issue has been automatically marked as stale because it has not had recent activity :sleeping:
It will be closed in 120 days if no further activity occurs. To unstale this issue, add a comment with a detailed explanation.
There can be many reasons why some specific issue has no activity. The most probable cause is lack of time, not lack of interest. AsyncAPI Initiative is a Linux Foundation project not owned by a single for-profit company. It is a community-driven initiative ruled under open governance model.
Let us figure out together how to push this issue forward. Connect with us through one of many communication channels we established here.
Thank you for your patience :heart:
This issue has been automatically marked as stale because it has not had recent activity :sleeping:
It will be closed in 120 days if no further activity occurs. To unstale this issue, add a comment with a detailed explanation.
There can be many reasons why some specific issue has no activity. The most probable cause is lack of time, not lack of interest. AsyncAPI Initiative is a Linux Foundation project not owned by a single for-profit company. It is a community-driven initiative ruled under open governance model.
Let us figure out together how to push this issue forward. Connect with us through one of many communication channels we established here.
Thank you for your patience :heart:
Should we have this issue opened when we have this one https://github.com/asyncapi/parser-js/issues/283? cc @derberg
I think it is separate. One thing is to integrate TCK (this issue) and the other is making sure parser is actually valid against it.
I would rather suggest to split https://github.com/asyncapi/parser-js/issues/283 into separate good first issues, with good description
Ok we will that one, but
I would rather suggest to split https://github.com/asyncapi/parser-js/issues/283 into separate good first issues, with good description
There will be a lot of these issues so I don't know if these should be good first issues if we have to fix/do a lot of things.
There will be a lot of these issues so I don't know if these should be good first issues if we have to fix/do a lot of things.
well it depends, probably case by case
This issue has been automatically marked as stale because it has not had recent activity :sleeping:
It will be closed in 120 days if no further activity occurs. To unstale this issue, add a comment with a detailed explanation.
There can be many reasons why some specific issue has no activity. The most probable cause is lack of time, not lack of interest. AsyncAPI Initiative is a Linux Foundation project not owned by a single for-profit company. It is a community-driven initiative ruled under open governance model.
Let us figure out together how to push this issue forward. Connect with us through one of many communication channels we established here.
Thank you for your patience :heart: