feat: add new custom resolver to fetch the reference from private repo

[AayushSaini101]

Resolves: #1796

• Create a custom resolver to fetch the reference from the private repo
• Add new command auth add for storing the repo name and repo token.
• Tests and documents need to be updated

[github-actions[bot]]

Changeset has been generated for this PR as part of auto-changeset workflow.

Please review the changeset before merging the PR.

---
'@asyncapi/cli': minor
---

feat:  add new custom resolver to fetch the reference from private repo

- 0213926: Add new custom resolver to fetch the reference from private template and add new command auth add
- 26037eb: Merge remote-tracking branch 'remote/master' into 1796-b
- 6473326: chore: separate cli and api

Signed-off-by: Shurtu-gal <[email protected]>
- d484f2f: update
- ab58365: remove invalid changes
- 932c09c: update package-lock.json
- ec5eb66: update package-lock.json
- 7e4b523: fix github jobs
- c7501a1: Fix elint issue
- 925b90a: remove uncessary files
- 5d651f7: remove invalid file
- 5d76066: update-changeset
- 5d89ce7: update-doc
- 3461288: update-suggestions
- 4f63af9: add new testcase
- 5d83ac7: add new testcases
- f1a4d39: add new test case for the validationService
- af11ce4: update test case
- 824bdb9: update changeset
- cf4d070: Update src/domains/services/validation.service.ts

Co-authored-by: Fran Méndez <[email protected]>
- 150865c: update validation service for the github
- 4c5be56: Update src/domains/services/validation.service.ts

Co-authored-by: Fran Méndez <[email protected]>
- fcc864d: Update src/apps/api/middlewares/validation.middleware.ts

Co-authored-by: Fran Méndez <[email protected]>
- 2f69485: Update src/domains/services/validation.service.ts

Co-authored-by: Fran Méndez <[email protected]>
- 5328d62: Update src/domains/services/validation.service.ts

Co-authored-by: Fran Méndez <[email protected]>
- 407bacc: Update src/domains/services/validation.service.ts

Co-authored-by: Fran Méndez <[email protected]>
- 3b7159b: Update src/domains/services/validation.service.ts

Co-authored-by: Fran Méndez <[email protected]>
- f601cf7: Update src/domains/services/validation.service.ts

Co-authored-by: Fran Méndez <[email protected]>
- 8943766: updateValidationservice
- aa37d14: remove invalid comment
- 3b882ed: Add new custom resolver to fetch the reference from private template and add new command auth add

# Conflicts:
#	package-lock.json
#	src/core/parser.ts
- 811bc08: update

# Conflicts:
#	asyncapiTest.yml
#	index.js
#	src/apps/api/middlewares/validation.middleware.ts
#	src/apps/cli/commands/diff.ts
#	src/apps/cli/commands/generate/models.ts
#	src/apps/cli/commands/validate.ts
#	src/domains/services/config.service.ts
#	src/domains/services/validation.service.ts
- bb0fb6c: remove config service test file
- 32298a9: remove invalid statements
- ad3b3b8: add missing dependencies
- 7662fca: remove warnings
- 40b4b50: update validation-service
- d7db432: remove invalid file
- 98dfd4c: fix lint issue
- 75dda7b: update validation-service
- 5c23530: fix lint issue

If you are a maintainer or the author of the PR, you can change the changeset by clicking here

[!TIP] If you don't want auto-changeset to run on this PR, you can add the label skip-changeset to the PR or remove the changeset and change PR title to something other than fix: or feat:.

[changeset-bot[bot]]

🦋 Changeset detected

Latest commit: d6d0402eabaad16c872678af1801c6cfdddc950a

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
@asyncapi/cli	Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[AayushSaini101]

Tests and documentation need to be updated, i will update after getting ack from this approach : )

• Spec containing private repo:

• Trying to parse using the existing version of the asyncapi cli:

New feature:

• Add auth value for the github token:

Repo URL format: https://github.com/AayushSaini101/AayushSaini101/tree/main

• Trying to fetch the private ref:

cc: @fmvilas @Souvikns @Shurtu-gal

[AayushSaini101]

/u

[sonarqubecloud[bot]]

Quality Gate passed

Issues
5 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[AayushSaini101]

@fmvilas For the private Github URL should be in this format: like https://api.github.com/repos/AayushSaini101/AayushSaini101/contents/user-signedup.yaml, not direct URL like we simple copy the path of the URL from the private repo, this is the github url, not sure about the private repo, in short we need to compare each and every url and create special check to find whether it is matching with the stored pattern or not, we can start with the github and gitlab pattern then we can slowly slowly add the comparison format as we grow thanks

[aeworxet]

@asyncapi/bounty_team

[AayushSaini101]

/u

[fmvilas]

@fmvilas For the private Github URL should be in this format: like https://api.github.com/repos/AayushSaini101/AayushSaini101/contents/user-signedup.yaml, not direct URL like we simple copy the path of the URL from the private repo, this is the github url, not sure about the private repo, in short we need to compare each and every url and create special check to find whether it is matching with the stored pattern or not, we can start with the github and gitlab pattern then we can slowly slowly add the comparison format as we grow thanks

@AayushSaini101 I'm not sure I follow. I think you can translate from one to another, right? For instance, if I give you https://github.com/AayushSaini101/a-repo/blob/master/asyncapi.yaml (which is the URL I can copy from the browser), you can transform it to https://api.github.com/repos/AayushSaini101/a-repo/contents/asyncapi.yaml or whatever format you need it to be, right? The idea is to make it really easy for the user.

Likewise, if my company has a Github installation behind the https://source.mycompany.com URL, then a URL like https://source.mycompany.com/AayushSaini101/a-repo/blob/master/asyncapi.yaml should also be possible to resolve.

In short, picture this, you're working at a company and have an AsyncAPI or JSON/YAML file in a private repo. You want to reference it using $ref and it should just work out of the box. Sure, the user has to provide the necessary details to resolve these URLs but it should be fairly easy once it's configured.

[AayushSaini101]

github.com/repos/AayushSaini101/a-repo/contents/asyncapi.yaml

yes, i can tranform/configured the URL for the Github because i know the format for the accepted URL in the case of the Github, but for the case of another services like registry or any private registry what should be the correct URL that we need to consider for the custom resolver, that's why for this issue we can enable feature Github then in future according to the requirements we can add bit by bit @fmvilas

[AayushSaini101]

/u

[AayushSaini101]

@fmvilas For the private Github URL should be in this format: like https://api.github.com/repos/AayushSaini101/AayushSaini101/contents/user-signedup.yaml, not direct URL like we simple copy the path of the URL from the private repo, this is the github url, not sure about the private repo, in short we need to compare each and every url and create special check to find whether it is matching with the stored pattern or not, we can start with the github and gitlab pattern then we can slowly slowly add the comparison format as we grow thanks

@AayushSaini101 I'm not sure I follow. I think you can translate from one to another, right? For instance, if I give you https://github.com/AayushSaini101/a-repo/blob/master/asyncapi.yaml (which is the URL I can copy from the browser), you can transform it to https://api.github.com/repos/AayushSaini101/a-repo/contents/asyncapi.yaml or whatever format you need it to be, right? The idea is to make it really easy for the user.

Likewise, if my company has a Github installation behind the https://source.mycompany.com URL, then a URL like https://source.mycompany.com/AayushSaini101/a-repo/blob/master/asyncapi.yaml should also be possible to resolve.

In short, picture this, you're working at a company and have an AsyncAPI or JSON/YAML file in a private repo. You want to reference it using $ref and it should just work out of the box. Sure, the user has to provide the necessary details to resolve these URLs but it should be fairly easy once it's configured.

@fmvilas User need to provide the direct valid url in the asyncapi spec that it https://api.github.com/repos/AayushSaini101/a-repo/contents/asyncapi.yaml , so that it would be easy to direct authenticate without chaning in any other kind of format. wdyt ?

[AayushSaini101]

@fmvilas For the private Github URL should be in this format: like https://api.github.com/repos/AayushSaini101/AayushSaini101/contents/user-signedup.yaml, not direct URL like we simple copy the path of the URL from the private repo, this is the github url, not sure about the private repo, in short we need to compare each and every url and create special check to find whether it is matching with the stored pattern or not, we can start with the github and gitlab pattern then we can slowly slowly add the comparison format as we grow thanks

@AayushSaini101 I'm not sure I follow. I think you can translate from one to another, right? For instance, if I give you https://github.com/AayushSaini101/a-repo/blob/master/asyncapi.yaml (which is the URL I can copy from the browser), you can transform it to https://api.github.com/repos/AayushSaini101/a-repo/contents/asyncapi.yaml or whatever format you need it to be, right? The idea is to make it really easy for the user. Likewise, if my company has a Github installation behind the https://source.mycompany.com URL, then a URL like https://source.mycompany.com/AayushSaini101/a-repo/blob/master/asyncapi.yaml should also be possible to resolve. In short, picture this, you're working at a company and have an AsyncAPI or JSON/YAML file in a private repo. You want to reference it using $ref and it should just work out of the box. Sure, the user has to provide the necessary details to resolve these URLs but it should be fairly easy once it's configured.

@fmvilas User need to provide the direct valid url in the asyncapi spec that it https://api.github.com/repos/AayushSaini101/a-repo/contents/asyncapi.yaml , so that it would be easy to direct authenticate without chaning in any other kind of format. wdyt ?

@fmvilas can you check the last question, i guess then after we can resume on this thanks a lot : )

[fmvilas]

@fmvilas User need to provide the direct valid url in the asyncapi spec that it https://api.github.com/repos/AayushSaini101/a-repo/contents/asyncapi.yaml , so that it would be easy to direct authenticate without chaning in any other kind of format. wdyt ?

Honestly, I think that would be good enough but since we're still here designing the feature, enabling the user to simply put URLs like https://github.com/AayushSaini101/a-repo/blob/master/asyncapi.yaml (which is the URL they can copy from the browser) would produce a higher DX. That's not to say the user can't put the https://api.github.com/repos/AayushSaini101/a-repo/contents/asyncapi.yaml URL. An amazing DX is one that anticipates this kind of things IMHO. And it's really a low effort thing.

[AayushSaini101]

@fmvilas User need to provide the direct valid url in the asyncapi spec that it https://api.github.com/repos/AayushSaini101/a-repo/contents/asyncapi.yaml , so that it would be easy to direct authenticate without chaning in any other kind of format. wdyt ?

Honestly, I think that would be good enough but since we're still here designing the feature, enabling the user to simply put URLs like https://github.com/AayushSaini101/a-repo/blob/master/asyncapi.yaml (which is the URL they can copy from the browser) would produce a higher DX. That's not to say the user can't put the https://api.github.com/repos/AayushSaini101/a-repo/contents/asyncapi.yaml URL. An amazing DX is one that anticipates this kind of things IMHO. And it's really a low effort thing.

Thanks for the suggestions @fmvilas, from now i am resuming the pending work on this PR that is allowing private pull from the Github Repo : )

[AayushSaini101]

/u

[AayushSaini101]

/u

[AayushSaini101]

Summary for all the conversations

• For every private URL that the user passes in the Asyncapi document has to be normalised in the particular format in order to fetch the information properly, in order to maintain the better DX we will allow the user to pass the default URL

For example in the case of Github Private URL we have to convert into normalised URL: Spec private URL: 'https://github.com/AayushSaini101/AayushSaini101/blob/main/user-signedup.yaml#/payload Normalised URL: https://api.github.com/repos/${owner}/${repo}/contents/${path}?ref=${branch};

The function should be generic so that in the future, we can extend it properly

• No need to use another parameter for the customAuth
• The format of the secerts: asyncapi config auth add github.com/myorg/myrepo/**/. bearer super-secret-here
• All the secrets should be stored in the file so that we can check whether the secrete is present in the file or not

This is all the final requiments : cc: @fmvilas

[fmvilas]

For every private URL that the user passes in the Asyncapi document has to be normalised in the particular format in order to fetch the information properly, in order to maintain the better DX we will allow the user to pass the default URL

This only applies for GitHub URLs. For all other URLs, private or not, the URL shouldn't change.

[AayushSaini101]

@fmvilas completed the implementation and tested also with cc: @neoandmatrix

[Shriya-Chauhan]

@AayushSaini101 Tested it. Working fine.

[neoandmatrix]

@fmvilas completed the implementation and tested also with cc: @neoandmatrix

Yes @fmvilas tested this and it's working fine.

[AayushSaini101]

I think you should, at least, test that the resolver is working well for some GitHub URLs and also that the resolver is not interfering with other non-GitHub URLs.

Added some test cases thanks ; )

[AayushSaini101]

/u

[fmvilas]

@AayushSaini101 let me know if you need guidance.

[AayushSaini101]

/u

[AayushSaini101]

/u

[AayushSaini101]

/u

[AayushSaini101]

@AayushSaini101 let me know if you need guidance.

@fmvilas tried to added some test cases thanks: : )

[fmvilas]

The code looks fine to me. Just wanted to know the use cases for this (is it just limited to github private repo).

To clarify, this is not just for GitHub private repos. The URL translation feature is just for GitHub though.

Also @fmvilas what should be the proposed way of testing the auth?

We should mock the resolver and fake the status code and response we get. We should be testing that unless we pass the Authorization header with a specific value (a random value), the resolver will respond with 401 or 404. Basically, imitate what a real server would do without making the actual HTTP call.