Create New page for /tools/

[iamVP7]

Latest description for the issue -> - list can still be enhanced manually as some tools are not on GitHub, there are some on GitLab

Reason/Context

• Why we need this improvement? Creating new index page for tools page.

• How will this change help? We can say what each and every tool does.

• What is the motivation? We are moving pages related to tools under /tools/* so a dedicated index page for tools alone

Description

• What changes have to be introduced? A new page.

• Will this be a breaking change? New page; mostly changes are restricted to new page. If broken it wll be mostly this page.

• How could it be implemented/designed?

[ritik307]

@iamVP7 can you provide the mockup for the page. I would love to work on this

[iamVP7]

Raised this feature request during code refactor (#378 )

I feel @derberg will be good person to share the thoughts on designing new page.

[quetzalliwrites]

I would ask our design contributor extraordinaire, @mcturco for design contributions. 😀👍🏽

Lukasz is awesome, but not in design 😄🙊

[derberg]

but not in design 😄🙊

I could not agree more 🤣

---

I'm not sure we are at the stage where we can start working on it. @alequetzalli isn't it somehow connected also with new docs page? I mean there will be docs restructuring, and we also want to incorporate docs from tools, and we also want to clean up that some tools are in one location and some are now described in the documentation. A lot of factors that can influence design, @mcturco will ask questions I personally do not have a clear answer to yet. This might be a good candidate for GSoC related task, as it requires discussions with community, and driving it end to end. Don't know 🤷🏼

[mcturco]

@derberg I was about to say the same thing. Before we can start developing a new page we definitely should discuss what the purpose is if different from the tooling docs and what content we would need to present. Once we have a clear direction and content written, we can begin designing a layout for this (again, if it will be different than the docs)

[quetzalliwrites]

I'm not sure we are at the stage where we can start working on it. @alequetzalli isn't it somehow connected also with new docs page?

I don't think so? I think I see the Docs as a different item than what I understood @iamVP7 was asking to do in this issue. I understand this issue to mean he wanted to know if we could add a page to our website to mention Tools. I guess I thought he opened this issue to ask if we could consolidate all our diverse tool landing pages into a single one. We currently have several individual pages or links to see the tools, but not a single one in the website just briefly summarizing each one. Thus, I thought it would be a question for say, @derberg, @mcturco, or maybe @magicmatatjahu to start a discussion w/ community on the why for this page.

That said, if I misunderstood his question, and he was instead asking about having the Tools documented, then yes.... this would be a question for Docs.

@iamVP7, can you confirm to us if you were wanting to document the Tools (for AsyncAPI Dev Docs), or if you were asking about making a landing page in our website for all the Tools?

[iamVP7]

As per @derberg comment, we were in need of landing page for tools.

Even I felt, it might be helpful to have landing page (say something like feature page for random project), as it can help with SEO.

[quetzalliwrites]

Got it, he may have forgotten that is what you were referring to! Thank you for clarifying @iamVP7.

So if you're proposing a landing page, which is not related to Docs, then I will let @derberg and @mcturco take over this with you. You'll prob work with them to propose your solution to the community and see what the community thinks this page would need. 😀👍🏽

[mcturco]

Great thank you both for the clarification @alequetzalli @iamVP7 I am understanding the idea now 😄

I think this is a great idea! It definitely will help to have that main /tools slug not reach a 404 😅 good catch! We will first have to start thinking about what kind of content we would need to include on this new page. Are there any other sites that we can use for inspiration? I think generally we can use this new page to showcase and link out to each tool, but just thinking about what else we can include on that page.

[quetzalliwrites]

@mcturco What if it had cards in a grid, and each card represented a tool?

[mcturco]

@alequetzalli yeah I think that's a great idea! Let's do it!

[derberg]

This issue is about having a single landing page that lists all the tools for AsyncAPI, not only the ones that are maintained by AsyncAPI:

We have tools:

• here https://www.asyncapi.com/docs/community/tooling
• here under Tools button/dropdown

We still need pages like:

• https://www.asyncapi.com/tools/cli
• or pages with full documentation, like we plan for generator for example

We should have one single landing page under https://www.asyncapi.com/tools where:

• we have a list of all the tools for AsyncAPI
• we have nice filtering and not hard to navigate tables like https://www.asyncapi.com/docs/community/tooling
• we can still have a category that given tool is maintained by AsyncAPI, so I can filter out only tools from AsyncAPI initiative
• filters should be enabled on URL level, so I can have for example a link like https://www.asyncapi.com/tools?asyncapi to get a list of tools that are owned by AsyncAPI Initiative
• I should be able to filter out tools by languages (JS, Java), area (CI&CD, testing) or type (library, cli, web)

Key - discoverability:

• now I can add a tool by opening a PR to the website repo but it requires extra effort that could be automated. Manual == boring and pushing away
• list should not be a markdown file like now but json or yaml file that is built by some GH workflow
• list is build based on GitHub search. Take this example. Arjun should just be required to only add certain topics to the repository, maintain accurate description, and that is it. We will automatically pull the tool to the list
• list can still be enhanced manually as some tools are not on GitHub, there are some on GitLab
• we should have some alerts when a new tool gets discovered? Slack channel alert? a maybe automated Tweet?

For now I think about simple repo description and topics, might be that long term better to introduce .asyncapi file that Arjun would need to maintain in his repo and all metadata for the tooling list would be provided there in the file.

For GSoC participants

This issue is interesting for GSoC because:

• you get to do some frontend work,
• JS coding and GitHub API usage is included
• You get to learn basics of GitHub Actions
• you will need to have discussions with different folks from AsyncAPI that will show you the real job work when in theory simple solution requires discussion with multiple folks in a company. The difference is that in open source it is even more complicated :smiley:

[akshatnema]

@derberg Hmm :thinking: this idea looks fascinating and interesting to me. I am designing the solution in my plan(for gsoc) and looking to make a proposal on it. But, I want more information regarding this idea. As discussed by you on Slack, you proposed me this:

To make a .asyncapi file from which data is extracted using GitHub API and is updated manually by organization members. So, where should we place this file? in website repo or .github repo so that it can be accessed to every repository and can be fetched easily from other repos?

Also, later we will have a prebuilt json file built using scripts but will be under .gitignore because the file will be built during build time and then rendered on the website (Same what we do in posts.json). This file will be updated using GH workflows on the production side. This is the overall plan you give me and I thought of. Also, I am actively looking for some more suggestions on this approach from everyone :smile:. We can have other approaches too for the listing of data. If you have anything more to correct me or suggest me, do tell me. It will be my pleasure to discuss with you.

Regarding filtering of tools in the list and design of the page, it is pretty much straightforward that whenever a user will filter out some tools, those filters will be updated on URLs as well as query params to search in the list. We can have a customized search bar at top of list, where categories are mentioned in a dropdown for users to search. New page (or route) /tools/ will be made easily in the repo and it can be further extended in the future to add more information related to it.

[akshatnema]

Hey @derberg, I am willing to continue to work on this issue in Mentorship program. Kindly look on my previous comment and do give your suggestions on how I should improve it.

[magicmatatjahu]

@akshatnema Hi!

To make a .asyncapi file from which data is extracted using GitHub API and is updated manually by organization members. So, where should we place this file? in website repo or .github repo so that it can be accessed to every repository and can be fetched easily from other repos?

As I understand that file should be located on the tool's repository side and then in website workflow we should filter all repos with that file and make appropriate logic. Let me know if you understand. Also I prefer .asyncapi-tool or .asyncapi-webiste due to simple fact - we can introduce .asyncapi config file in our CLI, so it's better to have a different name.

Also, later we will have a prebuilt json file built using scripts but will be under .gitignore because the file will be built during build time and then rendered on the website (Same what we do in posts.json). This file will be updated using GH workflows on the production side. This is the overall plan you give me and I thought of. Also, I am actively looking for some more suggestions on this approach from everyone 😄. We can have other approaches too for the listing of data. If you have anything more to correct me or suggest me, do tell me. It will be my pleasure to discuss with you.

We'd have to hear from Łukasz here, but from what I understand it's a good approach. However, as Łukasz mentioned you need to figure out a way to retrieve this data from the repositories via Github API, this will probably be the hardest part here.

Regarding filtering of tools in the list and design of the page, it is pretty much straightforward that whenever a user will filter out some tools, those filters will be updated on URLs as well as query params to search in the list. We can have a customized search bar at top of list, where categories are mentioned in a dropdown for users to search. New page (or route) /tools/ will be made easily in the repo and it can be further extended in the future to add more information related to it.

👌🏼 great.

[akshatnema]

you need to figure out a way to retrieve this data from the repositories via Github API, this will probably be the hardest part here.

@magicmatatjahu Yep, it is pretty much difficult to filter out the repos of the specific tools, but what I think is that if we assign specific tags to the repositories having tools, we can make a json fiile according to it. But the condition is, we need to specify tag, category or topics on the repo and each repo should contain exactly one tool in it. Using GitHub GraphQL API, we can extract the tags attached to each repository and store there information in the prebuilt.json file. The .asyncapi-tool file will contain the name of the topics, and categories that will be used to filter out the tools from the repositories. Additionally, some manual tools can also be added to asyncapi-tool file, which will be later added to the prebuilt.json file during build time.

What I didn't get is the GitHub search that Lukasz mentioned in his previous comment. Some Github search implementation has been made in the cupid repository by Arjun, but I wasn't able to find in it. If you know, kindly send me the reference link of that particular function in the codebase.

[magicmatatjahu]

@akshatnema

The .asyncapi-tool file will contain the name of the topics, and categories that will be used to filter out the tools from the repositories.

Yep, but it should contain also another info like description, title, maintainer etc. Of course we can also extract that info package.json, but we should discuss it.

What I didn't get is the GitHub search that Lukasz mentioned in his previous comment. Some Github search implementation has been made in the cupid repository by Arjun, but I wasn't able to find in it. If you know, kindly send me the reference link of that particular function in the codebase.

Łukasz specified cupid as repo for searching - given repo will have topics and .asyncapi-tool file and based on that we will filter repos :) It's only an example for your task.

[derberg]

@akshatnema I love how far detailed you went with the research. The final solution is not written in stone, like file names and stuff like that. The only strong requirement:

• not PR based but discovery based
• enable an easy way to add support to other platforms for discovery, like GitLab (I know about at least one AsyncAPI tool that is hosted there, docs generator for Asciidoc)

I'm not sure if topics is a better solution over .asyncapi file. IMHO best to propose 2 alternative solutions and show to some tooling owners (that will be users of this solution) to say what is the best for them. For sure .asyncapi is easier to extend in future with more metadata, more than you can get with topics

Super happy to see you engaged here so much. Will advertise it among TSC to make sure it is selected as it is going to be a great addition for the community IMHO

[akshatnema]

Hey @derberg, I understood your previous comment and thus, came up with another approach to target this. Do give your views on it. Here it is as follows:

• First part of the approach - We will be using Web scraping here on the JSON data provided by GitHub and GitLab APIs because we can only rely on these APIs since most of the tools are present here only (Do tell me if we have a centralized place/file to collect metadata of all tools we want to target in this issue). Using GraphQL APIs, we will get the description and other required information of all the repositories in AsyncAPI. This data(in JSON) will not be stored permanently but will be used for scraping purpose.

• Second part of the approach - The .asyncapi-tool file will contain the necessary tooling words which will be used to identify the required repositories containing the tools of AsyncAPI. The words or the keywords in the file will be manually added using the PRs and can be changed in future to more customize our scraping process as well as categorizing the various tools in different sections/topics. The file will also contain the other necessary information like the category names in which the tools are to be categorized on the website and the schema of the details needed for each tool that will be required while rendering it on website. In short, this file contains all the metadata related to the scraping process and the tools.

• Third part of the approach - The description (if needed, the topics) of all repositories containing the tool will be updated in such a way that they describe the tool as well as they get identified the scraper function to add its details (acquired from the API) in the tools.json (which will be built on the build time). The text in the description will be scanned on the given keywords in the .asyncapi-tool file and hence will be added in tools.json. This file is then used to render data in the respective components in the website.

Key points to consider

• Since, we are using web scraping, it's not going to be a fast process. Scanning around 60+ repositories in the JSON will take some time but since we are using JS, it will not affect the build time. (:thinking: Maybe it will take less time than build time of posts.json).
• This approach will not make some weird additions on the repository. As mentioned by you earlier, we won't be adding too many topics in the repository to identify the tool. We are using the Description of each repo for the identification and categorization.
• The description of each repository needs to be precise so that the scraper won't identify a non-tooling repository as a tool and add it in the tools,json.\
• I made the approach mentioned above keeping in mind that all the tools are present either in GitHub or GitLab. If it is not so, do tell me about the other sources. OR do we have any centralized file/place which keeps account of every tool used in AsyncAPI.
• An alternative for the description can be to use a certain prescribed file in the repositories that describes a tool and the file contains all the necessary topics/tags needed to identify and categorize it.

Waiting for an awesome reply from Lukasz (@derberg). I will be happier if @magicmatatjahu reviews this one as it deals with some code and algorithms as well :slightly_smiling_face:.

[magicmatatjahu]

I think you guys like to make life difficult for yourselves. When I read about the .asyncapi-tool approach I immediately had in mind situations where we use this file on the repository side to describe the tool (its metadata), and you describe about scraping data and identifying by description etc...

(Maybe you have the same thing in mind as me, but reading your previous comments I'm confused and don't know if I understood your suggestions correctly)

My suggestion:

• the owner of a given repository adds a topic (tag) to his/her repository asyncapi-tool.
• he/she creates an .asyncapi-tool file where provides information about the project/tool like description, title, maintainers, href, docs (documentation link), category (of tool, like template, parser etc), maybe also website etc.
• on building the website we download all the repositories from github using the asyncapi-tool topic, download the contents of the .asyncapi-tool file and we have all the metadata - for other sources like gitlab we should check if there is possibility for similar solution.

If someone provides additional topics he/she will be able to do that, but the most important thing is that there should be asyncapi-tool.

[derberg]

Definitely let us not consider web scrapping as a possible solution. GitHub search allows you to search for specific file in repos and as result, you get a list of repos. Then, afaik API allows you to just grab the content of the specific file, without cloning (something to confirm)

tl;dr

• we do not need topics at all
• all info for list of tools should be in .asyncapi-tool file
• just take advantage of GitHub API

[akshatnema]

Yep, we can extract the file contents but not directly through the Github search. What you mentioned in your message are the features used to find files/repos using the Github search bar but we need want it using Github API. I found this in the Github REST API. I think using the above-mentioned API, we can find the file present in the respective repo but need to access the content which is quite hard as the API is not returning the contents of the file directly. We need to discuss regarding this on how to extract the contents of the file using the present JSON. For finding the AsyncAPI repositories using API, we can use this GET request of GitHub API.

As mentioned by you, if we only need to add .asyncapi-tool file in each tool repository and can extract data from it, then it is quite an easy and straightforward implementation that can be implemented inside scripts/ folder. We only need a addition of .asyncapi-tool file added to each tool and if it is not possible, the tools will be manually added to the /tools/ in the website. Do you need to add anything more in this that I missed and is important?

[derberg]

• you have some example of search in action here: https://github.com/derberg/npm-dependency-manager-for-your-github-org/blob/main/lib/api-calls.js
• search returns 2 props, repository with typical repository object where you can grab org/profile and repo name + path where found file is actually located in repo, root? maybe .github?
• then you call for example https://api.github.com/repos/asyncapi/website/contents/README.md
• decode content string with base64

and enjoy life 🍹 🏖️

the only thing now is I think to agree on the structure of the .asyncapi-tool file

[akshatnema]

then you call for example api.github.com/repos/asyncapi/website/contents/README.md decode content string with base64

As what I see in the API docs and its schema, we get a url variable containing the link of the contents inside the file (in base64 encoded form). The schema of the url is as follows:

 "url": {
            "type": "string",
            "format": "uri"
          },

We can call the GET request on this url to get the contents and then convert them into the normal text then. Do confirm if you want to convey this only?

the only thing now is I think to agree on the structure of the .asyncapi-tool file

I will try to make one draft schema in the upcoming days but in the format of a javascript object (that's my most preferable way :sweat_smile:). If you want it in another format, do tell your suggestions.

[derberg]

We can call the GET request on this url to get the contents and then convert them into the normal text then. Do confirm if you want to convey this only?

sounds good to me

I will try to make one draft schema in the upcoming days but in the format of a javascript object (that's my most preferable way 😅). If you want it in another format, do tell your suggestions.

yeah, leave out schema for now, first focus on example JSON objects. Let us agree on the structure, and then to get schema out of it later is easy. Definitely do not start with schema first

[akshatnema]

Hey @derberg @magicmatatjahu can you please tell me some categories for the tools on which we can filter the tools?

I'm trying to make a JSON object for the .asyncapi-tool file.

[akshatnema]

I made a template JSON object for .asyncapi-tool but I think it still needs some more fields to be added. Do give your suggestions regarding the JSON object and schema.

{ 
    "title": "",
    "description": "",
    "maintainers": ["", ""],
    "links":{
        "url": "", // Website url of the tool. Ex: http://asyncapi/tools/cli
        "docs_url": "", // Documentation url of the tool.
        "repo_url": "" // Repository url of the tool
    },
    "category": {
        "languages": [""], // Like NodeJS, React JS
        "area": [""], //  Like CI/CD, testing
        "type": [""], // cli, parser, web, library
        "isAsyncAPIowner": true //boolean
    } 
}

[magicmatatjahu]

@akshatnema Nice! 🚀 My insights:

• I think that we should have shape for single maintainer like:

  {
    "name" : "Barney Rubble",
    "email" : "[email protected]",
    "url" : "http://barnyrubble.tumblr.com/"
  }
  

• about links.repo_url. If we will fetch data from Github API we will have url for repo, yes? So I don't think so that we need it.

• category. isAsyncAPIowner I don't think so that we need it, because we can check on the fetching side that given repo is under asyncapi organisation.

• maybe we should also add technology next to category.languages? For example, ReactJS is technology (library) not language.

The rest is good for me. @derberg What do you think?

[akshatnema]

I think that we should have shape for single maintainer like:

Fine, I'll update this change.

about links.repo_url. If we will fetch data from Github API we will have url for repo, yes? So I don't think so that we need it.

Yeah, it will not be needed in the .asyncapi-tool file but will surely be added in the tools.json file when compiling all the tools.

category. isAsyncAPIowner I don't think so that we need it, because we can check on the fetching side that given repo is under asyncapi organisation.

Hmm, it will be helpful if this field is added, because there will be need to add manually this field in the JSON object in the script.

maybe we should also add technology next to category.languages? For example, ReactJS is technology (library) not language.

Yeah, definitely :+1:

[derberg]

@akshatnema this idea was selected for AsyncAPI Mentorship 2022 -> https://github.com/asyncapi/community/discussions/376#discussioncomment-2890658

I love how much you are engaged with the work on AsyncAPI website and how active you are, especially in this issue. Would you like to be an official AsyncAPI-sponsored mentee for this issue?