Creating a tool to extract metadata from ecosyste.ms

[Ly0n]

@andrew created an fantastic way to get the metadata of all listed projects via his platform: https://ecosyste.ms/ Here a simple visualization of the whole dataset: https://summary.ecosyste.ms/collections/1

For the next report we need a simple python tool to extract the data automatically into a table (CSV). Additionally, the tool allows us to identify which metadata is missing. This includes, among other things, organizational data. Afterwards we have to decide whether it is in the interest of ecosyste.ms to provide this data via the platform or whether we should write our own tool that obtains the data via the GitHub API.

[jmertic]

We should compare what is there vs what is in the landscape. Ideally, aligning on one tool would be best.

[Ly0n]

The ecosyste.ms platform is truly exceptional. @andrew developed the platform based on his experience with Libraries.io, which is now unfortunately no longer maintained under the hood of Tidelift.

Don't be fooled by the minimalist front-end as the platforms are not officially released yet. From his platform we can get the download numbers from almost all package managers and all the dependencies of the projects. The website could become the central search engine for open source projects. The entire significantly open source ecosystem could benefit from it.

I absolutely agree with you that we need to look at how we can bring the best of all 3 platforms together. Ecosyste.ms, OpenSustain.tech and the Landscape Tooling. We should have our own meeting on the topic. Perhaps Andrew could also give a brief overview of the capabilities of his platform.

[jmertic]

Hey - sorry if I came across diminishing the tool; lots of great work has gone into this for sure and we certainly appreciate the work put into it.

I would agree seeing how we could align/integrate makes a ton of sense. I'd love to get the landscape developers involved as well.

[Ly0n]

@jmertic We also have to ask ourselves whether we write the Ecosystem Analysis Tooling just for us or as a generic open source tool that can be reused by others.

I think it is little extra effort to develop it directly and correctly as a generic project. As a description for an ecosystem, one must then enable different standards as input:

1. LF Interactive Landscape yml files.
2. awesome lists
3. dependency trees.

We should talk about this closely with Andrew and the landscape developers.

[andrew]

This includes, among other things, organizational data. Afterwards we have to decide whether it is in the interest of ecosyste.ms to provide this data via the platform or whether we should write our own tool that obtains the data via the GitHub API.

I've added a new section to each project in the summary service for the owner of each project which could be an organization or an individual, example:

which is also available in the json representation: https://summary.ecosyste.ms/api/v1/projects/966:

[jmertic]

@andrew - I sent you a CNCF Slack invite to connect you with @AndreyKozlov1984, who can help you leverage landscape as a data source; the infrastructure with landscape should help some of the lift.