feature: Expose Singer ecosystem activity data

[pnadolny13]

We have a private superset dashboard of Singer ecosystem activity. @tayloramurphy said in slack "This is awesome! I kind of want this just embedded on the Hub…". I wonder if we can find a low effort way of exposing some of this data on the hub.

I'm not super familiar with Evidence.dev yet but seems like a use case where we could compile some static markdown dashboards and serve them with the hub.

Theres probably other low effort ways of doing this too.

[aaronsteers]

@pnadolny13 - I love this idea. Evidence would be great for this! You'd write your report as markdown, with SQL queries powering the markdown template. Since it generates static HTML, we'd just need to output that in a way that it could be mounted to the Netlify-managed web content.

The hybrid deploy challenge is similar to what we are working through in

• https://github.com/meltano/hub/issues/954 and
• https://github.com/meltano/internal-general/issues/444

This would not be blocked by the above, but it might make sense to solve those first, and then see if the pattern is something that would make sense repeated here.

You definitely would not be blocked on those items in what you'd need to do to build the evidence.dev reports themselves. That could start whenever you have time and we would figure out how to serve them afterwards.

@edgarrmondragon - what do you think?

[edgarrmondragon]

@aaronsteers @pnadolny13 there's a couple of options:

Embed rendered Evidence.dev HTML

Evidence uses SvelteKit, which is a framework similar to Vue. It's possible to build those HTML files and serve them with the Hub site, in a similar way to how the API is built and served currently, just using spawn to call the svelte-kit CLI.

Also see https://github.com/evidence-dev/evidence/issues/445#issuecomment-1302350469

• Pros: we get benefits from Evidence.dev like existing connectors and graphs
• Cons: we mix frameworks, local dev experience wouldn't be great because we lose hot-reloading for the evidence parts of the site.

Use a (custom) Gridsome plugin

Similar to gridsome-source-mysql, we build a Gridsome plugin for Snowflake.

• Pros: we use a single framework for the site
• Cons: we need to write a plugin from scratch, we would need to write our own plotting components or use something from the Vue ecosystem

[archiewood]

This is a super interesting use case of Evidence. Where would you imagine the data being shown? On each page of relevant Meltano hub page? Or somewhere else?

[aaronsteers]

This is a super interesting use case of Evidence. Where would you imagine the data being shown? On each page of relevant Meltano hub page? Or somewhere else?

Hi, @archiewood. Good question.

I could see us going either direction with this. I imagine in a first iteration, it's probably harder to put a report in the existing web pages, and most likely less friction/difficulty if we create net-new pages. What we discuss above is related to integrating with the existing Hub.meltano.com site, but I'm wondering now if it makes sense to start with something fresh in https://github.com/meltano/squared, perhaps publishing to a squared repo GitHub pages target that is a non-production aspect of our marketing sites, available for viewing by our community members but not in the CI/build flow of hub.meltano.com.

Also, worth noting our current Hub CI pipelines don't get access to the SQL warehouse. We could potentially change that, but for now, it might be easier to build and iterate separately.