Proposal: build lightweight dashboard of jobs performance into Admin

[jgmize]

An approach similar to https://medium.com/@hakibenita/how-to-turn-django-admin-into-a-lightweight-dashboard-a0e0bbf609ad could be viable if we decided to store more granular performance summary data for each job instead of just the totals.

[jgmize]

Alternatively, we could potentially link to grafana dashboards using the same postgres storage

[glogiotatidis]

I'm strongly for building metrics outside the Django App -and especially the Admin- for a few of reasons:

• We 're going to need lots of boilerplate and libraries not currently in the project
• There 're tools designed for graphing and filtering data and we don't have to re-event the wheel
• Most importantly anything we build now can and will be obsoleted by new KPIs. More and new questions will be asked and new dashboards will be needed. Experience has proven that dealing with built-in dashboards has been painful for both coders and data consumers (and in the end not working anyway eh :)

On the other hand building outside the Django App will allow us to leverage existing projects (like Grafana or similar) and will make iterating and answering questions faster. Building a new Dashboard in Grafana will be quicker than html /css coding, reviewing and pushing to dev, staging, production.

I understand that building something with Python in Django is our comfort zone atm but this is also a good opportunity to explore new things.

This is an ongoing discussion, I'm just capturing here what we've already discussed with jgmize

[jgmize]

How about we start with private jupyter notebooks?

• with their own docker images
• deployed into the snippets-admin namespace
• can execute summary queries against all the data sources
  • snippets-service DB
  • BigQuery
  • Redshift
• merge & process the summary data in pandas dataframes
  • serialize the output to files (csv, json, parquet, etc) in s3 &/or GCS
  • save/export notebooks with graphs & summary data to s3 &/or GCS &/or github
    • also embedded data use case might be interesting to export to iodide notebooks
  • save relevant summary data in snippets-service RDS as desired for minimal admin numbers
    • later potential integration: link to saved/exported notebooks from admin
• extract relevant code to cron jobs for quarterly reports

[glogiotatidis]

Looks interesting. I've no experience with jupyter notebooks, let's try this out.