[Feature][dora] Support DORA metrics

[klesh]

Search before asking

• [X] I had searched in the issues and found no similar feature requirement.

Description

The DORA metrics are very popular these days, but they are quite complicated to calculate. In order to support DORA metrics, I suggest that we create a set of plugins to do the job.

• The webhook plugins https://github.com/apache/incubator-devlake/issues/3023
• The dora plugin

Prototype: https://www.figma.com/file/Eux4Dhorxp7M0TwzOnL6Yh/Config-UI-for-V1.0?node-id=1447%3A11776

Idea

1. Define an interface PluginDashboards for the plugin to declare all granfana-dashboards that it could offer. including id, title, description, grafana_dashboard_json. The framework would call grafana API to make sure dashboards exist and are up to date
2. Define an interface PluginIcon for the plugin to return an SVG icon, (related to #2847) go embedf
3. The dora plugin would offer a set of subtasks for data-enrichment， i.e. to connect deployments and incidents, and a set of dashboards via PluginDashboards interface.
4. Another plugin webhook plugin would offer a set of APIs to support creating cicd_pipeline deployment and issue incident with curl

The Domain Layer Entities need to be updated (related to #2875):

• Add environment to cicd_tasks to indicate target environment, i.e. Production/Staging/Testing
• Add the following fields to pull_request
  • coding_timespan: from first commit author time to PR open time
  • review_lag: from PR open time to first review completed (this could be a comment or an approval)
  • review_timespan: first review completed to PR merged
  • deploy_timespan: PR merged to deploy
  • change_timespan: from first commit author time to deploy time
• Add deployment_id to issues: to indicate which cicd_pipeline caused the issue
• Update Domain Layer Document

The webhook plugin would provide the following APIs (related to #3023)

• Define connection struct as (id, name), with DYNAMIC GENERATED issue_endpoint, cicd_pipeline_endpoints
• POST /plugins/webhook/connections creates webhook connection
• GET /plugins/webhook/connections get all webhook connections
• GET /plugins/webhook/connections/:connection_id get detail of the webhook connection
• PATCH /plugins/webhook/connections/:connection_id update webhook connection
• DELETE /plugins/webhook/connections/:connection_id delete the webhook connection
• POST /plugins/webhook/:connection_id/cicd_pipeline creates cicd_pipline and cicd_pipeline_repos if not existed (ci/cd webhook)
• POST /plugins/webhook/:connection_id/issues creates issues (with type=INCIDENT normally) and board if not existed (incident webhook)
• POST /plugins/webhook/:connection_id/issues/:id/close close specified issue
• Add swagger document for webhook plugin

The dora plugin would contain the following subtasks

1. enrichTaskEnv for Deployment Frequency accepts argument environmentRegex and enrich cicd_tasks.environment against cicd_tasks.name, along with scope parameter (related to #2876 ) (notes: update environmentRegex to productionPattern, stagingPattern and testingPattern)

2. calcChangeLeadTime for Change Lead Time aka Lead Time for Changes to calculate newly added fields of pull_request(related to #2878 )

3. connectIssueCicdPipelineByTime for Change Failure Rate (depends on issue.type=incident) connects issues.deployment_id(type=incident) and cicd_pipelines based on time (assuming the incident is caused by the most recent deploymentunder the sameproject` maybe?) (related to #2879 )

4. Implement PluginDashboards

5. Implement PluginIcon

6. Add swagger document for pipeline/blueprint

Config-UI

• #2981

@yumengwang03 Please post the final design here https://www.figma.com/file/Eux4Dhorxp7M0TwzOnL6Yh/Config-UI-for-V1.0?node-id=1396%3A11624

Update existing data-source plugins and framework

• Update Gitlab to accept a task option named deployTagPattern and set type=DEPLOYMENT to cicd_task if matched #3114
• Update Jenkins to accept a task option named deployTagPattern and set type=DEPLOYMENT to cicd_task if matched #3122
• Update Github to accept a task option named deployTagPattern and set type=DEPLOYMENT to cicd_task if matched #3102
• Hardcode dora task in runner.RunPipeline if core.GetPlugin("dora") succeeded #3127

Use case

measuring DevOps quality and efficiency

Related issues

No response

Are you willing to submit a PR?

• [ ] Yes I am willing to submit a PR!

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[klesh]

@abeizn @likyh with me

[warren830]

Here is a question, lots of teams choose to squash commits, so the time of first commit will be a problem

[klesh]

Config-ui proposal