incubator-devlake icon indicating copy to clipboard operation
incubator-devlake copied to clipboard
verified publisher icon apache

[Feature][framework] CI/CD Domain Layer Entities Design & Development

Open klesh opened this issue 3 years ago • 3 comments

Search before asking

  • [X] I had searched in the issues and found no similar feature requirement.

Description

Where we at

The following status is how we support the CI/CD data analysis :

  1. We collect Gitlab Pipeline data into Tool Layer as _tool_gitlab_pipeline
  2. We collect Jenkins data into both Tool Layer and Domain Layer as _tool_jenkins_jobs _tool_jenkins_builds jobs and builds

In another word, we modeled the CI/CD Domain Layer Tables based on Jenkins, and the Gitlab Pipeline data never get into Domain Layer because they are quite different in terms of modeling. This is also true for other kinds of CI/CD systems.

Why we need it

However, CI/CD plays an important role during the R&D process, metrics like Build Success Rate, etc (@Startrekzky please add more examples) depend on it.

What is the problem

CI/CD modeling varies in different platforms, there is no de facto modeling. It is hard to design a sensible model for CI/CD Domain Layer Data.

How should we approach

I propose that we address the problem by the following step:

  1. Set up a couple of designated charts depending on CI/CD data that we would like to have @Startrekzky
  2. Assign a Veteran Developer to investigate the modeling of the most popular platforms, such as Github/Gitlab/Jenkins/ArgoCD, and propose a CI/CD Domain Layer Table design
  3. The design shall be evaluated by the PPMC members in 3 workdays while all Committers could share their opinions
  4. A Data-Oriented PoC (a set of tables with data from different platforms) should be presented for the Analyst (@Startrekzky @leglar @hezyin) to evaluate
  5. And then we implement

Use case

  1. We may create Standard CI/CD charts based on Unified Domain Layer Tables
  2. Users may exploit the Unified CI/CD Domain Layer Tables to create customized charts

Related issues

No response

Are you willing to submit a PR?

  • [ ] Yes I am willing to submit a PR!

Code of Conduct

klesh avatar Jul 19 '22 07:07 klesh

CI/CD Metrics：

  1. No. of Builds (Similar to Deployment Frequency- D in DORA, which can be interpreted as the total No. of builds that meet a specific condition)
  2. Build success rate %
  3. Build result distribution %
  4. Build duration
  5. Mean Lead Time for Changes (Depending on the definition of a change, it may be a “build” that meets specific conditions.)
  6. Mean Time to Recovery (MTTR) (Depending on the definition of recovery, it may start when a “bug” is created, and end when it's fixed.)
  7. Change Failure Rate (Depending on the definition of a change and failure, it may be the failing percentage of “build” that meets specific conditions.)

Analysis Dimensions

  1. time: e.g. measure the average build duration over the calendar date
  2. job: e.g. compare the average build duration under different jobs
  3. repo (code base): e.g. compare the average build duration triggered by different repos

Startrekzky avatar Jul 19 '22 10:07 Startrekzky

I created three issues for separate CI/CD tools. We will investigate those tools first, and after a new discussion, we will implement the collector and extractor of the plugins.

  • https://github.com/apache/incubator-devlake/issues/2666
  • https://github.com/apache/incubator-devlake/issues/2637
  • https://github.com/apache/incubator-devlake/issues/2583
  • https://github.com/apache/incubator-devlake/issues/2584
  • https://github.com/apache/incubator-devlake/issues/2585
  • https://github.com/apache/incubator-devlake/issues/2596
  • https://github.com/apache/incubator-devlake/issues/2604

warren830 avatar Jul 22 '22 14:07 warren830

According to our discussion from mailing list, Metrics to be considered:

  1. No. of Builds (Similar to Deployment Frequencyhttps://linearb.io/blog/dora-engineering-metrics/#deployment-frequency- D in DORA, which can be interpreted as the total No. of builds that meet a specific condition)
  2. Build success rate %
  3. Build result distribution %
  4. Build duration in hour -- should this be minute? @Startrekzky
  5. Mean Lead Time for Changeshttps://linearb.io/blog/dora-engineering-metrics/#mean-lead-time-for-changes (Depending on the definition of a change, it may be a “build” that meets specific conditions. measure it beginning from merging the code to the main branch.)
  6. Mean Time to Recovery (MTTR)https://linearb.io/blog/dora-engineering-metrics/#mean-time-to-recovery-mttr (Depending on the definition of recovery, it may start when a “bug” is created, and end when it's fixed.)
  7. Change Failure Ratehttps://linearb.io/blog/dora-engineering-metrics/#change-failure-rate (Depending on the definition of a change and failure, it may be the failing percentage of “build” that meets specific conditions. Normally, this metric is calculated by counting the number of times a deployment results in a failure and dividing by the number of total deployments to get an average number. A lower average is better.)

Analysis Dimensions to be considered:

  1. time: e.g. measure the average build duration over the calendar date
  2. job: e.g. compare the average build duration under different jobs
  3. repo (code base): e.g. compare the average build duration triggered by different repos

In addition, the naming of tables should avoid confusion for users of any CI/CD tools, such as Jenkins, GitLabCI, GitHub Actions or CircleCI. For example, job means different entity in GitLabCI and Jenkins, thus we should consider not to use it in the domain layer.

We also need to consider Azure DevOps/CircleCI/Travis CI/AWS CodePipeline

warren830 avatar Jul 25 '22 01:07 warren830

This ticket can be closed for now

warren830 avatar Aug 25 '22 01:08 warren830