spring-cloud-alibaba Spring Cloud Alibaba Auto-Test Design | Spring Cloud Alibaba自动化测试方案设计

Spring Cloud Alibaba Auto-Test Design | Spring Cloud Alibaba自动化测试方案设计

Open kaori-seasons opened this issue 1 year ago • 2 comments

背景

相关议题: https://github.com/alibaba/spring-cloud-alibaba/issues/2566 目前我们需要为项目设计一套集成测试方案,为解决版本发布造成用户的issue复现困难，基于testcontainer + Github Action进行测试，以便于覆盖spring-cloud-alibaba每次接入新组件特性时候的用例，提升项目稳定性

设计方案

在社区年初的尝试当中,通过先在需要进行集成测试的类上标记 @HasDockerAndItEnabled 注解的方式 , 在Github Action当中执行mvn clean -Dit.enabled=true test, 运行所有集成测试

但是这种方式在集成测试当中，无法对测试的配置初始化, bean的预加载进行准备。并且针对 nacos和 rocketmq 的测试，需要保证在测试运行之前，保持相关testcontainer的测试镜像稳定运行并测试, 所以目前对于Github Action的落地有两种方案可供讨论:

方案一 junit5 callback + docker-compose 目前的PR 是基于 org.junit.jupiter.api的回调函数在用户运行测试之前，从上下文里面获取docker-compose文件的信息。然后在callback函数内完成容器运行. 之后执行测试
方案二 maven Profile + dockerfile-maven-plugin构建出Docker镜像和完成推送 + docker Load加载打包 spring-cloud-alibaba-tests 的构建产物 spring-cloud-alibaba-tests.tar包:
通过参考shardingsphere的CI方式, 构建SCA的CI环境

shardingsphere构建步骤
- 部署镜像
- Docker持续集成
目前实现的方式分为两步骤:
- 登陆dockerhub，将nacos, rocketmq, sentinel ,seata等镜像放在同一个dockerhub账号下,执行集成测试时，拉取相关镜像(默认是在alibaba的容器镜像服务当中，所以在docker.yml中会在alibaba仓库中搜索)
- 将待测试的类打包成一个jar，通过 docker load执行归档文件
目前方案一目前处于poc阶段，方案二处于测试阶段

设计思路以及折中

Github Action
- 方案一:
  - 部署镜像并进行docker集成测试

name: Integration Testing
on:
  push:
    branches:
      - 2021.x
  pull_request:
    branches:
      - 2021.x
jobs:
  deploy-docker-image:
    runs-on: ubuntu-latest
    timeout-minutes: 60
    steps:
      - name: Check out the repo
        uses: actions/checkout@v2

      - name: Set up QEMU
        uses: docker/setup-qemu-action@v1

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v1

  integration-testing:
    name: Integration Testing
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: Set up JDK 8
        uses: actions/setup-java@v2
        with:
          java-version: '8'
          distribution: 'adopt'
      - name: Dependies Cache
        uses: actions/cache@v2
        with:
          path: ~/.m2/repository
          key: ${{ runner.os }}-maven-${{ hashFiles('**/pom.xml') }}
          restore-keys: |
            ${{ runner.os }}-maven-
      - name: Compile & Checkstyle
        run: mvn clean compile
      - name: Testing
        run: mvn clean test
#        run: mvn clean -Dit.enabled=true test

方案二:
- 部署镜像

name: Deploy Docker Image

on:
  push:
    branches:
      - 2021.x
      paths:
        - '**/pom.xml'
        - '**/src/main/**'
      pull_request:
        branches:
          - 2021.x
        paths:
          - '**/pom.xml'
          - '**/src/main/**'
  release:
    types:
      - published
  workflow_dispatch:

env:
  MAVEN_OPTS: -Dhttp.keepAlive=false -Dmaven.wagon.http.pool=false -Dmaven.wagon.http.retryHandler.count=3 -Dmaven.wagon.httpconnectionManager.ttlSeconds=30 -DskipTests

jobs:
  deploy-docker-iamge:
    runs-on: ubuntu-latest
    timeout-minutes: 60
    steps:
      - uses: actions/checkout@v2

      # setup docker
      - name: Set up QEMU
        uses: docker/setup-qemu-action@v1

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v1

      - name: Login to DockerHub
        uses: docker/login-action@v1
        with:
          username: ${{ secrets.DOCKERHUB_USERNAME }}
          password: ${{ secrets.DOCKERHUB_TOKEN }}

      # build target with maven
      - name: Cache Maven Repos
        uses: actions/cache@v2
        with:
          path: ~/.m2/repository
          key: ${{ runner.os }}-maven-${{ hashFiles('**/pom.xml') }}
          restore-keys: |
            ${{ runner.os }}-maven-
      - name: Set up JDK 8
        uses: actions/setup-java@v1
        with:
          java-version: 8

      - name: set environment
        run: export MAVEN_OPTS=' -Dmaven.javadoc.skip=true -Drat.skip=true -Djacoco.skip=true $MAVEN_OPTS'
      - name: Build Project
        run: |
          ./mvnw -B -Prelease,docker -DskipTests clean install
          docker image ls --format "{{.ID}} {{.Repository}} {{.Tag}}" | grep alibaba| sed  's/apache\//${{ secrets.DOCKERHUB_USERNAME }}\//' |tr A-Z a-z |awk '{system("docker tag "$1" "$2":latest;docker tag "$1" "$2":"$3";")}'
      - name: Push Docker Image
        run: |
          echo Docker Images:
          echo `docker image ls|grep -i ${{ secrets.DOCKERHUB_USERNAME }}|awk '{print $1":"$2}'`
          docker image ls|grep -i ${{ secrets.DOCKERHUB_USERNAME }}|awk '{print $1":"$2}'|xargs -i docker push {}

docker集成测试

           name: Integration Testing
on:
  push:
    branches:
      - 2021.x
    paths:
      - '.github/workflows/integration-test.yml'
      - '**/pom.xml'
      - '**/src/main/**'
  pull_request:
    branches:
      - 2021.x
    paths:
      - '.github/workflows/integration-test.yml'
      - '**/pom.xml'
      - '**/src/main/**'
jobs:
  build-it-image:
    name: build-it-image
    runs-on: ubuntu-latest
    timeout-minutes: 20
    steps:
      - uses: actions/checkout@v2
      - name: Maven resolve ranges
        run: ./mvnw versions:resolve-ranges -ntp -Dincludes='org.springframework:*,org.springframework.boot:*'
      - name: Cache Maven Repos
        uses: actions/cache@v2
        with:
          path: ~/.m2/repository
          key: spring-cloud-alibaba-it-cache-${{ github.sha }}
          restore-keys: |
            ${{ runner.os }}-maven-
      - name: Set up JDK 8
        uses: actions/setup-java@v2
        with:
          distribution: 'temurin'
          java-version: 8
      - name: Build IT image
        run: ./mvnw -B clean install -am -pl spring-cloud-alibaba-tests/spring-cloud-alibaba-testcontainers -Pit.env.docker -DskipTests -Dspotless.apply.skip=true
      - name: Save IT image
        run: docker save -o /tmp/spring-cloud-alibaba-testcontainers.tar apache/spring-cloud-alibaba-testcontainers:latest
      - name: Upload IT image
        uses: actions/upload-artifact@v3
        with:
          name: it-image
          path: /tmp/spring-cloud-alibaba-testcontainers.tar
          retention-days: 1

  integration-testing:
    name: Integration Testing
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: Set up JDK 8
        uses: actions/setup-java@v2
        with:
          java-version: '8'
          distribution: 'adopt'
      - name: Dependies Cache
        uses: actions/cache@v2
        with:
          path: ~/.m2/repository
          key: ${{ runner.os }}-maven-${{ hashFiles('**/pom.xml') }}
          restore-keys: |
            ${{ runner.os }}-maven-
      - name: Compile & Checkstyle
        run: mvn clean compile
      - name: Testing
        run: mvn clean test
#        run: mvn clean -Dit.enabled=true test
      - name: Download IT image
        uses: actions/download-artifact@v3
        with:
          name: it-image
          path: /tmp/
      - name: Load IT image
        run: docker load -i /tmp/spring-cloud-alibaba-testcontainers.tar
      - name: "run install by skip tests"
        if: ${{ steps.check_changes.outputs.docs_only != 'true' }}
        run: mvn -q -B -ntp clean install -DskipTests
      - name: run integration tests
        if: ${{ steps.check_changes.outputs.docs_only != 'true' }}
        run: ./build/run_integration_group.sh CLI

如何添加测试用例
- 方案一: 添加@SpringCloudAlibaba以及@TestExtend两个注解，在测试用例当中分为构建一个@BeforeEach标记的静态方法以及@BeforeAll标记的非静态方法, 如果测试线程的执行时间较长，可以修改@TestExtend里面的timeout属性
- 方案二: 当Github Action执行完 spring-cloud-alibaba-tests.tar测试jar打包，镜像构建推送等流程之后,基于脚本对于-Denv=docker的环境做测试(可以在PR中定位到 run: ./build/run_integration_group.sh CLI这一行) , 运行打包的集成测试方法

方案对比

拓展性对于方案一而言：由于没有将测试统一打包执行，所以每运行一个测试就要从github拉一次镜像对于方案二而言：镜像的构建和推送统一交给Github Action做管理。测试统一打包后执行不需要每次进行镜像构建和推送
可维护性对于方案一而言: 会随着测试的增加，延长每次用户提交PR时触发ci的时间，属于短线规划对于方案二而言: 统一管理镜像的构建和推送收敛测试范围，每次用户提交PR时触发ci时间可控, 可通过脚本拓展分组测试

Aug 10 '22 05:08 kaori-seasons

如果有任何建议或者希望社区实现的自动化测试方案可以在这里讨论

Aug 10 '22 06:08 kaori-seasons

Englist Version：

Background

At present, we need to design a set of integration testing solutions for the project. In order to solve the problem of user issue reproduction caused by version release, we will test based on testcontainer and Github Action, so as to cover the use cases of spring-cloud-alibaba every time when new component features are connected, to improve project stability.

Program Design

In the community's attempt at the beginning of the year, marking the @HasDockerAndItEnabled annotation on the class that need integration testing, executing mvn clean -Dit.enabled=true test in Github Action to run all integration tests.

However, in this way, in the integration test the configuration initialization of the test and the preloading of the bean cannot be prepared. And for the test of nacos and rocketmq, it is necessary to ensure that the test image of the relevant testcontainer is kept running and tested stably before the test is run. Therefore, there are currently two options for the implementation of Github Action:

Solution One: junit5 callback + docker-compose.

[Current PR](https://github.com/alibaba/spring-cloud-alibaba/pull/2694) is a callback function based on org.junit.jupiter.api.Get information from the docker-compose file from the context before the user runs the tests. Then running the container inside the callback function. At last executing the test.
Solution Two: Through maven Profile , using dockerfile-maven-plugin to build a Docker image and complete the push and using docker Load to load and package spring-cloud-alibaba-tests ‘s build product spring-cloud-alibaba-tests.tar :

By referring to the CI method of shardingsphere, build the CI environment of SCA.

Shardingsphere build steps:
- [Deploy mages](https://github.com/apache/shardingsphere/blob/master/.github/workflows/it.yml)
- [Docker Continuous Integration](https://github.com/apache/shardingsphere/blob/master/.github/workflows/docker-per-commit.yaml)
[Current implementation](https://github.com/alibaba/spring-cloud-alibaba/pull/2524) is divided into two steps:
- Log in to dockerhub, put nacos, rocketmq, sentinel, seata and other images under the same dockerhub account, when performing integration tests, pull the relevant images. (Default is in alibaba container image service, so it will be searched in the alibaba repository in docker.yml)
- Package the class to be tested into a jar and execute the archive with docker load.
At present, the first solution is in the POC stage, and the second solution is in the testing stage.

Design Ideas and Compromises

Github Action
- Solution One:
  - Deploy the image and run docker integration tests

name: Integration Testing
on:
  push:
    branches:
      - 2021.x
  pull_request:
    branches:
      - 2021.x
jobs:
  deploy-docker-image:
    runs-on: ubuntu-latest
    timeout-minutes: 60
    steps:
      - name: Check out the repo
        uses: actions/checkout@v2

      - name: Set up QEMU
        uses: docker/setup-qemu-action@v1

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v1

  integration-testing:
    name: Integration Testing
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: Set up JDK 8
        uses: actions/setup-java@v2
        with:
          java-version: '8'
          distribution: 'adopt'
      - name: Dependies Cache
        uses: actions/cache@v2
        with:
          path: ~/.m2/repository
          key: ${{ runner.os }}-maven-${{ hashFiles('**/pom.xml') }}
          restore-keys: |
            ${{ runner.os }}-maven-
      - name: Compile & Checkstyle
        run: mvn clean compile
      - name: Testing
        run: mvn clean test
#        run: mvn clean -Dit.enabled=true test

Solution Two:
- Deploy images

name: Deploy Docker Image

on:
  push:
    branches:
      - 2021.x
      paths:
        - '**/pom.xml'
        - '**/src/main/**'
      pull_request:
        branches:
          - 2021.x
        paths:
          - '**/pom.xml'
          - '**/src/main/**'
  release:
    types:
      - published
  workflow_dispatch:

env:
  MAVEN_OPTS: -Dhttp.keepAlive=false -Dmaven.wagon.http.pool=false -Dmaven.wagon.http.retryHandler.count=3 -Dmaven.wagon.httpconnectionManager.ttlSeconds=30 -DskipTests

jobs:
  deploy-docker-iamge:
    runs-on: ubuntu-latest
    timeout-minutes: 60
    steps:
      - uses: actions/checkout@v2

      # setup docker
      - name: Set up QEMU
        uses: docker/setup-qemu-action@v1

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v1

      - name: Login to DockerHub
        uses: docker/login-action@v1
        with:
          username: ${{ secrets.DOCKERHUB_USERNAME }}
          password: ${{ secrets.DOCKERHUB_TOKEN }}

      # build target with maven
      - name: Cache Maven Repos
        uses: actions/cache@v2
        with:
          path: ~/.m2/repository
          key: ${{ runner.os }}-maven-${{ hashFiles('**/pom.xml') }}
          restore-keys: |
            ${{ runner.os }}-maven-
      - name: Set up JDK 8
        uses: actions/setup-java@v1
        with:
          java-version: 8

      - name: set environment
        run: export MAVEN_OPTS=' -Dmaven.javadoc.skip=true -Drat.skip=true -Djacoco.skip=true $MAVEN_OPTS'
      - name: Build Project
        run: |
          ./mvnw -B -Prelease,docker -DskipTests clean install
          docker image ls --format "{{.ID}} {{.Repository}} {{.Tag}}" | grep alibaba| sed  's/apache\//${{ secrets.DOCKERHUB_USERNAME }}\//' |tr A-Z a-z |awk '{system("docker tag "$1" "$2":latest;docker tag "$1" "$2":"$3";")}'
      - name: Push Docker Image
        run: |
          echo Docker Images:
          echo `docker image ls|grep -i ${{ secrets.DOCKERHUB_USERNAME }}|awk '{print $1":"$2}'`
          docker image ls|grep -i ${{ secrets.DOCKERHUB_USERNAME }}|awk '{print $1":"$2}'|xargs -i docker push {}

Docker integration test

name: Integration Testing
on:
  push:
    branches:
      - 2021.x
    paths:
      - '.github/workflows/integration-test.yml'
      - '**/pom.xml'
      - '**/src/main/**'
  pull_request:
    branches:
      - 2021.x
    paths:
      - '.github/workflows/integration-test.yml'
      - '**/pom.xml'
      - '**/src/main/**'
jobs:
  build-it-image:
    name: build-it-image
    runs-on: ubuntu-latest
    timeout-minutes: 20
    steps:
      - uses: actions/checkout@v2
      - name: Maven resolve ranges
        run: ./mvnw versions:resolve-ranges -ntp -Dincludes='org.springframework:*,org.springframework.boot:*'
      - name: Cache Maven Repos
        uses: actions/cache@v2
        with:
          path: ~/.m2/repository
          key: spring-cloud-alibaba-it-cache-${{ github.sha }}
          restore-keys: |
            ${{ runner.os }}-maven-
      - name: Set up JDK 8
        uses: actions/setup-java@v2
        with:
          distribution: 'temurin'
          java-version: 8
      - name: Build IT image
        run: ./mvnw -B clean install -am -pl spring-cloud-alibaba-tests/spring-cloud-alibaba-testcontainers -Pit.env.docker -DskipTests -Dspotless.apply.skip=true
      - name: Save IT image
        run: docker save -o /tmp/spring-cloud-alibaba-testcontainers.tar apache/spring-cloud-alibaba-testcontainers:latest
      - name: Upload IT image
        uses: actions/upload-artifact@v3
        with:
          name: it-image
          path: /tmp/spring-cloud-alibaba-testcontainers.tar
          retention-days: 1

  integration-testing:
    name: Integration Testing
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: Set up JDK 8
        uses: actions/setup-java@v2
        with:
          java-version: '8'
          distribution: 'adopt'
      - name: Dependies Cache
        uses: actions/cache@v2
        with:
          path: ~/.m2/repository
          key: ${{ runner.os }}-maven-${{ hashFiles('**/pom.xml') }}
          restore-keys: |
            ${{ runner.os }}-maven-
      - name: Compile & Checkstyle
        run: mvn clean compile
      - name: Testing
        run: mvn clean test
#        run: mvn clean -Dit.enabled=true test
      - name: Download IT image
        uses: actions/download-artifact@v3
        with:
          name: it-image
          path: /tmp/
      - name: Load IT image
        run: docker load -i /tmp/spring-cloud-alibaba-testcontainers.tar
      - name: "run install by skip tests"
        if: ${{ steps.check_changes.outputs.docs_only != 'true' }}
        run: mvn -q -B -ntp clean install -DskipTests
      - name: run integration tests
        if: ${{ steps.check_changes.outputs.docs_only != 'true' }}
        run: ./build/run_integration_group.sh CLI

How to add test cases ?
- Solution One: Add two annotations @SpringCloudAlibaba and @TestExtend. In the test case, it is divided into a static method marked with @BeforeEach and a non-static method marked with @BeforeAll. If the execution time of the test thread is long, you can modify the timeout property of @TestExtend .
- Solution Two: When Github Action completes the spring-cloud-alibaba-tests.tar jar-packaging test , image building and pushing, then test the -Denv=docker environment based on the script (It can be located in this line of PR run: ./build/run_integration_group.sh CLI ) , at last, run the integration test method of packaging.

Scheme comparison

Extensibility

For solution one: Because the tests are not packaged and executed uniformly, it must pull an image from github every time when you run a test.

For solution two: The image construction and push are managed uniformly by Github Action. The test is executed after unified packaging, and there is no need to build and push the image every time.
Maintainability

For solution one: as the number of tests increasing, extending the time for triggering ci each time when user submit a PR.It is a short-term plan.

For solution two:Managing image construction and push uniformly and weaken the test scope.You can control the ci time triggered every time when user submit a PR.Group testing can be extended through scripts.

Aug 10 '22 08:08 BillLthree

spring-cloud-alibaba spring-cloud-alibaba copied to clipboard

Spring Cloud Alibaba Auto-Test Design | Spring Cloud Alibaba自动化测试方案设计

背景

设计方案

设计思路以及折中

方案对比

Background

Program Design

Design Ideas and Compromises

Scheme comparison

spring-cloud-alibaba
spring-cloud-alibaba copied to clipboard