Setup & Teardown actions for Pipeline stages

[jithine]

What happened:

Pipeline stages introduces logical grouping of various jobs, for example group all your production jobs as one stage of your build pipeline. It would be beneficial to provide setup and teardown actions for a Pipeline stage as a whole so that any initialization or cleanups needed before jobs in a stage runs can be executed. Idea is similar to the setup & teardown steps for an individual job

What you expected to happen:

Users should be able to define pre & post actions for a stage. UI should provide interface where users can see the logs for pre & post actions. Pre actions should run exactly once before any job in the stage runs. Post actions should run exactly once when a stage ends due to either success or failure of individual jobs. Pre & Post actions could be fashioned as jobs

Sub-Tasks:

• [ ] https://github.com/screwdriver-cd/screwdriver/issues/3022
• [x] https://github.com/screwdriver-cd/screwdriver/issues/3026
• [x] https://github.com/screwdriver-cd/screwdriver/issues/3045

References:

Mocks - TBD

[jithine]

cc @adong @tkyi

Alan pls share mocks with the actions mocked up

[tkyi]

Mocks:

[tkyi]

Some things for discussion:

Pre/postaction in stage

• is this just another SD job, but it will always run (like how a setup/teardown step is just another SD step)?
• can you rerun a pre/postaction job manually?
• can you enable/disable a pre/postaction job?
• job name ideas: pre/post{stage}, pre/postaction, setup/teardown{stage}, sd-setup-{stage}/sd-teardown-{stage}
• what image will the job use? generic image, or will we let the user configure?
• will we skip sd-setup-scm(git clone) step?

SD Config

Current stage config:

shared:
  image: node:14
  steps:
    - init: echo 'init'
jobs:
  job1:
    requires: [~pr, ~commit]
  job2:
    requires: [job1]
  job3:
    requires: [~pr, ~commit]
  job4:
    requires: [job3]

stages:
  seahorse:
    jobs: [job3, job4]
    description: "seahorse starfish deploy jobs"
  dolphin:
    jobs: [job2, job1]
    description: "dolphin deploy jobs"

Potential new:

shared:
  image: node:14
  steps:
    - init: echo 'init'
jobs:
  job1:
    requires: [~pr, ~commit]
  job2:
    requires: [job1]
  job3:
    requires: [~pr, ~commit]
  job4:
    requires: [job3]
  job5:
    image: alpine
    steps:
      - notify: set banner
  job6: 
    image: alpine
    steps:
      - announce: publish blog post

stages:
  seahorse:
    setup: job5
    jobs: [job3, job4]
    teardown: job6
    description: "seahorse starfish starfish deploy jobs"
  dolphin:
    jobs: [job2, job1]
    description: "dolphin deploy jobs"

stages:
  seahorse:
    setup: job5
    jobs: [job3, job4]
    teardown: job6
    description: "seahorse starfish deploy jobs"
  dolphin:
    jobs: [job2, job1]
    description: "dolphin deploy jobs"

Stage status

• should it just be SUCCESS/FAILURE or should there be more granularity? (see https://github.com/screwdriver-cd/data-schema/blob/master/models/build.js#L16-L27)
• if a pre/postaction job fails, will that stop the stage from continuing?

[jithine]

@tkyi

is this just another SD job, but it will always run (like how a setup/teardown step is just another SD step)?

yes

can you rerun a pre/postaction job manually?

yes, no need to restrict a jobs behavior.

can you enable/disable a pre/postaction job?

yes

job name ideas: pre/post{stage}, pre/postaction, setup/teardown{stage}, sd-setup-{stage}/sd-teardown-{stage}

I like the setup & teardown, will be consistent with job's setup & teardown semantics.

what image will the job use? generic image, or will we let the user configure?

based on the proposal in the config, it looks like we can go with a full job config. which means users can configure every aspects of the pre/post stage including an image.

will we skip sd-setup-scm(git clone) step?

I am not sure if we need to skip, but I can see value in making it configurable and not do checkout by default But for a initial pass, we can consider pre/post actions as just a job, which would reduce the needs to have any custom workflow

should it just be SUCCESS/FAILURE or should there be more granularity? (see https://github.com/screwdriver-cd/data-schema/blob/master/models/build.js#L16-L27)

since we are modelling it after a job, any of the job's current status should be valid.

if a pre/postaction job fails, will that stop the stage from continuing?

yes

[tkyi]

After discussing with @jithine @sagar1312 this is the new proposed yaml below where a setup/teardown job will have all the characteristics of a job except it must be defined under stages and it cannot have keyword requires.

shared:
  image: node:14
  steps:
    - init: echo 'init'
jobs:
  job1:
    requires: [~pr, ~commit]
  job2:
    requires: [job1]
  job3:
    requires: [~pr, ~commit]
  job4:
    requires: [job3]

stages:
  seahorse:
    setup:
      image: alpine
      steps:
        - notify: set banner
    jobs: [job3, job4]
    teardown:
      image: alpine
      steps:
        - announce: publish blog post
    description: "seahorse starfish starfish deploy jobs"
  dolphin:
    jobs: [job2, job1]
    description: "dolphin deploy jobs"

[VonnyJap]

@sagar1312 @tkyi - I believe this is implemented, so can we close it?

[sagar1312]

This has been implemented