TEP-0114: Custom Tasks Beta - Testing

[jerop]

In TEP-0114: Custom Tasks Beta, we proposed migrating Custom Tasks and Runs to beta.

Ahead of the promotion, we agreed to improve the test coverage for Custom Tasks and Runs. This issue tracks the testing work.

Related issues:

• https://github.com/tektoncd/pipeline/issues/5120
• https://github.com/tektoncd/pipeline/issues/4686

[XinruZhang]

/assign XinruZhang

[XinruZhang]

Currently, we have two ways to run custom tasks: w/ CRD, w/o CRD

• With CRD: task-loop
• Without CRD: wait-task

The highest priority of this issue is to create an e2e test to ensure the basic functionalities of a custom task work as expected, which is a blocker of V1 Pipeline release.

Then we can write a new CRD and controller with more functionalities implemented – as @lbernick suggested in issue #5120.

[XinruZhang]

• [x] ~move wait-task to test/~
• [x] ~write e2e test (install tekton, custom controller)~
• [x] ~put this e2e test together with other tests? (not sure if this is required)~
  • as part of a release test etc.
• [x] ~documentation and future work (stage 2)~

[XinruZhang]

After talking to @dibyom offline, we reached agreement on implementing a new controller for the testing purpose instead of using the exisitng wait-task because

1. wait-task will be eventually replaced by the new controller, no need to write code and make people review something that will definitely be removed;
2. the effort of implementing a new test controller is at the similar level of implementing a new controller.

[XinruZhang]

• [x] Create a new controller under test/ with functionalities mentioned in #5120
• [x] Write e2e test (install the controller, test cases)

Done in PR #5332

[XinruZhang]

Next steps of improving the test coverage for Custom Task:

• [x] Test Custom Task (without CRD) in PipelineRun #5356
• [x] Extensive Test Cases
  • [x] Timeout https://github.com/tektoncd/pipeline/pull/5389
  • [x] Retries
    • [x] ~Update the run.md to clarify the behavior. https://github.com/tektoncd/pipeline/pull/5393~
    • [x] Implement retries in wait-task controller. https://github.com/tektoncd/pipeline/pull/5435
    • [x] Test the behavior in custom_task_test.go https://github.com/tektoncd/pipeline/pull/5523

[XinruZhang]

@jerop For the retries, we need to be aligned with the behavior of TaskRun, IMHO it would be better to implement it after addressing https://github.com/tektoncd/pipeline/issues/5248. wdyt?

[lbernick]

For the retries, we need to be aligned with the behavior of TaskRun, IMHO it would be better to implement it after addressing #5248. wdyt?

I think since we've decided to add retries to TaskRun rather than changing how Run retries work, we can implement testing for Run retries in parallel. (We might want to wait for the TEP to be accepted, but I don't think we need to wait for the implementation to land.)

[XinruZhang]

@jerop I think we can mark this issue as done, wdyt?

[jerop]

yes, making it as done - thank you so much @XinruZhang 😻

/close

[tekton-robot]

@jerop: Closing this issue.

In response to this:

yes, making it as done - thank you so much @XinruZhang 😻

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.