kubernetes-sigs
[env] Don't run next assess when failNow is called
What type of PR is this?
/kind bug
What this PR does / why we need it:
When writing tests, this allows the writer to control whether the test should continue running the next assess when a failure occurs. Basically, they can chose between using t.FailNow() or t.Fail() depending on if the failure should stop the test or not. It works independently from the -fail-fast flag specific to the framework and allows a more granular control on test failures.
Which issue(s) this PR fixes:
Fixes #386
Special notes for your reviewer:
I tried to add a new test for this case but couldn't find a way to make it work. Since failNow is called, the test fails and I couldn't find a way to expect a test to fail in go :sweat_smile:. Happy to add one if someone has an idea on how to do it though.
Code to reproduce
You can use the following code to illustrate what I'm trying to solve
package envfuncs_test
import (
"context"
"os"
"testing"
"sigs.k8s.io/e2e-framework/pkg/env"
"sigs.k8s.io/e2e-framework/pkg/envconf"
"sigs.k8s.io/e2e-framework/pkg/features"
)
var testenv env.Environment
func TestMain(m *testing.M) {
testenv = env.New()
os.Exit(testenv.Run(m))
}
func TestFailNow(t *testing.T) {
feat1 := features.New("fail now").
Assess("Assess 1", func(ctx context.Context, t *testing.T, cfg *envconf.Config) context.Context {
t.Log("Assess 1 (should be printed)")
t.FailNow()
return ctx
}).
Assess("Assess 2", func(ctx context.Context, t *testing.T, cfg *envconf.Config) context.Context {
t.Log("Assess 2 (should NOT be printed)")
return ctx
}).
Feature()
feat2 := features.New("succeed").
Assess("Assess 1", func(ctx context.Context, t *testing.T, cfg *envconf.Config) context.Context {
t.Log("Assess 1 (should be printed)")
return ctx
}).
Assess("Assess 2", func(ctx context.Context, t *testing.T, cfg *envconf.Config) context.Context {
t.Log("Assess 2 (should be printed)")
return ctx
}).
Feature()
testenv.Test(t, feat1, feat2)
}
func TestFail(t *testing.T) {
feat := features.New("fail (not now)").
Assess("Assess 1", func(ctx context.Context, t *testing.T, cfg *envconf.Config) context.Context {
t.Log("Assess 1 (should be printed)")
t.Fail()
return ctx
}).
Assess("Assess 2", func(ctx context.Context, t *testing.T, cfg *envconf.Config) context.Context {
t.Log("Assess 2 (should be printed)")
return ctx
}).
Feature()
testenv.Test(t, feat)
}
- Before the changes, here is the output:
❯ go test ./pkg/envfuncs -test.v
=== RUN TestFailNow
=== RUN TestFailNow/fail_now
=== RUN TestFailNow/fail_now/Assess_1
ns_funcs_test.go:39: Assess 1 (should be printed)
=== RUN TestFailNow/fail_now/Assess_2
ns_funcs_test.go:44: Assess 2 (should NOT be printed)
=== RUN TestFailNow/succeed
=== RUN TestFailNow/succeed/Assess_1
ns_funcs_test.go:51: Assess 1 (should be printed)
=== RUN TestFailNow/succeed/Assess_2
ns_funcs_test.go:55: Assess 2 (should be printed)
--- FAIL: TestFailNow (0.00s)
--- FAIL: TestFailNow/fail_now (0.00s)
--- FAIL: TestFailNow/fail_now/Assess_1 (0.00s)
--- PASS: TestFailNow/fail_now/Assess_2 (0.00s)
--- PASS: TestFailNow/succeed (0.00s)
--- PASS: TestFailNow/succeed/Assess_1 (0.00s)
--- PASS: TestFailNow/succeed/Assess_2 (0.00s)
=== RUN TestFail
=== RUN TestFail/fail_(not_now)
=== RUN TestFail/fail_(not_now)/Assess_1
ns_funcs_test.go:66: Assess 1 (should be printed)
=== RUN TestFail/fail_(not_now)/Assess_2
ns_funcs_test.go:71: Assess 2 (should be printed)
--- FAIL: TestFail (0.00s)
--- FAIL: TestFail/fail_(not_now) (0.00s)
--- FAIL: TestFail/fail_(not_now)/Assess_1 (0.00s)
--- PASS: TestFail/fail_(not_now)/Assess_2 (0.00s)
FAIL
FAIL sigs.k8s.io/e2e-framework/pkg/envfuncs 0.008s
FAIL
- Now it is:
❯ go test ./pkg/envfuncs -test.v
=== RUN TestFailNow
=== RUN TestFailNow/fail_now
=== RUN TestFailNow/fail_now/Assess_1
ns_funcs_test.go:153: Assess 1 (should be printed)
=== RUN TestFailNow/succeed
=== RUN TestFailNow/succeed/Assess_1
ns_funcs_test.go:165: Assess 1 (should be printed)
=== RUN TestFailNow/succeed/Assess_2
ns_funcs_test.go:169: Assess 2 (should be printed)
--- FAIL: TestFailNow (0.00s)
--- FAIL: TestFailNow/fail_now (0.00s)
--- FAIL: TestFailNow/fail_now/Assess_1 (0.00s)
--- PASS: TestFailNow/succeed (0.00s)
--- PASS: TestFailNow/succeed/Assess_1 (0.00s)
--- PASS: TestFailNow/succeed/Assess_2 (0.00s)
=== RUN TestFail
=== RUN TestFail/fail_(not_now)
=== RUN TestFail/fail_(not_now)/Assess_1
ns_funcs_test.go:180: Assess 1 (should be printed)
=== RUN TestFail/fail_(not_now)/Assess_2
ns_funcs_test.go:185: Assess 2 (should be printed)
--- FAIL: TestFail (0.00s)
--- FAIL: TestFail/fail_(not_now) (0.00s)
--- FAIL: TestFail/fail_(not_now)/Assess_1 (0.00s)
--- PASS: TestFail/fail_(not_now)/Assess_2 (0.00s)
FAIL
FAIL sigs.k8s.io/e2e-framework/pkg/envfuncs 0.013s
FAIL
Does this PR introduce a user-facing change?
Stopped running the following assess if FailNow() was called during a test
Additional documentation e.g., Usage docs, etc.:
Hi @Fricounet. Thanks for your PR.
I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.
Once the patch is verified, the new status will be reflected by the ok-to-test label.
I understand the commands that are listed here.
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: Fricounet, harshanarayana
The full list of commands accepted by this bot can be found here.
The pull request process is described here
- ~~OWNERS~~ [harshanarayana]
Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment
