Start multi clusters during the tests

[helayoty]

Are we able to use the e2e-framework to start more than one cluster in the setup?

[helayoty]

I was able to create 2 kind clusters in the Setup main_test. How I can switch context between them? @vladimirvivien

[vladimirvivien]

Not sure you can switch context now. But that sounds like it's the ask: ability to switch kind context.

[helayoty]

I'd love if we can have this feature as soon as we can.

[vladimirvivien]

@helayoty can you drop in a code snippet that shows exactly where/how this would be useful. Or, at least elaborate on the user story a bit? That way anyone of us can pick it up and run with an implementation (maybe if I have cycle I can do something with it). Thanks again for bringing this upl.

[helayoty]

@vladimirvivien Something like

func TestMain(m *testing.M) {
	testEnv = env.NewWithConfig(envconf.New())
	// Create kind clusters
	testEnv.Setup(
		envfuncs.CreateKindClusterWithConfig("cluster1", "kindest/node:v1.23.5", "kind-config.yaml"),
		envfuncs.LoadDockerImageToCluster("cluster1", "image"),
		deployChart("cluster1","helm_chart_path"),
		envfuncs.CreateKindClusterWithConfig("cluster2", "kindest/node:v1.23.5", "kind-config.yaml"),
		envfuncs.LoadDockerImageToCluster("cluster2", "image"),
	).Finish( // Cleanup KinD Cluster
		envfuncs.DestroyKindCluster("cluster1"),
		envfuncs.DestroyKindCluster("cluster2"),
	)

	os.Exit(testEnv.Run(m))
}

func TestSucessScenairo(t *testing.T) {
	successFlow := features.New("Test happy flow").
		Setup(func(ctx context.Context, t *testing.T, cfg *envconf.Config) context.Context {
			kubeConfig1 := cfg.GetContext("cluster1")
			//deploy something in cluster1
                        kubeConfig2 := cfg.GetContext("cluster2")
                        // deploy something else in cluster2
			return ctx
		}).Assess(" something is happening", func(ctx context.Context, t *testing.T, cfg *envconf.Config) context.Context {
		//TODO
	}).Feature()

	testEnv.Test(t, successFlow)
}

[harshanarayana]

This seems doable. But Might need add a few new API to support this with backward compatibility. Especially for things such as cfg.NewClient() and cfg.Client which can take a cluster name context and then use the right kube-config from there. Also some of the current behavior of how the cfg.NewClient works needs a bit of tuning. It does a whole lot to optimize creation of new client.

I for sure like the idea of being able to support multiple clusters at the same time and switch between them using a context during the test execution. However, can the same not be achieved by running the same test binary twice by setting different env variables that you can use to define your Setup and then under Assess accordingly ?

[helayoty]

However, can the same not be achieved by running the same test binary twice by setting different env variables that you can use to define your Setup and then under Assess accordingly ?

@harshanarayana I tried to do so but it seems if something went wrong the shutdown/destroy test env is not working correctly.

[harshanarayana]

@helayoty interesting. Does that leave traces of test cluster behind? What kind of issue did you run into?

[harshanarayana]

I think we should start a bit more details Design doc if we really want to tackle this. I tested a few things for quick check and there is a considerable amount of places where this will spread into. Retaining backward compatibility should also be accounted for.

[vladimirvivien]

Maybe solution/design is the following:

• Start 1 or more cluster in env.Setup, each having its respective context
• In assessment, have the ability to set the cluster context like cfg.WithClusterContext(...).Resources().ClusterOperation(...)
• Then delete clusters in env.Finish

Essentially, have the ability to set cluster context name and the ability to specify the cluster context during cluster operation.

[harshanarayana]

This was the similar thing i had in mind.. I was thinking of extending the context to the NewClient calls as well though.. Let me draft a design and open a PR to discuss further

/assign

[harshanarayana]

/assign

[harshanarayana]

@helayoty @vladimirvivien This was always possible without actually adding a new API. Turns out there was a small change required to get it working. I am opening a draft PR with the example of how to run multi kind cluster test with the small change required to get it working. Let me know what you think about it. Do we still need to add a new API or document the current one ? I am kind of inclined towards document with example.