Create pool of test namespaces to enable cross-namespace Scenarios

[ti-mo]

Currently, we're creating a fixed cilium-test Namespace that contains a single stack of client/echo Pods. In order to support cross-namespace tests, we should move away from this model in order to make implementing new cross-namespace Scenarios less painful.

I propose to:

• Make testing Namespaces randomly-generated
• Create a sync.Pool (or similar) construct that allocates Namespaces on-demand (cattle, not pets)
• Scenarios request one or more Namespaces at the start. If there are Namespaces in the pool that were Put() by finished Scenarios, they are re-used.
• Make the harness take care of applying CNPs/KNPs to all Namespaces automatically, and to mutate the policies in a predictable way, like overwriting the Namespace field or overwriting certain annotations/labels. We could opt for placeholder values or (my preference) empty values (e.g. requiring metadata.namespace or certain label values or matchers to be empty).

This would also imply test Namespaces get removed when the connectivity test suite finishes. Remove cilium-test deletion behaviour from cilium uninstall as well.

Also, this brings us a step closer to being able to schedule Tests across Namespaces and being able to run them concurrently.

[LarsBingBong]

Any news on this? We have to delete the cilium-test namespace after the test ... a little sad when the test is part of some automation actually verifying the key core services of X Kubernetes cluster are running.

Thank you for the great product all in all - Cilium really kicks some tires.