[Question] Unable to find out how to test reconiler actually reconciled

[mwoudstra]

Hello,

I'm a beginner in the whole operator framework, and have some problems settings up certain testcases and/or cannot find the relevant documentation for it.

In essence, what I want to test, is that my reconciler is actually reconciling. Therefore, in my test I apply a CR that the reconciler listens to, and expect the reconciler to act upon it.

However, I am struggling how to setup this, due to several options for mocking which seem to not tick all my boxes:

• When using the @EnableKubernetesMockClient, my reconciler seems not to be able to watch (atleast it does not get any events, while the masterUrl is the same as the test cases)
• When using @EnableKubernetesMockClient does not allow for SSA, which is troublesome with the recent versions of JOSDK.

Another struggle for me, is to inject a KubernetesClient in my testcases, that is also shared with my real application. Since Spring does not scan @TestConfigurations, I need to mark my KubernetesClient as a @Configuration, which I doubt is what i really should do.

How would one approach such a test, where the requirements are it must be fully mocked (thus not be using my .kubeconfig).

Spring Boot:3.5.3 io.javaoperatorsdk:operator-framework:5.1.1 io.javaoperatorsdk:operator-framework-junit-5:5.1.2 io.fabric8:kube-api-test:7.3.1 io.fabric8:kubernetes-server-mock:7.3.1

[csviri]

Hi @mwoudstra , are you using JOSDK Spring Boot starter or just plain spring boot with the framework?

[mwoudstra]

Ive tried using the spring boot starter, but it worked against me instead of in favor to be honest. So i started with it, but dont have it anymore.

[csviri]

Ok, so, what we could do is support this explicitly in the Spring boot strater. Will take a look on that in coming weeks.

Meanwhile if you have a hard time setting it up consider using alternatives, either with core JOSDK without spring boot - where there are examples of usage, or just Spring boot with kind/minikube.

[michaelkoepf]

Hi @mwoudstra,

I am not sure if this fully addresses your problem, but is using @EnableKubeAPIServer from io.fabric8:kube-api-test a possibility? This should not use .kubeconfig.

I am not an expert in this, but I think I made it work with io.javaoperatorsdk:operator-framework-junit-5 using a custom TestInstancePostProcessor. Maybe @csviri can confirm that this works correctly -- or, at the very least, that the approach is not severely flawed. And not sure if this maybe can achieved in an easier way. I have implemented this a while ago and could not find any other way to make this combination work together back then.

KubeAPITestExtension.java

public class KubeAPITestExtension implements TestInstancePostProcessor {

  private static final String TEST_NAMESPACE_PREFIX = "kubeapitest-";

  @Override
  public void postProcessTestInstance(Object testInstance, ExtensionContext context) {
    if (!(testInstance instanceof KubeAPIOperatorTestBase)) {
      throw new RuntimeException(
          "Test class must extend " + KubeAPIOperatorTestBase.class.getName());
    }

    KubeAPITestBase kubeAPITest = (KubeAPITestBase) testInstance;

    if (kubeAPITest.getKubeConfig() == null) {
      throw new RuntimeException(
          "Configuration was not injected.");
    }

    String namespace = TEST_NAMESPACE_PREFIX + UUID.randomUUID();
    kubeAPITest.setNamespace(namespace.substring(0, Math.min(namespace.length(), 63)));

    // create a new client, since the client is automatically closed by the extension
    KubernetesClient kubernetesClient = new KubernetesClientBuilder()
        .withConfig(Config.fromKubeconfig(kubeAPITest.getKubeConfig()))
        .editOrNewConfig()
        .withNamespace(namespace)
        .endConfig()
        .build();

    if (testInstance instanceof KubeAPIOperatorTestBase) {
      KubeAPIOperatorTestBase kubeAPIOperatorTest = (KubeAPIOperatorTestBase) testInstance;

      Reconciler<?> reconciler = kubeAPIOperatorTest.getReconciler();

      AbstractOperatorExtension operator = LocallyRunOperatorExtension.builder()
          .withNamespaceNameSupplier(__ -> kubernetesClient.getNamespace())
          .waitForNamespaceDeletion(false)
          .withKubernetesClient(kubernetesClient)
          .withReconciler(reconciler)
          .build();

      kubeAPIOperatorTest.setOperator(operator);
    }
    // if it is not an instance of KubeAPIOperatorTestBase, we just inject the client
    kubeAPITest.setKubernetesClient(kubernetesClient);
  }
}

KubeAPITestBase.java

public abstract class KubeAPITestBase {

  public abstract String getKubeConfig();

  private String namespace;

  private KubernetesClient kubernetesClient;

  public String getNamespace() {
    return namespace;
  }

  public void setNamespace(final String namespace) {
    this.namespace = namespace;
  }

  public KubernetesClient getKubernetesClient() {
    return kubernetesClient;
  }

  public void setKubernetesClient(final KubernetesClient kubernetesClient) {
    this.kubernetesClient = kubernetesClient;
  }
}

KubeAPIOperatorTestBase.java

public abstract class KubeAPIOperatorTestBase extends KubeAPITestBase {

  public abstract Reconciler<?> getReconciler();

  @RegisterExtension
  private AbstractOperatorExtension operator;

  public AbstractOperatorExtension getOperator() {
    return operator;
  }

  public void setOperator(final AbstractOperatorExtension operator) {
    this.operator = operator;
  }
}

YourTestClass.java

Note: @EnableKubeAPIServer and @KubeConfig can be put on parent class as soon as https://github.com/fabric8io/kubernetes-client/issues/7223 is released.

@EnableKubeAPIServer
@ExtendWith(KubeAPITestExtension.class)
public class FlinkScaleDriverIT extends KubeAPIOperatorTestBase {
  @KubeConfig
  static String kubeConfigYaml;

  @Override
  public String getKubeConfig() {
    return kubeConfigYaml;
  }

  @Override
  public Reconciler<?> getReconciler() {
    return new YourReconciler();
  }

  @Test
  void test() {
    // here you can access the Kubernetes client via getKubernetesClient(), apply your CRD and assert that everything works as expected
  }
}

[csviri]

Hi @michaelkoepf , I don't think this would address the issue in Spring Boot starter, but I plan to do an explicit support there. But in case pls create a PR there, and will take a look

[michaelkoepf]

I don't think this would address the issue in Spring Boot starter

@csviri you are totally right. sorry if i was not clear on that.

@mwoudstra said that they dropped the spring boot starter, so i thought my approach could work for them as well. what i didn't realize is that they still use spring (and only dropped the starter), so i am not sure if my approach is even useful at all. but maybe it can be modified for their scenario.