Container based executors

[exu]

linked: https://github.com/kubeshop/testkube/issues/2061

• Allow users to define an executor by specifying - docker image name - credentials (in case of private registries) - name of the executor (and other misc data t.b.d)

• Test can be defined to use this executor. The same behaviour is expected from a users's standpoint

• There should be a clear folder structure documented that specifies - Which folder should the artifacts be saved too ? - Q: should the logs be saved in artifacts ?

• Pass & fail is solely determined by the exit status of the pods execution. but allow a Junit.xml report to be retrieved for extra info.

• Variables of type basic and secret should be mounted as environment variables on the container.

• Arguments defined by the user should be sent in a similar faction as this example:

  • if the user were to run a nginx image in k8s and send the command ls –la / in k8s, he would do it like this -> kubectl exec nginx -- ls –la /. A similar behaviour is expected for the this container based executor and its arguments.

• Are resource limit important to set ? resources: requests: memory: "64Mi" cpu: "250m" limits: memory: "128Mi" cpu: "500m"

[TheBrunoLopes]

@ili-shmidt-wib here's the description of the feature we were discussing last week. Feel free to add your input 👌

[TheLeanMachine]

Hey @TheBrunoLopes ,

hate to ask, but: Is there a rough(!) ETA on this feature?

(I seriously don't want anyone to rush, but I will have to make a decision: Should I wait for this feature ...OR... seek out and try to find an alternative for running my tests.)

[TheLeanMachine]

btw: @exu , @povilasv

...as soon as this is released - or some kind of beta-version is publicly available -, I would try out this "generic container executor".

[TheLeanMachine]

one question @exu: Does the image used in this executor have to be a public one (e.g. available in Docker hub)?

...because oftentimes, corporations prepare their own, private images (specific JDKs, certificates, etc) that reside in a private Docker Registry.

[TheBrunoLopes]

Hi @TheLeanMachine it's awesome that you are also wanting to test this feature out!

• The images don't need to be public, you will be able to define the pull secret of this image in case it's not already configured in your k8s cluster.

• We started implementing it yesterday, I can't promise you an exact date yet, but I would venture saying that it will probably take ~2 weeks to make a release out of it.

• Is there a specific type of test that you want to run ? maybe we can help you with that.

[TheLeanMachine]

Hey @TheBrunoLopes ,

(first of all: Thanks 4 the quick and comprehensive reply)

• Cool, will learn about "pull secrets & friends", then
• Awesome! - It's really only about knowing that it shouldn't be 2 months or 2 years ;-)
• Yes, indeed. I still have the problem with running Cypress tests that are aggregated using NX and "good old" Java-API-tests implemented with Maven (that fails using testkube's Maven runner - due to some Java version problems)...BUT:
  • I thought, I would wait and try out both using the "Generic Image Runner" (or how do you wanna call it)
  • From my "naive end-user viewpoint", I imagine the existing custom-runners (Maven, Cypress, ...) to delegate to the "Generic Image Runner", at some point, anyway. But - of course - I lack implementation details, so maybe that's not feasible...

[exu]

@TheLeanMachine please pass those Java problems in separate issue we can try to help with that.

[TheLeanMachine]

Thanks @exu for reaching out,

in fact, when I was adding the JDK-18-executor following the corresponding documentation (section "Using different JDKs"), it worked just fine.

[povilasv]

Hey @TheLeanMachine we merged initial support for Container executors, there are docs available in documentation page

The latest release already has this functionality, so if you update TestKube you can try it out :)

Would be awesome if you can try it / give us feedback :balloon:

One thing - we currently don't have a way to support artifacts yet, but we are figuring it out :package:

[TheLeanMachine]

Hey @povilasv ,

awesome! Will definitely have a look at the docs and checkout ASAP, but...

...ATM, I am sadly blocked by this bug: https://github.com/kubeshop/testkube/issues/2361 ;-)

After this is resolved - and the latest testkube release is deployed by our OPS-guys - , I will jump on this and will feed back the results.

[TheLeanMachine]

Hey @povilasv ,

I know that it's not the right thread here, but: Please don't make the mistake of CI/CD-providers - like bitbucket - to only obtain the artefacts in case of build success... ;-)

[TheBrunoLopes]

Thanks for the feedback @TheLeanMachine. We will be aware of that. @povilasv noticed that #2361 is closed. Is it fixed ?

[TheLeanMachine]

to @TheBrunoLopes and @povilasv

I am not 100% sure, if this is still a problem. Maybe, I simply have to install a more recent testkube client. Will do that tomorrow and update you, here.

(my colleague that reported that bug did not have any problems any more, but i did ... again, the testkube client version is probably the problem)

[TheLeanMachine]

Hi @povilasv , @TheBrunoLopes ,

I dunno, if https://github.com/kubeshop/testkube/issues/2361 is still a problem...

At least I, personally, was able to open the testkube dashboard, again, on my MacBook again after using testkube client from kubeshop's tap...

(brew uninstall testkube) // removing old client
brew tap kubeshop/homebrew-testkube
brew install kubeshop/testkube/testkube

[povilasv]

Thanks for trying again :heart: I think the issue with tk dashboard command is that once we updated the API container port from 80 to 8080 (in order to run as rootless) we broke old clients :cry:

[TheLeanMachine]

Hey @povilasv,

(using these versions, now)

Client Version 1.5.41
Server Version v1.5.41

here some initial feedback: I created my own custom test - still want to run NX-agregated tests - using an official NX-image, now. This is the executor CRD:

apiVersion: executor.testkube.io/v1
kind: Executor
metadata:
  name: nx-container-executor
  namespace: testkube
spec:
  image: soleware/nx-cli:8.5.2
  command: ["nx"]
  executor_type: container
  types:
  - nx-container/test

...with which I continue to create this test...

kubectl testkube create test --name nx-containerexecutor-test-v1 --type nx-container/test --executor-args '--version' --git-branch master --git-uri https://bitbucket.org/<MY_ORG>/<NX_REPO>.git --git-username kai_hoelscher --git-token <SECRET_TOKEN>

Running the test yields this output (I expected it to print the NX-version)...

...which made me realise two things:

1. Apparently no git-checkout is performed
2. I forgot to ask for sth like a pre-command-build-hook (a command used to build/bundle the tests, before execution: here: npm install)

[povilasv]

Apparently no git-checkout is performed

@TheLeanMachine I've tested both git and git-dir options and they seem to work:

  content:
    repository:
      branch: main
      type: git-dir
      uri: https://github.com/kubeshop/testkube-executor-init
    type: git-dir

Downloads into /data/repo directory

ls /data/repo
CODE_OF_CONDUCT.md  CONTRIBUTING.md  LICENSE  Makefile  README.md  build  cmd  go.mod  go.sum  pkg

and:

  content:
    repository:
      branch: main
      type: git-file
      uri: https://github.com/kubeshop/testkube-executor-init
    type: git-file

ls /data/repo
CODE_OF_CONDUCT.md  CONTRIBUTING.md  LICENSE  Makefile  README.md  build  cmd  go.mod  go.sum  pkg

[povilasv]

I forgot to ask for sth like a pre-command-build-hook (a command used to build/bundle the tests, before execution: here: npm install)

Regarding this, we will discuss this internally, at the moment you can make a custom image based on that one and do some bash scripting to do what you need , no?

[TheLeanMachine]

Hey @povilasv,

sounds reasonable. I will check that out, soon...

[TheLeanMachine]

Hey @povilasv,

it seems, that - in fact - I will be able to "hack multiple commands together'. One way to implement this, is using a startup-command in the executor-definition (see command below) like this...

apiVersion: executor.testkube.io/v1
kind: Executor
metadata:
  name: nx-custom-executor
  namespace: testkube
spec:
  image: soleware/nx-cli:8.5.2
  command: ["/bin/bash", "-c", "pwd;echo 'Navigate to checked-out repository located in /data/repo';cd /data/repo; npm install"]
  executor_type: container
  types:
  - nx-custom/test

...but sadly that leads me to the next problem: Non writable file-system!

...so, when I create a testkube test using the above executor, I get this output (as a result of the npm install, I guess):

Any idea, how I could address this? (spoiler: chmod -R 777 ... did not help...)

[povilasv]

...but sadly that leads me to the next problem: Non writable file-system!

Any idea, how I could address this? (spoiler: chmod -R 777 ... did not help...)

@TheLeanMachine I think it's best that you make your custom docker image:

FROM: soleware/nx-cli:8.5.2
RUN: npm install ...

Because, do you really want to install the dependencies every time you run a test?

IMO this is what containerization solves, the /data directory is really more for testdata data not for programs.

@vsukhin wdyt?

[TheLeanMachine]

Hey @povilasv ,

thanks for your feedback, so far. Sounds reasonable. Will think about this again...

(but basically, you're right)

[vsukhin]

So,remained part for this ticket is a storing artifact to external folder

[TheBrunoLopes]

AC:

• There should be an environment variable that is possible to change/configure that specifies the dir path to where to retrieve the artifacts.
• User should not need to do any DevOps work to use this feature.