Are you using Sonobuoy? We want to hear from you!

[johnSchnake]

We would love to know about your use of Sonobuoy. Share anything Sonobuoy related:

• Info about you and your use of Sonobuoy
• Links to plugins/scripts you use that others may find helpful
• blog posts involving Sonobuoy and workflows involving Sonobuoy
• etc

Why?

Tracking usage in OSS is hard, if you leave a comment on this issue we can better understand who our users are and deliver a better experience for YOU!

• We'd like to give your organization the opportunity to collaborate more closely with contributors (if you want to)
• We want to help promote your organization
• We want to show off all the awesome stuff our users are doing

How you can help

Submit a comment in this issue to include the following information:

Organization/Company: 
Website: 
Country: 
Contact: 
Usage scenario: e.g. We use Sonobuoy for XYZ
Status: Development?/Production?/CI?
Link: [If you are willing, share a link to your awesome script/workflow/plugin so others can learn and benefit]

[JTarasovic]

My company has been using it in an ad-hoc manner to test new K8s releases with our bootstrapping to help verify that everything still runs as expected.

We are looking to automate this in the future so that a new release automatically spins up a cluster, runs a series of tests (including conformance) and outputs the results in a consistent and friendly manner.

I hope that helps and is what you are looking for. Thanks for reaching out to the community!

[leodotcloud]

@johnSchnake I am evaluating this repo to see if we can run the kube-bench plugin.

[xdrus]

• we use sonobuoy in our air gaped env to run e2e test on all configuration changes for our bespoke k8s cluster management scripts.
• Also looking into using it as a smoke-testing tool after cluster upgrades to run custom test specific for our environment.
• kube-bench is another use-case we are looking into, but ATM we run it directly without sonobuoy.

[ashutoshishere04]

how to run only a specific test? how to get that test if you have error message from e2e.log

[johnSchnake]

@ashutoshishere04 You use the --e2e-focus flag (which is the same as setting --plugin-env=e2e.E2E_FOCUS=<value>

See https://sonobuoy.io/docs/v0.16.0/results/ for how to see results in various ways.

Just based on the timing of this question, I would also like to point you towards #910 which found a problem with the k/k testing image which caused it to not save results (and report them to Sonobuoy) when a test error was encountered (on v1.16.0 test image).

Our release of Sonobuoy v0.16.1 today should mitigate that problem though.

[ashutoshishere04]

@johnSchnake
oh nice my current kubernetes cluster has version=v1.12.8, I'm having 2 failure cases.
failed tests: 2 [sig-network] DNS should provide DNS for services [Conformance] [sig-network] DNS should provide DNS for the cluster [Conformance] apparently their solution is this commit(correct me if i'm confused) https://github.com/kubernetes/kubernetes/pull/43421/commits/3b98c808917bc4947e328d15f47f9f91521c6cc6 I have to use this version1.12.8 only, I've used aks-engine for cluster bootstrapping, is there a way to use solution provided in my current scenario?

Thanks it saved a lot of my time(not running all cases again for one case's result),

[johnSchnake]

@ashutoshishere04 You'll have to look into the exact logs to determine that; I'm not familiar with all the different error modes for these tests. However, the commit you linked to was committed into Kubernetes v1.7.0 so it is a part of the code paths you're using for v1.12.8.

[usahai30]

Team, I would like to have an understanding on below - As part of regular image(kubernetes) building process, I want to run conformance tests in a pipeline which creates clusters using the image we create everytime. Right now I am using the sonobuoy cmd tool for this. My problem is every-time I have to download this binary in my test pipeline and issue commands through it. Often due to some interrupt my tests breakdown. I am interested to know if somehow I can directly integrate sonobuoy in my test framework in pipeline. By the way - Sonobuoy is an amazing product. I just loved it overall. Thanks

[erikwilson]

k3s is using Sonobuoy for running the conformance tests on every PR, and we try to minimize the runtime as much as possible. To achieve this we create two virtual clusters, one to run the serial portion of the conformance tests, and the other to run the parallel portion of the conformance tests. This allows us to test the full conformance suite in about 20 minutes on multiple architectures (currently amd64 & arm64). We would also like to enable testing for arm but probably need lots of supporting images to be created. We have also disabled syslog support and enabled the e2e go runner.

For reference how we are separating the serial and parallel portion: https://github.com/rancher/k3s/blob/d2756a76c1/scripts/sonobuoy-e2e-tests#L66-L72

And how we are launching sonobuoy itself: https://github.com/rancher/k3s/blob/d2756a76c1/scripts/sonobuoy#L127-L131

Thanks for an awesome utility!

[stale[bot]]

There has not been much activity here. We'll be closing this issue if there are no follow-ups within 15 days.

[johnSchnake]

Stalebot! Just talking about how I thought this issue existed on https://github.com/vmware-tanzu/sonobuoy/issues/1371

We wanted to reopen this so people can record where/how they use Sonobuoy to help share workflows/plugins etc in a very informal way.

[negz]

We use Sonobuoy to implement the Crossplane CNCF conformance suite per https://github.com/crossplane/conformance and https://github.com/cncf/crossplane-conformance.

[leodotcloud]

In Rancher, we use Sonobuoy to invoke kube-bench on all the nodes of a Kubernetes cluster for running CIS scan.

Reference: https://github.com/rancher/security-scan/blob/master/package/run.sh

[eddytruyen]

Hi John,

The following is long overdue but I have published last year a paper about comparing K8-as-a-Service vendors based on analysis of various information sources, one of which are results of the end2end conformance testing suite of Kubernetes.

I've written some code for automating the comparison based on the e2e results (extracted from [conformance-image:1.13, sonobuoy 0.14.3):

It does the following steps:

ubuntu@kubeadm-ubuntu19:~/githubrepos/MigratingKubernetes/validation$ ./run.sh
1. Extracting and classifying sonobuoy's e2e-tests => junit.xml files and e2e-test logs per K8S vendor aggregated in directory run-results...
2a. Processing all tests: classifying succeeded and failed tests => For each K8s vendor, 2 csv files are generated in directory run-results: one with succeeded tests and one with failed tests. This processing may take several minutes...
2b. Processing all tests: counting number of succceeded and failed tests and generating summary => run-results/Summary.csv....
Run completed successfully!
To delete all results of this run, execute './cleanup.sh'

As you can see, my code does similar things as sonobuoy results. It is possible to aggregate results from multiple tar balls from multiple vendors and the main output is a Summary.csv file. Each entry in the csv corresponds with a documented k8s feature, a specific set of e2e test regular expressions that relate to that k8s feature, and per vendor the number of succeeded and failed tests.

For each vendor there are two other files with the same structure as Summary.csv. "vendor_fail.csv" en "vendor_success.csv". Each entry contains the snippets of the junit file that correspond with the selected tests.

I am now starting a new project based on this work and I did a very small test of my code for output of [Kubernetes-conformance image19.4, sonobuoy 0.52.3]. It still seems to work. However it contains some hacky grep commands some of which maybe replaced to invocations of sonobuoy results | jq

Maybe my code is usable in some sense.

Do you know of related projects?

Below you find the references

• the paper

• the frozen code and documentation for the paper is here.

• New updates will be available here

Don't hesitate to contact me in case you have questions,

Best regards,

Eddy

[johnSchnake]

@eddytruyen I think the paper link should be https://ieeexplore.ieee.org/document/9298825

Thanks so much for your submission, I'm definitely going to take some time to read through. Sounds great though :)

Update: I just wanted to comment a few things for visibility for people looking at the paper/our comments later.

• As you pointed out there have been lots of updates to sonobuoy results so that you can look at results in a handful of different ways instead of relying on bash/grep alone
• Sonobuoy has added a new feature (recently and not heavily promoted yet) to allow plugins to specify configmaps along with their other plugin settings. This means that cloud provider data may be easily provided for a run instead of custom images for each. This would be a big win for this type of use case so I may do a write up/updated blog about how to do that. I'm glad the original one on customizing the e2e tests worked out for you though!
• You mentioned that DevicePlugin tests were giving false positives because of some shortcomings in sonobuoy being able to properly configure the tests. I was looking for more details on that because I'm not sure if that has been resolved or if it is a possible improvement we can make now. (Let me know here, in slack or in a new issue @ eddytruyen)

I'm so thrilled that we were of use in your analysis. Thank you for sharing this with us all!

[matthias50]

I've been looking at sonobuoy to replace our home grown tools for running k8s e2e tests. So far it looks like a huge improvement. My favorite feature so far the ability to easily re-run just the e2e tests which failed previously. One usability nit I can see is that if you are using air gapped clusters (or gatekeeper), you have to specify command line flags to point to various config which will map the location to pull the sonobuoy image as well as the images used by the e2e tests themselves.

While this will obviously work, it would be nice if sonobuoy would also look for config in some default location like say ~/.sonobuoy. That way one could do a one time config setup and then just use the "normal" CLI without having pass the extra options. Thanks!

[johnSchnake]

it would be nice if sonobuoy would also look for config in some default location like say ~/.sonobuoy

Thanks for the feedback! We just introduced that location as the home for plugins so it does open the gates for ideas/use cases like this. 👍 @matthias50

[stale[bot]]

There has not been much activity here. We'll be closing this issue if there are no follow-ups within 15 days.