TEST="xxxx" make test-e2e doesn't filter tests

[akihikokuroda]

TEST="kustomize" make test-e2e

It should run just 6 specs but it ran 77.

root@akuroda-0310:~/go/src/operator-framework/rukpak# TEST="kustomize" make test-e2e
/root/go/src/operator-framework/rukpak/hack/tools/bin/ginkgo --tags upstream "" -trace -progress -v -focus "kustomize" test/e2e
Running Suite: E2E Suite - /root/go/src/operator-framework/rukpak/test/e2e
==========================================================================
Random Seed: 1665091731

Will run 77 of 80 specs

[tylerslaton]

Could you try doing this with FOCUS= instead? Looking at the Makefile I believe that is the intended way of accomplishing this.

https://github.com/operator-framework/rukpak/blob/e4303270f33b3b88f1c97b04f49a6cf3042d158d/Makefile#L103-L106

[timflannagan]

Looking at Aki's output above, it does look like the $TEST variable was properly substituted into the -focus flag when running the test-e2e target. Maybe an issue with our ginkgo version?

[akihikokuroda]

If I do like TEST="kustomize" E2E_FLAGS= make test-e2e, it works. It seems like "" is taken as a tag.

[timflannagan]

Interesting - probably an oversight after merging https://github.com/operator-framework/rukpak/commit/1bea93e12126887b393e56e40a5dc0ed75dcc7df. I wonder whether an integration tests would've caught this.

[tylerslaton]

We went over this issue during issue triage and it looks like something we'll want to fix. The context here should be sufficient to get started but to whoever is picking this up - feel free to ask any follow-up questions!

[github-actions[bot]]

This issue has become stale because it has been open 60 days with no activity. The maintainers of this repo will remove this label during issue triage or it will be removed automatically after an update. Adding the lifecycle/frozen label will cause this issue to ignore lifecycle events.