kaskada icon indicating copy to clipboard operation
kaskada copied to clipboard

feat: Label integration tests; update or eliminate makefile

Open bjchambers opened this issue 1 year ago • 3 comments

Summary

When running locally, it would be useful if we could rely on the labels to filter integration tests. For instance --query-filter='!pulsar && !redis' to run all of the tests other than Pulsar and Redis tests.

We already have many of the tests properly labeled. For example: https://github.com/kaskada-ai/kaskada/blob/main/tests/integration/api/mat_pulsar_to_obj_store_test.go#L28

However, some of the tests rely on Pulsar without being labeled. For example: https://github.com/kaskada-ai/kaskada/blob/main/tests/integration/api/queries_v1_test.go#L36

In general, we should prefer to label entire functions as we currently do. But based on the Gingkgo docs, there are other options if we really need these tests that use Pulsar internally:

It("can save books remotely", Label("network", "slow", "library query") {
  if Label("performance").MatchesLabelFilter(GinkgoLabelFilter()) {
    exp := gmeasure.NewExperiment()
    // perform some benchmarking with exp...
  }
  // rest of the saving books test
})

(from https://onsi.github.io/ginkgo/#spec-labels)

The other part of this is that the current structure of the makefile makes it difficult to pass arguments -- we need to create additional targets for every change. We should look at passing other arguments through the makefile, or using a shell script or something. Then we could more easily run something like ENV=local-local go run github.com/onsi/ginkgo/v2/ginkgo ./... --label-filter='!pulsar && !redis' (note that this also omits the -vv since the extra verbosity is just spammy in many cases).

bjchambers avatar Jul 05 '23 21:07 bjchambers

#490 adds the remaining pulsar test labels.

epinzur avatar Jul 10 '23 20:07 epinzur

I think we should make a small CLI tool for running the most of the commands currently in the Makefile.

Options on commands would be much more straightforward than enumerating every potential command in the Makefile.

epinzur avatar Jul 10 '23 20:07 epinzur

I'm 50/50 -- I think it may be nice to just document the commands and people can create aliases if they want? Specifically, it seems easier to "know" how things are being run if we just document "use the ginkgo runner". But :shrug:

bjchambers avatar Jul 10 '23 21:07 bjchambers