Passing arguments to `test` command

[KaranAhlawat]

Many testing frameworks support passing in additional CLI arguments to the test runner, often for specifiying only a subset of tests to run. Currently, Bleep does not seem to support providing additional arguments while running the bleep test command, aside from the test project name. This would be good to have as running all tests while working on a particular feature is rarely necessary.

For example, while using Scalatest and sbt, you can type

testOnly com.dummy.app.MyTestClass

in the console to run only the tests contained in MyTestClass.

Finally, I'd like to take a few words to thank you for this amazing project (it's so fast!) and it has truly been a joy to use so far :)

[oyvindberg]

Hey there, thanks for the kind words

I haven't looked into this since I always tests either in CI or in intellij, but according to BSP there are extensions we can use to thread through details https://build-server-protocol.github.io/docs/extensions/scala#scalatestparams

[oyvindberg]

oh and this https://build-server-protocol.github.io/docs/specification#testparams

[KaranAhlawat]

So I guess in this file (Test.scala), we can accept arguments, and then pass those in to the TestParams using setArguments, which accepts a Java List of String?

[oyvindberg]

Yes exactly. Probably the hardest part is to decide on a syntax on the command line. Right now the bleep test command takes one or more project names, and it'll run all tests for all of them

[hamnis]

syntax suggestion: -- args here seems to be a pretty standard way of adding parameters. docker for instance does this.

[KristianAN]

If no-one is working on this I can do it.

[KristianAN]

I have taken a look at bloop-core regarding test arguments (setArguments) and as far as I can tell bloop does not support any arguments for tests through BSP.

I will look into contributing this to bloop-core so that we can just keep using BSP.

[KristianAN]

Regarding syntax I think it would be very beneficial to follow scala-cli where possible? bleep test project is the same as scala-cli. The syntax used there is scala-cli test project --test-only A B C

[oyvindberg]

I have done some preliminary work on this, I can share ab branch tomorrow, I don't have time to make it compile now.

There are some interesting discussions to be had here actually.

Selecting which test suite(s) to run

As far as I understand, the most important reason to pass parameters to tests is to choose which ones to run. To cover that use case I would rather provide a bleep test-only command with explicit surface and good completions.

You can ask BSP for all tests suites in a project

bloop.buildTargetScalaTestClasses(new bsp4j.ScalaTestClassesParams(targets)).get()

And you can specify which test suites to run when running tests. It's done like this:

      val targets = BleepCommandRemote.buildTargets(started.buildPaths, distinctProjects)
      val params = new TestParams(targets)
      params.setDataKind("scala-test")
      params.setData {
        val items = tests.groupBy(_.project).map { case (project, tests) =>
          val testClasses = tests.map(_.cls).toList.asJava
          new bsp4j.ScalaTestClassesItem(BleepCommandRemote.buildTarget(started.buildPaths, project), testClasses)
        }
        val testParams = new ScalaTestParams()
        testParams.setTestClasses(items.toList.asJava)
        testParams
      }
      val result = bloop.buildTargetTest(params).get()

So with those two components we can build working auto-completion from your shell into your scala test files. It'll be slow for the initial compile, but it's a too wonderful opportunity to pass up.

So let's say bleep test-only covers this first case, let's continue to the other case I know

Selecting groups of tests based on tags

I have occasionally seen people use this. I don't necessarily think it's very useful, though we can build functionality if we have to. Since bleep has "free" projects (easy to define, fast to load) I think it's a better approach to define different test-projects for this. That would at least cover the fast/slow distinction I think people are after.

Anything else?

what else do people use this for? do you think what I sketched above is sufficent?

[oyvindberg]

It's documented as a BSP extension by the way https://build-server-protocol.github.io/docs/extensions/scala#testparamsdata-kinds

[KristianAN]

The second use-case is probably useful for a few scenarios, tagging slow tests, tagging integration tests, tagging tests that should only run in CI, tagging flaky tests and so forth.

However I think the approach used by bleep is better as it is, and it seems like sbt is also moving towards a similar structure as it has deprecated the IntegrationTest setting and prefers another module for it.

[oyvindberg]

Sorry, very busy week. I pushed a test-only branch now, though I don't think it's worth much in its current state - it doesnt work

If you want to work on this let me know, and I'll do a writeup of what I was aiming for

[oyvindberg]

some foundations for this is going into #387 now - a command to list tests in projects

[KristianAN]

Yeah. I would like to contribute to this.

[oyvindberg]

Great!

So I'm not sure how useful the branch I pushed was, feel free to disregard it if you want.

BSP endpoints

I think the first thing to discover is how the BSP endpoints actually work. Try to adjust the commands and see if it's possible to make bloop run just a test suite, and if possible a test in a test suite.

This may be written in the BSP docs already, so read those well first. If we discover something which could make those docs better, maybe we can improve them slightly as part of this work.

CLI interface

Then after we get that working, we have to design a usable CLI interface for it. I'll introduce the complexity: Cross projects.

A given test suite may appear in multiple cross projects, and if it does it's fairly likely that you want to only run it in one particular cross project. Yet it would be beneficial if the syntax would let you specify which (cross) projects bleep should try to find the test suite in. For bigger projects it's crucial to be able to compile as little as possible to run the test.

So that likely means it should support the project globs from all the other commands somehow. The issue is that these are typically specified as varargs so you can have arbitrary many of them. And we also want to specify one or more (partial) test suite names. If we task BSP for test suite names we can match partial test suite names to actual test suites. Otherwise maybe the test frameworks do an OK job at that - I never tried. would be good to standardize the syntax if we can do it without too much overhead.

And if we want good tab completion for this, we first need to scope which projects, then lookup test suite names in those, then complete. So project glob(s) must go first.

To that end I started to look into the syntax <project glob>/<test suite name>. Then tab completion can trivially expand the project glob (including a trailing /). Then another tab completion afterwards should be able to ask bloop for test suite names in the specified projects and offer those as completions.

Thoughts?

Expand tab-completion support.

So the command line parser library used, decline, does not support tab completion at all. what is in bleep is homegrown, and I've meant to upstream it forever. It is however not quite clever enough, there would be no way to know earlier things from partially parsed command line the way the code works right now.

I think I know how to fix it, so if you find time to look into test-only, you can defer this particular task to me in a follow-up PR.

[oyvindberg]

Another interesting thought is that at some point we can add a tui interface to bleep with tui-scala.

In that case we could make all of this interactive, with check boxes for what you want to run, turn on/off watch mode, rerun failing tests and so on. Rerun failed tests would actually a fantastic feature in the CLI too, it would just require some state somewhere.

[KristianAN]

BSP endpoints

This is where I have been looking into so far and what I will look into further. As far as I can tell bloop just does not support this through BSP at all, but it should be fairly simple to implement support for it. Scala-cli get's around this the same way it gets around the linking issues that exist through BSP, it just does not use it for tests (as far as I can tell at least).

I still think that contributing this to bloop/bsp is the best way forward as it keeps the maintenance burden/internal complexity of bleep lower. The downside of going this way is that it will probably take a bit longer, but if that is not an issue and you agree that BSP/Bloop is the way to go I will continue in this direction.

[oyvindberg]

Yes, absolutely 👍