A universal way to hook into frameworks

[RealyUniqueName]

Is it possible to implement something like the subject? So that a framework maintainer could provide a required API and get the framework automatically integrated into VSCode test runner.

[AlexHaxe]

Basically test adapter needs three hooks / callbacks:

1. filtering of testcases
2. reporting one or more test results
3. reporting finished

Right now we hook into supported frameworks quite aggressively through macros, but that also makes it work without modification in user code or in framework code - so you can use it even if the project in question didn't plan for it.

If frameworks start to have an interface and become "test adapter aware", then we could reduce macro injection for those frameworks. We would have to find a way to create those hooks without adding a permanent dependency to lib test-adapter, because that would enable test adapter permanently for everyone. I am not sure frameworks want to sprinkle their code with #if test_adapter, but we could create a an adapter aware class, that has all imports and functionality hidden behind conditionals, so that a normal compile works as before and doesn't have test adapter functionality. Only when compiled against -lib test-adapter that class passes things on to test adapter.

We could create a sample class for frameworks to use, but ultimately that class needs to sit inside the frameworks themselves since using the lib activates it.

Maybe we need a separate framework hook lib, that holds a hooking class and nothing else. Frameworks could link against that and wouldn't have to bother with any test adapter internals or changes. (I guess this would also work for #1)

[Gama11]

I think we're already doing quite well using utest's reporter API for 2. and 3., and that's probably the approach that gives us the most flexibility (since we have access to everything the reporter API offers). Most test frameworks seem to have something like that.

If the reporter API isn't powerful enough in some areas, that should probably be what's extended rather than having some test-adapter specific solution, so everybody using reporters can benefit from it.

The two things that are missing are ways to register a "default reporter" for a TestRunner and a way to filter tests. I wonder if it would make sense for UTest to expose an API that looks something like this:

class UTestHook {
    public static dynamic function shouldRunTest(test:utest.ITest):Bool {
        return true;
    }

    public static dynamic function createDefaultReporter():Null<utest.ui.common.IReport> {
        return null;
    }
}

Not sure if we'd want to utilize the dynamic modifier to redefine the methods, since we'd need to find a way to do that before main(), maybe in a static initializer? Otherwise, attaching a build macro to UTestHook should work fine too..

[RealyUniqueName]

Another approach could be detecting test-adapter library from the test framework side:

//somewhere in a test framework
#if test_adapter_api
this.subscribeShouldRun(testadapter.Api.shouldRun);
#end

[Gama11]

But what's the advantage of that? Just seems a lot less flexible, those generic hooks might be useful for others as well, not just test-adapter.

[RealyUniqueName]

I mean test-adapter should not be aware of all the test frameworks out there. Instead, it could provide a service, which will be consumed by test frameworks.

[Gama11]

I'm not sure about that, if we ever want to change something on the test-adapter side, we'd then have to make pull requests to 6 different frameworks? :) It's a lot simpler if it has all the control and doesn't expose any API that has to remain compatible.

I'm not sure how many frameworks would implement such an API anyway. haxe.unit certainly wouldn't, and there's not much development for munit beyond Haxe compatibility fixes. The hexUnit guys mostly use FD I think. So we wouldn't have a consistent solution, but a situation where half use the API and half macro hooks as before.

[RealyUniqueName]

Ok, let's do it your way. Could you design an API utest should provide?

[Gama11]

I already gave an example of what it could look like in my earlier comment. Not sure if it should be a separate UTestHook class, or live in the existing UTest.hx, or somewhere else entirely?

[RealyUniqueName]

A default reporter is already created in utest.ui.Report.create(). I think I can make it a dynamic function.

[Gama11]

Hm, wouldn't that mean that utest suddenly no longer prints anything when using it with test-adapter?

[RealyUniqueName]

But why? I thought test adapter is only plugged in the build when running through the vscode ui

[Gama11]

Not necessarily, sometimes it's useful to put it into the hxml as well, for instance for debugging (we can't inject -lib there). And some people still want to see the results in ther Terminal even when running it through the UI.