support `(inline_tests)` for Rely

[tcoopman]

Is there a reason why Rely doesn't support the inline_tests stanza from dune (https://jbuilder.readthedocs.io/en/latest/tests.html)?

This would make it more obvious for dune users how to run the tests (esy dune runtest)

[jordwalke]

I actually hadn't encountered dune's (inline_tests) field and I assumed it would only apply if you were using the ppx inline tests extension. Is my understanding wrong? We're open to suggestions about how to structure this. The main thing I'm aware that we could take advantage of is the promote operation to promote snapshot tests outputs.

[tcoopman]

I'm also not very familiar, but it's afaik not only for this ppx inline tests extension. There is a section on how to write a custom backend for it: https://dune.readthedocs.io/en/latest/tests.html#defining-your-own-inline-test-backend

It might be a better option to just go for the option of running custom tests with dune: https://dune.readthedocs.io/en/latest/tests.html#custom-tests

Maybe it's a good option to ask someone who is more familiar with dune what the best approach would be.

[kyldvs]

I believe @bandersongit looked into using the inline_test stanza. I think the problem was that the output is only flushed at the very end so it looks like tests are frozen and not doing anything, then when they are done you get all the output at once.

(I could be wrong, Ben could confirm)

[bandersongit]

I looked into using the test stanza (see https://github.com/bandersongit/rely-dunetest-example). I ran into exactly what Kyle described and didn't really like the look of the output.

I didn't specifically look into inline tests and imagine that if you wanted you could pretty trivially implement a custom backend. Based on the docs, it looks like it uses a predefined ppx (ppx_inline_test), but you can write whatever backend you want (the qtest example they gave may be worth looking into more). I think it would absolutely be cool to play around with if you are passionate about it, but in general I don't think that inline tests are a user experience that we are really looking into supporting/promoting right now.

[bandersongit]

I've looked at it a bit more in depth since my previous comment. I think we would probably need some kind of lexer a la qtest to actually implement this. I think the rest of my previous conclusions are still valid in that I think the primary focus of Rely is the separate test library/files workflow a la Jest/xUnit rather than encouraging people to write inline tests for a variety of reasons, and that while we would certainly welcome an inline tests backend that used Rely under the hood, I don't think it's something that I am likely to build.