Feature Request: allow for let blocks

[schlichtanders]

Hi,

currently, @testitem does not allow for let end block instead of a begin end block. Instead it throws the following error:

The final argument of a @testitem must be a begin end block.

Allowing also for let has some advantages:

• It improves using @testitem inside Pluto. Background: I am using @testitem inside Pluto such that I run the content only when being inside Pluto. Following standard Julia semantics, a let block does not pollute the Pluto global namespace, while a begin block indeed does. Because @testitem most often will be self-contained, a let semantics makes a lot of sense. (This also enables the reuse of variable names in different @testitems)
• Even in normal Julia code, where nothing is run, the semantics of a let block fit better to how @testitems get executed, which would probably benefit code analysis, code jumping, linting and such.

[davidanthoff]

Hm, I'm confused, to be honest :) In particular, I don't understand

I am using @testitem inside Pluto such that I run the content only when being inside Pluto

The general idea is that when a @testitem appears in regular Julia code that whole macro block just gets removed before the actual Julia code runs. To run @testitem code one uses this package here, and it will extract the relevant code via static analysis, put it into its own module and then run. So the code in a @testitem should never run in the global namespace, and reuse of say variable names should never be a problem in the existing situation.

[schlichtanders]

Yes, this must be confusing. I created my own little @testitem macro which will run the code ONLY within Pluto. As TestItemRunner does not depend on the actual TestItems.@testitem, but just on the syntax @testitem this works seamlessly.

[davidanthoff]

Ah, ok. To be honest, I think that just sounds like a different test framework, and it might be better to just keep these separate? It sounds like the semantics of how code is run in your system is very different from how code is run in the test item system, so that just sounds confusing to me to try to use the same macro for both...

[schlichtanders]

I like to challenge this. The @testitem framework has many benefits which makes it the perfect fit for a combination with Pluto:

• you can put them right next to the code
• the macro itself does nothing

Imagine the following workflow:

1. you use a pluto notebook, say src/mynotebook.jl and edit it within Pluto
2. you have an include statement in your package src/MyPackage.jl which plainly includes the Pluto file
3. in test/runtests.jl you use the TestItemRunner.jl standard run

This enables you to use the very same test framework throughout the test systems.

• you can use julia -e "import Pkg; Pkg.test()"
• you can use vscode testitem integration (maybe for some other part of your package you prefer to use VSCode, or just use it for the testitem integration directly)
• you can use PlutoTest.jl inside Pluto (and only inside Pluto) to have interactive Pluto support for inspecting your tests.

This is really cool I can tell you from practice, and I think it has the potential to become widespread among Pluto users.

[davidanthoff]

If we can make this work, great! But I think let just doesn't provide the level of separation between different test items that we assume right now, really the design is that every test item is run inside its own module. Code like this is valid:

@testitem "a" begin
  struct Foo
    a
  end
end

@testitem "b" begin
  struct Foo
    b
  end
end

If you were to execute that code in the same module, even with let blocks, it won't work.

But I think more generally it would be helpful if you could explain how the integration with PlutoTest.jl would work? How would that detect tests and run them?

[schlichtanders]

thank you for the example - yes that wouldn't work within Pluto. But it is tiny exception. The cases which work out of the box are really helpful.

The integration with Pluto is setup like follows:

1. you have a pluto notebook.jl file which you can edit via Pluto
2. in your main module you include("notebook.jl") and make sure you export all variable names
3. in test/runtests.jl you have the usual using TestItemRunner; @run_package_tests

Now you can use @testitem inside Pluto by just defining your own version inside notebook.jl

is_running_in_pluto_process() = isdefined(Main, :PlutoRunner)
is_running_in_pluto_process() && using PlutoTest
macro testitem(exprs...)
    if is_running_in_pluto_process()
        esc(Expr(:toplevel, exprs...))
    end
end

and then use it directly

@testitem "mytest" begin
    var1 = 2
    @test var1 == 3
end

This will run inside Pluto using PlutoTest and run outside Pluto using the normal Test.

back to let vs begin blocks

Now comes the key difference: In Pluto, a begin block introduces new variable bindings to the global scope, while a let block does not, which follows standard julia semantics. Hence in most cases you probably want to use let blocks to reuse variable names which anyway shouldn't go to the global namespace. Sometimes however it is nice to have a test using begin/end in order to inspect the testitem more quickly in other pluto cells.

[davidanthoff]

Couldn't you just replace the begin end block that is passed to the macro with a let block in the macro if you want to do that? Or even better, surround that code with a module in the macro, then the semantics would really match the semantics of the original test item.

[schlichtanders]

Thank you for the tip with module wrap. Yeah that sounds quite reasonable.

I was feeling like it would be great to have some @testitem "name" let blocks, which leave the global namespace untouched, and others with begin, for polluting the namespace intentionally. I think the movitation was, because I thought the semantics were crystal clear.

But with the information that everything is run in its own module, I really appreciate to follow this semantics instead. Adding an alternative semantic looks confusing.

[schlichtanders]

Actually it does not work as nicely... the semantics of every testitem having its own module does not really translate well to Pluto:

• because you usually want to test other code from the same Pluto notebook

This is not easily possible when introducing a new module (you would also need to mimick somehow that every testitem starts with using MyPackage, where it is unclear what this should mean inside Pluto...)

• it would be far easier if a @testitem inside Pluto can plainly access all Pluto variables

Hence it seems to me that the let interpretation is the perfect compromise for use inside Pluto.

---

I would appreciate to be able to have both let and begin blocks in order to be more flexible inside Pluto, but I can also understand that TestItemRunner.jl wants to prevent confusions and just sticks to begin blocks.

[davidanthoff]

You could still just put it into a let block in your macro, right?

[davidanthoff]

I'm going to close this for now. In my mind the module-per-testitem is really core to the design.