Porting async tests

[lipp]

Just tried master (busted-v2) with http://github.com/lipp/lua-websockets and http://github.com/lipp/lua-jet projects, which use lot of async stuff. I get this output:

usr/local/opt/lua/bin/lua: .../share/lua/5.1/busted/outputHandlers/utfTerminal.lua:52: attempt to index field 'elementTrace' (a nil value)
stack traceback:
    .../share/lua/5.1/busted/outputHandlers/utfTerminal.lua:52: in function 'failureDescription'
    .../share/lua/5.1/busted/outputHandlers/utfTerminal.lua:165: in function 'fn'
    /usr/local/share/lua/5.1/mediator.lua:98: in function </usr/local/share/lua/5.1/mediator.lua:91>
    (tail call): ?
    (tail call): ?
    ...ocal//lib/luarocks/rocks/busted/2.0.rc0-0/bin/busted:182: in main chunk
    [C]: ?

Are there any breaking API changes for async tests?

[ajacksified]

There are some changes, although you shouldn't be getting elementTrace error output. I'll take a look today.

[ajacksified]

Try pulling latest; I'm now getting some actual errors.

That said, the async API changed slightly: check out the docs. The way we rewrote busted, we were able to rip out the ev/copas specific code and use generic async methods.

[lipp]

@ajacksified Ok. This will take some time. I guess I have to modify most tests. Some questions:

• Do tests still run one after another? Or are they "detached" after calling async()?
• Has finally gone?
• Is it guaranteed that the done callback does not mess up my variables (done is used in really code of mine).
• can i use async in setup/teardown after_each/before_each

[ajacksified]

• tests will start sequentially, but will not necessarily finish sequentially. They'll finish whenever they finish.
• finally still exists and works the same as before.
• done usage shouldn't have changed.

I think you can use async in before_each, but I'll pull in @DorianGray to confirm.

[lipp]

• that means they are started sequentially and then "detached" continuing with the next test? (after calling async)?
• great
• ok

One more question: Do I have to start the "event-loop" in copas/lua-ev manually? If so, where in the test is the designated place to do that?

[DorianGray]

I've removed async from busted 2.0 for now. I will be refactoring and reimplementing async in a much cleaner way soon.

[lipp]

@DorianGray Ok. It would be very nice, if we could keep the interface as it was in 1.x. Mocha has a pretty similar interface and it's not the worst around I guess.

[DorianGray]

The interface will be similar, although it will not provide an event loop for you like it did in busted 1.0. It will just make it so that done() is not called when the test ends, so you can call it yourself when async code is complete.

I'm currently debating two different approaches...

it('does a thing', function()
  local done = async() -- optionally async would inject done into local environment
    foo_async(function()
      done()
    end)
end)

or

async('does a thing', function()
    foo_async(function()
      done()
    end)
end)

[ajacksified]

The former is nice because it's explicit (I prefer async returning done to it to injecting done), although the latter is nice because it's shorter. I'm :+1: for either approach.

[tarruda]

@DorianGray I'm not sure if the following is possible with lua, but this API would be nice:

it('does a thing', function(done)
     foo_async(function()
      done()
    end)
end)

This is how mocha(javascript test framework) implements async testing. What happens is that when a function with one argument is passed ad the it callback, mocha assumes it is an asynchronous test and passes the completion callback to the test function.

As I said, it requires detecting how many parameters were declared by a function, and I don't know if its possible in lua.

[mpeterv]

@tarruda detecting number of parameters is possible using debug.getinfo(func, "u").nparams; however, this is not supported in Lua 5.1 (LuaJIT does support it).

[DorianGray]

I want to take async in a different direction than mocha. In javascript, everything is single threaded and the event loop is core to the language. In lua, we have threads, event loops, etc and we need to support all the concurrency models.

I'm thinking of adding hooks into busted so you can configure a test scheduler yourself, tie in to copas or libev or lanes or whatever. The interface would be something like saying local done = async() in an it(). The async() function might take an optional argument to use a different scheduler than the default. If you call the async function, you are responsible for calling done() at the end of your test. Each scheduler would be responsible for implementing timeouts, etc.

The reason I want to decouple the scheduler from the core is because of what happened with busted 1. Massive spaghetti code to support multiple schedulers, and the introduction of a weird stack overflow condition that was the reason for the rewrite for busted 2.

[milani]

What is the plan here? Anyone wants to continue the work?