ex_unit: Add :capture_io tag

[wojtekmach]

Example:

defmodule MyTest do
  use ExUnit.Case, async: true
  @tag :capture_io
  test "with io", %{capture_io: io} do
    IO.puts("Hello, World!")
    assert StringIO.flush(io) == "Hello, World!\\n"
  end
end

---

Possible future work:

1. Allow ExUnit.start(capture_io: true) to mimic capture_log: true?

2. Add api to programmatically append to the StringIO input buffer, right now it's only possible on creation:

   {:ok, pid} = StringIO.open("3")
   "3" = IO.gets(pid, "1 + 2")
   

   The idea is something like this:

   {:ok, pid} = StringIO.open("")
   
   StringIO.buffer("1 + 2")
   "1 + 2" = IO.gets(pid, "> ")
   
   :eof = IO.gets(pid, "> ")
   
   StringIO.buffer("2 + 3")
   "2 + 3" = IO.gets(pid, "> ")
   

   And so we could replace the usage of:

   capture_io(prompt, fn -> ... end)
   

   too.

   Maybe instead of calling it StringIO.buffer it could be called StringIO.input_write and StringIO.input_puts to mimic IO.write and IO.puts. Not sure if we'd need .input_binwrite too.

3. @tag capture_io: :standard_error?

   I think we could do this as long as the test is running in async: false which we'd know and then we can raise otherwise. This is as opposed to capture_io(:standard_error, ...) where ExUnit does not know if it's sync or async.

   If we do this, do we want to capture both? I guess we could accept a list of devices and set it in context:

   @tag capture_io: [:stdio, :standard_error]
   test "foo", %{capture_io: [stdio, stderr]}
   

   Not sure if :stdio is even semantically correct, my understanding is it's not stdio per se but the group leader. So yeah, it's probably a pass, and a reason to use capture_io in this case, but I thought I'd mention this for completeness.

4. Make the StringIO available to @tag :capture_log tests in context as %{capture_log: io}. This could be considered a breaking change since today the context is usually set as %{capture_log: true} or %{capture_log: options} but I can't think of a reason to read them back in a test.

[sabiwara]

StringIO.buffer("1 + 2")

StringIO.buffer(pid, "1 + 2") right?

Make the StringIO available to @tag :capture_log tests in context as %{capture_log: io}. This could be considered a breaking change since today the context is usually set as %{capture_log: true} or %{capture_log: options} but I can't think of a reason to read them back in a test.

This one seems solvable by picking a slightly different name, captured_log: / captured_io:?

[josevalim]

@tag capture_io: :standard_error?

Right, this doesn't play well with StringIO.flush, because then you are flushing data between tests. I think it is fine to keep this specific to stdin/stdout. It is potentially the same reason why adding a similar API to capture_log would be confusing, you cannot use flush and that would lead to more imprecise tests. So I believe this is good to go as is.

[josevalim]

The only thing holding this PR is to decide if we want to add a similar API to capture logs, such as:

@tag :capture_log
test "foo", %{capture_log: log} do
  assert StringIO.flush(log) =~ "..."
end

The issue is that we cannot add capture_log: log as that would be backwards incompatible. Therefore one option is to store the device under another name, such %{stdio: device, logs: device}.

---

I also realize that I lied. We could totally do capture_io: :stderr, as we give specific StringIO devices per process. So we also need an API for multiple devices. One option is to just place them as tags:

test "foo", %{captured_stdio: ..., captured_stderr, captured_logs: ...} do

But I am thinking some namespacing is probably best?

test "foo", %{devices: devices} do
  devices.stdio
  devices.stderr
  devices.logs
end

Possible names are devices or captured. Thoughts?

[wojtekmach]

test "foo", %{devices: devices} do
  devices.stdio
  devices.stderr
  devices.logs
end

I like this direction.

I wonder, would the full example be something like this?

@tag :capture_io # shortcut for capture_io: :stdout?
@tag capture_io: :stderr
@tag :capture_log
test "foo", %{devices: devices} do
  devices.stdio
  devices.stderr
  devices.logs
end

Or maybe explicit tag names:

@tag :capture_stdio
@tag :capture_stderr
@tag :capture_log
test "foo", %{devices: devices} do

Cause I'm not sure if we'd want to capture other devices in practice.

Or did you have something else in mind?

Could be namespaced on both input and output, capture/captured:

@tag capture: [:stdio]
test "foo", %{captured: %{stdio: io}}

@tag capture: [:stdio, :stderr, :logs]
test "foo", %{captured: %{stdio: _, stderr: _, logs: _}}

but I think @tag :capture_* reads way better.

I think :captured reads better than :devices, especially logs have their own devices I guess, but both are good imo.

[josevalim]

I think this reads well:

@tag :capture_stdio
@tag :capture_stderr
@tag :capture_log

They may receve options:

@tag capture_stdio: "input to be given, no need for buffer"
@tag :capture_stderr
@tag capture_log: :error

So not sure if we should do the nesting as we may end up with this: @tag capture: [:stderr, stdio: "buffer", log: :error]. But we could have @tag capture: :all... although I don't think I ever wanted to capture all three in practice.

---

Undecided between devices and captured. captured may read weird because nothing may been captured when we define the variable. It is all imperative.