`mix test --warnings-as-errors` fail on runtime warnings

[wojtekmach]

Before this patch, mix test --warnings-as-errors failed only on compilation warnings. For example even though this emitted a warning:

defmodule WarningTest do
  use ExUnit.Case

  test "warning" do
    inspect(~c"foo", char_lists: :as_char_lists)
  end
end

warning: the :char_lists inspect option and its :as_char_lists value are deprecated, use the :charlists option and its :as_charlists value instead

it did not fail the test suite. After this patch it would.

The function IO.warning_emitted?/0 (exact name TBD) is useful outside of Elixir itself. For example, with it, implementing ExDoc mix docs --warnings-as-errors is trivial:

--- a/lib/mix/tasks/docs.ex
+++ b/lib/mix/tasks/docs.ex
@@ -322,7 +322,8 @@ defmodule Mix.Tasks.Docs do
     language: :string,
     open: :boolean,
     output: :string,
-    proglang: :string
+    proglang: :string,
+    warnings_as_errors: :boolean
   ]

   @aliases [
@@ -344,6 +345,18 @@ defmodule Mix.Tasks.Docs do

     {cli_opts, args, _} = OptionParser.parse(args, aliases: @aliases, switches: @switches)

+    if cli_opts[:warnings_as_errors] do
+      System.at_exit(fn _ ->
+        if IO.warning_emitted?() do
+          message =
+            "\nERROR! Docs generation aborted after successful execution due to warnings while using the --warnings-as-errors option"
+
+          IO.puts(:stderr, IO.ANSI.format([:red, message]))
+          exit({:shutdown, 1})
+        end
+      end)
+    end
+
     if args != [] do
       Mix.raise("Extraneous arguments on the command line")
     end

[wojtekmach]

Nevermind, all is good, I was testing it wrong. Sorry about that.

[wojtekmach]

I've updated the PR description to mention this would fail on "runtime warnings".

[josevalim]

I am not sure if we want to expose a warning API at runtime. Especially because we want to avoid runtime warnings and, if you emit them at compile time, the compiler will already handle those. I assume this can be used by projects like ExDoc, but ExDoc could set a similar flag when it calls IO.warn, no?

[wojtekmach]

Yes, adding public API to Elixir is not strictly necessary for ExDoc, wrapping its calls to IO.warn and checking those will be good enough.

[josevalim]

So let's go that route for now and we can revisit this if there is more evidence!