Tests failing for Elixir 1.7

[akash-akya]

78 test cases are failing with Elixir 1.7.4 (OTP 21.1). Most of them are documentation related failures, which is understandable. Few doesn't seem like version update error like

 9) ElixirSense.ServerTest: failure on setup_all callback, test invalidated
     ** (MatchError) no match of right hand side value: {:error, :nxdomain}
     stacktrace:
       test/server_test.exs:31: ElixirSense.ServerTest.__ex_unit_setup_all_0/1
       test/server_test.exs:18: ElixirSense.ServerTest.__ex_unit__/2

 1) test build metadata from kernel.ex (ElixirSense.Core.MetadataBuilderTest)
     test/elixir_sense/core/metadata_builder_test.exs:7
     ** (File.Error) could not read file "/private/tmp/elixir-20181026-69137-ms45yr/elixir-1.7.4/lib/elixir/lib/kernel.ex": no such file or directory
     code: assert get_subject_definition_line(Kernel, :defmodule, nil) =~ "defmacro defmodule(alias, do: block) do"
     stacktrace:
       (elixir) lib/file.ex:319: File.read!/1
       test/elixir_sense/core/metadata_builder_test.exs:598: ElixirSense.Core.MetadataBuilderTest.get_subject_definition_line/3
       test/elixir_sense/core/metadata_builder_test.exs:8: (test)

For second error, I think its because im using precompiled elixir from brew.

$ elixir --version

Erlang/OTP 21 [erts-10.2.1] [source] [64-bit] [smp:12:12] [ds:12:12:10] [async-threads:1] [hipe] [dtrace]

Elixir 1.7.4 (compiled with Erlang/OTP 21)

[axelson]

@msaraiva what version of Elixir and Erlang do the tests pass with? I've been trying many different versions but have been unable to get all the tests to pass.

[axelson]

Okay, I have the tests passing with Elixir 1.7.0 and OTP 21.2. BUT, you have to build elixir from source.

[OvermindDL1]

@axelson If you are using asdf does asdf install elixir 1.7.0-otp-21 work? Most elixir installers (like in most package repositories) are built for the lowest possible OTP version instead of the one that is installed on the system, being explicit (like asdf makes it easy to be) allows you to match the installed OTP version and thus gain any new abilities it may have that Elixir can take advantage of.

[axelson]

@OvermindDL1 no that doesn't work (although I really wish it did!). It looks like it has to truly be from source for these tests since they return the actual source code path:

    buffer = """
    defmodule MyModule do
      alias Mix.Generator
      use Generator
    end
    """
    definition = ElixirSense.definition(buffer, 3, 12)
    IO.inspect(definition, label: "definition")

Gives:

%ElixirSense.Providers.Definition.Location{
  column: 11,
  file: "/home/jason/.kiex/builds/elixir-git/lib/mix/lib/mix/generator.ex",
  found: true,
  line: 1,
  type: :module
}

I even tried installing ref:v1.7.1 through asdf but I ran into issues. Or maybe I didn't. I thought I did but it is working now and returning a path like: file: "/home/jason/.asdf/installs/elixir/ref-v1.7.1/lib/mix/lib/mix/generator.ex" which will work well for my purposes. Now if I can only get this working on Travis-ci...

[msaraiva]

Sorry for the late response. Two important things here:

• The current test suite is compatible with Elixir 1.7.1. The way documentation tests are implemented today is far from ideal. It leads to this kind of errors since a lot of the tests assert against real documentation. When those docs change, tests fail. I need to find some time to minimize this problem by testing the features I want against predefined modules with documentation written specially for the tests. I'll try to rewrite as many as possible when updating to Elixir 1.8 in the next few days.
• In order to have all tests passing, you also need to install Elixir with sources, otherwise lots of tests from "Go to definition" will fail since they will not be able to find the definitions.

[OvermindDL1]

It looks like it has to truly be from source for these tests since they return the actual source code path:

Oooo, that's very much a broken test then! Such return values should not be tested against directly but picked apart and those tested against (or match?/2 used). ^.^;

[msaraiva]

@OvermindDL1 the return values are not tested directly, they are tested using =~:

  test "find definition of aliased modules in `use`" do
    buffer = """
    defmodule MyModule do
      alias Mix.Generator
      use Generator
    end
    """
    %{found: true, type: :module, file: file, line: line, column: column} = ElixirSense.definition(buffer, 3, 12)
    assert file =~ "lib/mix/lib/mix/generator.ex"
    assert read_line(file, {line, column}) =~ "Mix.Generator"
  end

So, if the file exists, it should pass.

@axelson can you confirm which of the 2 assertions above is failing?

[axelson]

@msaraiva Neither of those assertions are failing if I have installed Elixir from source. But if I have installed with a binary, then the first assertion fails since ElixirSense.definition/3 returns:

%ElixirSense.Providers.Definition.Location{
  column: nil,
  file: nil,
  found: false,
  line: nil,
  type: nil
}