[BUG] Sometimes tests aren't detected

[igorlfs]

NeoVim Version Output of nvim --version

NVIM v0.9.0-dev-43-g6cd643dbf9
Build type: RelWithDebInfo
LuaJIT 2.1.0-beta3

Describe the bug

Sometimes tests aren't detected when using pytest on Linux.

To Reproduce

Minimal init.lua:

-- ignore default config and plugins
vim.opt.runtimepath:remove(vim.fn.expand("~/.config/nvim"))
vim.opt.packpath:remove(vim.fn.expand("~/.local/share/nvim/site"))
vim.opt.termguicolors = true

-- append test directory
local test_dir = "/tmp/nvim-config"
vim.opt.runtimepath:append(vim.fn.expand(test_dir))
vim.opt.packpath:append(vim.fn.expand(test_dir))

-- install packer
local install_path = test_dir .. "/pack/packer/start/packer.nvim"
local install_plugins = false

if vim.fn.empty(vim.fn.glob(install_path)) > 0 then
    vim.cmd("!git clone https://github.com/wbthomason/packer.nvim " .. install_path)
    vim.cmd("packadd packer.nvim")
    install_plugins = true
end

local packer = require("packer")

packer.init({
    package_root = test_dir .. "/pack",
    compile_path = test_dir .. "/plugin/packer_compiled.lua",
})

packer.startup(function(use)
    -- Packer can manage itself
    use("wbthomason/packer.nvim")

    use({
        "nvim-treesitter/nvim-treesitter",
        config = function()
            require("nvim-treesitter.configs").setup({
                ensure_installed = { "python" },
                highlight = {
                    enable = true,
                },
            })
        end,
    })
    use({
        "nvim-neotest/neotest",
        requires = {
            "nvim-lua/plenary.nvim",
            "nvim-neotest/neotest-python",
        },
        config = function()
            require("neotest").setup({
                adapters = {
                    require("neotest-python")({
                        dap = { justMyCode = false },
                        args = { "--log-level", "DEBUG" },
                        runner = "pytest",
                    }),
                },
            })
        end,
    })

    if install_plugins then
        packer.sync()
    end
end)

vim.cmd([[
command! NeotestSummary lua require("neotest").summary.toggle()
command! NeotestFile lua require("neotest").run.run(vim.fn.expand("%"))
command! Neotest lua require("neotest").run.run(vim.fn.getcwd())
command! NeotestNearest lua require("neotest").run.run()
command! NeotestDebug lua require("neotest").run.run({ strategy = "dap" })
command! NeotestAttach lua require("neotest").run.attach()
]])

Steps to reproduce the behavior:

1. Launch neovim with

nvim --clean -u minimal.lua test_my_module.py

Where "test_my_module.py" contains pytest's tests. 2. Try to run a test using :NeotestNearest, it may not find the tests. If so, saving the file allows running the tests.

Example test file:

class Mod:
    def __init__(self, s: str) -> None:
        self.s = s

    def cmp(self, rhs) -> bool:
        return self.s == rhs.s


def test_cmp():
    a: Mod = Mod("a")
    b: Mod = Mod("b")

    assert not a.cmp(b)

Expected behavior Tests always should be found.

Logs

neotest.log

ERROR | 2022-10-09T17:13:02Z-0300 | ...ig/pack/packer/start/neotest/lua/neotest/client/init.lua:274 | Couldn't find positions in path .../python/proj .../pack/packer/start/neotest/lua/neotest/lib/file/find.lua:28: bad argument #1 to 'fs_scandir_next' (uv_req expected, got nil)

[rcarriga]

Woops didn't mean to close :facepalm: Thanks for the report! I'm not sure what the cause is but the error message should be improved by latest changes if you can reproduce

[igorlfs]

Hello!

After updating I get the following error:

ERROR
 .../neotest/lua/neotest/client/init.lua:274
Couldn't find positions in path [[PATH_TO_FILE]] .../neotest-python/lua/neotest-python/base.lua:63: attempt to index field 'stdout' (a nil value)

If try to run the tests again in the same session I get this warning:

WARN  
.../neotest/lua/neotest/client/runner.lua:26
 Position already running: [[PATH_TO_PROJECT]]

[rcarriga]

Believe this is unrelated, can you check with the latest update to neotest-python?

[igorlfs]

With the latest changes, I now get the following error:

ERROR
 ../neotest/lua/neotest/client/init.lua:274
Couldn't find positions in path .../python/proj 
...neotest/lua/neotest/lib/file/find.lua:30: ENOENT: no such file or directory: src

Where .../python/proj is the path to the project and src is a directory inside proj containing a folder called test with the tests themselves, i.e., my test file is .../python/proj/src/test/test_my_module.py

[rcarriga]

Ah that's perfect, it was because the project root was not the cwd and so relative paths wouldn't work. Should be working as expected now :smile:

[igorlfs]

Just noticed something: while I can now always run the tests as expected, sometimes they are being loaded twice in the summary:

[rcarriga]

Can you provide another log at the info level? If the file paths are sensitive, feel free to alter them after src/test/ and substitute the root directory for ROOT

[halfdan]

This isn't just happening with Python - I am getting the same behaviour with my Elixir project. Saving the file unchanged will somehow make neotest discover the tests. Then, when I get to run the tests I'm seeing:

Error executing luv callback:
...ck/packer/start/plenary.nvim/lua/plenary/async/async.lua:18: The coroutine failed with this message: .../pack/packer/start/neotest/lua/neotest/lib/file/init.lua:96: Data deleted from file while streaming
stack traceback:
        [C]: in function 'error'
        ...ck/packer/start/plenary.nvim/lua/plenary/async/async.lua:18: in function 'callback_or_next'
        ...ck/packer/start/plenary.nvim/lua/plenary/async/async.lua:45: in function <...ck/packer/start/plenary.nvim/lua/plenary/async/async.lua:44>

[rcarriga]

Can you provide an info log as well please?

[halfdan]

@rcarriga Had it set to debug but here you go: https://gist.github.com/halfdan/23223cc1e762a69effde1173fcc7d711#file-neotest-log. Note that L286 is where I triggered a file save with :w.

In my case I'm on 0.8.0 stable. Latest version of both neotest and neotest-elixir.

[halfdan]

Ok, additional observations:

• Saving doesn't help unless I first try to trigger a run with `require'neotest'.run.run()'.
• Autocmds are only registered after I try to trigger a run

[halfdan]

• The call args here are never used: https://github.com/nvim-neotest/neotest/blob/master/lua/neotest/init.lua#L81

[halfdan]

Ok, so I think I know what is happening.

• I open neovim and neotest initialises. A client is initialized but not started.
• I enter a new buffer (in my case an exs file)
• I run require'neotest'.run.run() which calls get_adapters on the client. This in turn calls :ensure_started() which only then starts the client and registers all the autocmds. But this is too late for the run()
• When I :w the buffer the autocmd with BufWritePost gets triggered and causes a reindex with the adapter

[halfdan]

One way I found to fix the issue is by adding a new autocmd to NeotestClient:new() with:

  autocmd('BufReadPre', function ()
    neotest:ensure_started()
  end)

The better alternative is to extract the function from the BufAdd, BufWritePost autocmd and call it at the end of :_start() ensuring that the current file is getting evaluated without any autocmds triggering.

This at least makes discovery work.

[rcarriga]

OK I've fixed this issue https://github.com/nvim-neotest/neotest/issues/127#issuecomment-1286345292 @igorlfs

@halfdan Thanks for the PR but can you check with the latest changes if you're still seeing the same behaviour?

[halfdan]

@rcarriga Just tested and test discovery is still not working. Only after save will test discovery work. I debugged the issue extensively yesterday - the client is started after the buffer is already open and so the BufAdd event is missed when neotest is first initialized. It requires a BufWritePost to trigger the test scan - that's why it works after a save.

[rcarriga]

The problem is that the buffer should be properly detected when updating the adapters https://github.com/nvim-neotest/neotest/blob/f7db06d13157396077aecaff6b977d9360cbc9e6/lua/neotest/client/init.lua#L425

So your fix is hiding the underlying issue AFAIK, which I'd like to figure out. If you can provide a way to reproduce, I'm happy to debug further myself

[halfdan]

I'm able to reproduce this on a minimal config with just neotest-elixir and the elixir treesitter grammar. It seems to be weirdly tied to the file I'm using though.

I also just noticed that with neotest.run.run(vim.fn.getcwd()) the tests do actually run! But the signs don't show - and the summary is empty.

When I try to run NeotestNearest that's when I get the message that there's no tests found.

I'll continue to try and find a minimal case for you. Otherwise - given that we're in the same tz I'm happy to also hop on a call, if that's something you'd be up to.

[halfdan]

Ok. It's going to be a weird one!

minimal.lua:

-- ignore default config and plugins
vim.opt.runtimepath:remove(vim.fn.expand("~/.config/nvim"))
vim.opt.packpath:remove(vim.fn.expand("~/.local/share/nvim/site"))
vim.opt.termguicolors = true

-- append test directory
local test_dir = "/tmp/nvim-config"
vim.opt.runtimepath:append(vim.fn.expand(test_dir))
vim.opt.packpath:append(vim.fn.expand(test_dir))

vim.cmd("colorscheme desert")

-- install packer
local install_path = test_dir .. "/pack/packer/start/packer.nvim"
local install_plugins = false

if vim.fn.empty(vim.fn.glob(install_path)) > 0 then
  vim.cmd("!git clone https://github.com/wbthomason/packer.nvim " .. install_path)
  vim.cmd("packadd packer.nvim")
  install_plugins = true
end

local packer = require("packer")

packer.init({
  package_root = test_dir .. "/pack",
  compile_path = test_dir .. "/plugin/packer_compiled.lua",
})

packer.startup(function(use)
  use("wbthomason/packer.nvim")
  use({
	  "nvim-treesitter/nvim-treesitter",
	  config = function()
		  require("nvim-treesitter.configs").setup({
			  ensure_installed = { "elixir" },
			  highlight = {
				  enable = true,
			  },
		  })
	  end,
  })
  use({
    "nvim-neotest/neotest",
    requires = {
      "vim-test/vim-test",
      "nvim-lua/plenary.nvim",
      "nvim-treesitter/nvim-treesitter",
      "antoinemadec/FixCursorHold.nvim",
      "jfpedroza/neotest-elixir",
    },
    config = function()
      require("neotest").setup({
        adapters = {
          require("neotest-elixir"),
        },
      })
    end,
  })

  if install_plugins then
    packer.sync()
  end
end)

vim.cmd([[
command! NeotestSummary lua require("neotest").summary.toggle()
command! NeotestFile lua require("neotest").run.run(vim.fn.expand("%"))
command! Neotest lua require("neotest").run.run(vim.fn.getcwd())
command! NeotestNearest lua require("neotest").run.run()
command! NeotestDebug lua require("neotest").run.run({ strategy = "dap" })
command! NeotestAttach lua require("neotest").run.attach()
command! NeotestOutput lua require("neotest").output.open()
]])

What works:

1. Clone this repo: https://github.com/plausible/analytics.git
2. Open with nvim --clean -u ~/minimal.lua test/plausible/ingestion/event_test.exs
3. Test discovery works!

What doesn't work:

1. Clone this repo: https://github.com/plausible/analytics.git
2. Run mkdir -p apps/core/ && mv test apps/core (we have a similar structure in our private codebase)
3. Open with nvim --clean -u ~/minimal.lua apps/core/test/plausible/ingestion/event_test.exs
4. Tests remain unavailable until :w (as can be verified with :NeotestSummary

[igorlfs]

OK I've fixed this issue https://github.com/nvim-neotest/neotest/issues/127#issuecomment-1286345292 @igorlfs

Thanks! I haven't been able to reproduce the issue during the week, as it was happening rarely under unknown circumstances. I haven't run into any issues since the fix, though I haven't tested extensively.

[rcarriga]

Thanks for the reproduction! Managed to figure this one out. The elixir adapter has a filter_dir function means that tests not under a test/ root directory will not be discovered, so you'll need to open a request to make that configurable (background context https://github.com/nvim-neotest/neotest/issues/13#issuecomment-1243256345).

The reason it was being discovered was actually a bug where it was being discovered as a test file with no root dir which should now not happen

[jfpedroza]

Oops. Totally forgot about umbrella projects. That was just fixed on my side. @halfdan

[halfdan]

Amazing - thanks @rcarriga & @jfpedroza! I did look extensively through the plug-in code but somehow I didn't register the filter function. I'll check on Monday but sound like this is all fixed now 😀