Async lua picker job

[matheusfillipe]

Is your feature request related to a problem? Please describe. The current jobs are only for running commands in the background. I don't seem to find a way to run lua code in background for a picker. It is always blocking the main thread. There is only Dynamic:

  local dyn_finder = function(prompt)
      if not prompt or prompt == "" then
        return nil
      end

      query = prompt or query
      local results, lang_suggestions = grepclient.Grep(query, api_params, opts.max_results)
      return results
    end

  opts.entry_maker = result_entry_maker
  opts.fn = dyn_finder

  local live_grepper = finders.new_dynamic(opts)

  opts.sorting_strategy = "ascending"

  pickers.new(opts, {
    title = "Live grep.app",
    prompt_title = "Search",
    finder = live_grepper,
    previewer = make_previewer(opts),
    sorter = sorters.highlighter_only(opts),
    attach_mappings = function(prompt_bufnr)
      actions.select_default:replace(function()
        actions.close(prompt_bufnr)
        action_picker(opts, action_state.get_selected_entry())
    end)
    return true
  end
  }):find()

On this case grepclient.Grep(query, api_params, opts.max_results) is a lua function that makes http requests, so it is slow. The prompt keeps getting blocked.

Describe the solution you'd like A job like finders.new_job that doesn't expect a command to spawn and that runs lua code in the background for each prompt change.

Sorry if this already exists but I haven't been able to find a way.

[chuwy]

Also ran into this problem. It seems that existing async finders use pipes to populate the picker with results:

https://github.com/nvim-telescope/telescope.nvim/blob/dc192faceb2db64231ead71539761e055df66d73/lua/telescope/finders/async_job_finder.lua#L48-L72

I'm hoping that if we can write callbacks like following:

function async_fn(on_result)
  return function(prompt)
    curl.get(prompt, { callback = function(res)
      for i, item in ipairs(json.decode(res.body)) do
        on_result(i, item)
      end
    end})
  end
end

Where on_result is a callback provided by the finder itself, responsible for assigning an index and calling process_result.

Also, I think that way we can add debouncing into async functions like that.

[delphinus]

Another solution, I recently made an asynchronous finder to search files under a directory.

https://github.com/nvim-telescope/telescope-frecency.nvim/blob/2ac311a2666edb447db5139b326777c44adc1e2a/lua/frecency/async_finder.lua

It runs a thread (coroutine) created by plenary.async and makes candidates in background. The picker's main loop thread calls this finder and it communicates with a channel to receive candidates.

Perhaps more generalized approach can be included in telescope.nvim itself, I think.

[chuwy]

Here's a somewhat naive implementation I have for above API: https://gist.github.com/chuwy/e8476156d6dd4815611228dc96196554. I think it's generic enough... but unfortunately laggy and I cannot make debouncing working with it.

UPD: final version seems to be working fine.

[tsurumi-yizhou]

So what's the solution? How can I load remote data in fn of finders.new_dynamic?