Preview doesn't work for utf-16le files

[ghost]

Description

MS SQL Management Studio generates script files that are saved with utf-16le encoding. Neovim opens and edits these files with no issue, but the previewer shows extra characters.

Neovim version

NVIM v0.7.0
Build type: RelWithDebInfo
LuaJIT 2.1.0-beta3
Compiled by runneradmin@fv-az320-113

Operating system and version

Windows 10

checkhealth telescope

telescope: require("telescope.health").check()
========================================================================
## Checking for required plugins
  - OK: plenary installed.
  - WARNING: nvim-treesitter not found. 

## Checking external dependencies
  - OK: rg: found ripgrep 12.1.1 (rev 7cb211378a)
  - OK: fd: found fd 8.2.1

## ===== Installed extensions =====

## Telescope Extension: `fzf`
  - INFO: No healthcheck provided

Steps to reproduce

1. nvim -nu minimal.lua
2. cd to a directory with .sql files exported by SSMS as utf-16le
3. :lua require('telescope.builtin.files').find_files() (or any other telescope command)

Expected behavior

Telescope handles utf-16le text correctly when previewing, like Neovim does when the file is opened.

Actual behavior

Minimal config

vim.cmd [[set runtimepath=$VIMRUNTIME]]
vim.cmd [[set packpath=/tmp/nvim/site]]
local package_root = '/tmp/nvim/site/pack'
local install_path = package_root .. '/packer/start/packer.nvim'
local function load_plugins()
  require('packer').startup {
    {
      'wbthomason/packer.nvim',
      {
        'nvim-telescope/telescope.nvim',
        requires = {
          'nvim-lua/plenary.nvim',
          { 'nvim-telescope/telescope-fzf-native.nvim', run = 'make' },
        },
      },
    },
    config = {
      package_root = package_root,
      compile_path = install_path .. '/plugin/packer_compiled.lua',
      display = { non_interactive = true },
    },
  }
end
_G.load_config = function()
  require('telescope').setup()
  require('telescope').load_extension('fzf')
end
if vim.fn.isdirectory(install_path) == 0 then
  print("Installing Telescope and dependencies.")
  vim.fn.system { 'git', 'clone', '--depth=1', 'https://github.com/wbthomason/packer.nvim', install_path }
end
load_plugins()
require('packer').sync()
vim.cmd [[autocmd User PackerComplete ++once echo "Ready!" | lua load_config()]]

[eamonburns]

This also happens for me. Specifically, they were files created using Out-File on Windows Powershell. It seems the default encoding is UTF-16le, but that was changed sometime after Powershell 7 (now the default encoding is UTF-8)

[jamestrew]

The problem is telescope doesn't generate the previews in the same file neovim open files. Reason being is that we want to control exactly what happens when a preview opens for performance reasons (we don't have arbitrary autocommands and such blocking telescope).

I think when vim opens a file, it tries out a list of encodings, stopping at the first one that works. I'm not sure this is an approach telescope has the luxury of going with.

We do have a file_encoding option you can pass, but I'm finding that this has issues of it's own particularly with utf-16le. One that is difficult (maybe impossible?) without hurting performance on large files (needing to iconv the whole file at once).