:term should forward unknown sequences to host

[chemzqm]

iTerm2 version 3 beta support show image inside terminal with command like imgcat, but the image could not be seen with that command inside terminal of neovim. That would also be cool if neovim support show image in buffer with an api .

[justinmk]

Some people are already doing this, see https://github.com/neovim/neovim/issues/4056

[chemzqm]

@justinmk I'm using latest neovim, but I can't get any image ouput with command imgcat inside the terminal of neovim

[justinmk]

It looks like imgcat is using a custom DCS 1337. I am guessing that is not a code that is recognized by libvterm (@leonerd can you confirm), but on the other hand we should forward it to iTerm2, but maybe we aren't?

~~Clearly there is some seequence that is recognized by libvterm to display images as demonstrated by #4056. imgcat could be modified to emit those codes instead?~~

[DarkDefender]

@justinmk IIRC the method in #4056 does not use terminal codes. It instead calls a image display program that displays the image at the requested location.

That's probably why it doesn't output the image at the correct location when inside neovim. Because it assumes that it is using the full terminal window. So it will not work with splits and so on.

I guess that the iTerm2 codes will eliminate this problem because the terminal will then know where to put the image depending on where it reads the DCS 1337 code

[liubog2008]

imgcat can work well in vim but in neovim it prints chars @justinmk

[liubog2008]

~It looks that vterm_state_set_unrecognised_fallbacks is not called though libvterm is updated?~

[kdheepak]

Is there any update to this issue? It would be nice to have this feature.

I understand that this is related to forwarding unknown sequences to iTerm2. However, my understanding is that sixel support in libvterm is also an option for displaying images, right? I'm not familiar with the internals/architecture, but is it still the case that neovim's internal screen is composed of unicode characters of fixed size and hence will never be able to display images [1]? Is this unlikely to change?

With iterm2 3.0.0 beta it is now possible to display sixel images as well [2]. In the screenshot above on the left I'm able to display images using imgcat (iTerm2 escape codes) and using cat (SIXEL).

One valid use case for this feature is that these can be powerful tools for visualization / exploring data in scientific computing from the terminal interactively. It appears that tmux is not planning to support this [3], so neovim's terminal would make a highly appealing option to interactively run code and visualize data, all from the command line.

It'd be really cool to have one or both methods supported in neovim. Thoughts?

[1] https://github.com/neovim/neovim/issues/4500 [2] https://gitlab.com/gnachman/iterm2/issues/3543#note_157693474 [3] https://github.com/tmux/tmux/issues/1613

[bfredl]

but is it still the case that neovim's internal screen is composed of unicode characters of fixed size and hence will never be able to display images.

"never" is the wrong word, the question is if it is worth the effort or not. Does the sixel state of a single cell fit into 28 ASCII chars, or something like that? Then we could add support without changing the screen representation fundamentally.

[kdheepak]

I'm not sure what you mean by "sixel state" or "a single cell"? I'm also just learning about all of this, so take it for what it is worth. The following information is based on the vt100 reference manual [1].

A single "sixel" encodes information about a six pixel high * one pixel wide column. So each sixel in binary is encoded as a single 6-bit number. This is converted to ascii such that every sixel is within the printable character range of the ASCII character set.

Wouldn't just passing on sixel information to iTerm2 do the trick?

[1] https://www.vt100.net/docs/vt3xx-gp/chapter14.html

[kdheepak]

Other terminals also appear to be using their own format for displaying images. Kitty for example also has similar escape sequences to iTerm2: https://sw.kovidgoyal.net/kitty/graphics-protocol.html

[bfredl]

A single "sixel" encodes information about a six pixel high * one pixel wide column. So each sixel in binary is encoded as a single 6-bit number. This is converted to ascii such that every sixel is within the printable character range of the ASCII character set.

I understood so much. But how many sixels wide is a text cell defined to be? how many "sixel lines" per one text line? Or if this is not defined and varies among terminals/configs, can it be detected somehow?

Wouldn't just passing on sixel information to iTerm2 do the trick?

To keep the vterm in a consistent state we need at least to know how large the sixel area is. But I think users expect if they scroll/rezise the terminal, or display it in a different window, the graphics will be retained, and thus we would like to store the state in our internal screen buffers.

[nall]

I’ll comment since #11282 was marked a dupe of this. I am not interested in images so much as other features iterm2 exposes via these codes.

Please consider passing these codes through even if there are questions about the image support since that’s only one of the ways they are used.

[bradparks]

Although dithered, you can use asciimatics to view a dithered image in a neovim terminal or tmux:

https://stackoverflow.com/a/59223021/26510

[ElPiloto]

Any progress or decisions made on this? Adding my intended use case just in case that helps tip the scale:

• images (specifically graphs/plots) in neovim's terminal would go a long way to using neovim as a python IDE for machine learning.

[yatli]

Also curious about this. Could help to implement GUI elements for neovim.

[tejasvi]

It is possible to display images by disabling embedded libterm during build with -DUSE_BUNDLED_LIBVTERM=OFF.

[bfredl]

it requires this patch tho https://github.com/arakiken/libvterm/commit/2500d7ad1c7f74162d70e3089fc95934044b8759

@leonerd does it look like something that could be included upstream?

[sassanh]

It is not working with iTerm2:

Without this version of libvterm it wouldn't even show those question marks, but now that I installed the sixel branch of arakiken's version of libvterm it shows these question marks

[bluss]

Image support would be awesome. It seems the issue title is a bit of a misstatement, thought: just forwarding unknown sequences to host would not make for great support, then scrollback, windows and views etc will have no chance of working correctly. It looks like libvterm and neovim need to support the image protocols for terminals explicitly (and unofficial libvterm patches for this are posted in earlier comments in this issue). Issue #4500 has a more specific target and is sixel-specific, but any image protocol would be a good start.

[starptr]

Has arakiken's patch been merged into upstream libvterm yet? There is a build of neovim at nat-chan/Neovim-DRCS-SIXEL-dtach that uses that patch and shows sixels working

[leonerd]

What's the overall status here then? The original subject is about forwarding unrecognised sequences - I'm fairly sure libvterm already has facility to do that now. Is there something still missing in it? If so let me know and I can add it.

If not, and that ability is there now perhaps we can close this issue, in favour of a different one discussing particular things to be handled..?

[sassanh]

Thanks for the update! Are these facilities documented somewhere? How can I learn about it?

[leonerd]

Are these facilities documented somewhere? How can I learn about it?

The VTermStateFallbacks structure in include/vterm.h

[justinmk]

The typical use case mentioned here was imgcat. Assuming that works now, we can close this. Thanks @leonerd

[jaekyeom]

@justinmk I tried imgcat in terminal buffers with both v0.7.2 and the nightly build, but I think it still doesn't work.

[sassanh]

I registered this osc callback:

static int oscCallback(const char *c, size_t s, void *a)
{
  int fd;
  fd = open("/tmp/log", O_WRONLY | O_CREAT | O_APPEND, 0644);

  write(fd, "\e]", 2);
  write(fd, c, s);
  write(fd, "\7", 1);

  close(fd);
  return 1;
}

VTermParserCallbacks callbacks = {
  .osc = oscCallback,
};
...
  vterm_state_set_unrecognised_fallbacks(state, &callbacks, NULL);

then I was able to do these in Neovim terminal and see the results in another normal iTerm2 terminal by running tail -f /tmp/log in it:

1. Cursor shape changes
2. Changing iTerm 2 title

Everything worked as expected when I cat /tmp/log except imgcat, oscCallback is not triggered when imgcat runs. Seems like libvterm is doing its job. But when instead of fd I wrote to stdout, things got ugly and part of the prompt rendered out of the terminal window in another buffer. I guess in Neovim realm, communicating with the underlying terminal Is not as easy as directly writing to stdout.

We may be able to get rid of the virtual cursor and let the running application (bash, fish, node, etc) directly set the cursor shape by communicating with the underlying terminal emulator.

Also it may break some people's workflow, commands that were ignored in the past, now reach the underlying terminal emulator, there may be resize commands, setting title commands, setting working directory commands, etc so it will be a breaking change (someone may have an iTerm2 script that does something based on the title of the terminal and doesn't expect Neovim's embedded terminal change that, etc) so we may want to start with a feature flag that is disabled by default.

Note for @leonerd: imgcat did not work as none of the fallback callbacks were called after running imgcat, I'm going to read the code to find out what are the conditions for calling osc callback and which condition is not satisfied with imgcat output. Please let me know if you have any suggestions.

[leonerd]

I have no idea what imgcat is. Is it supposed to be outputting OSCs? I'll need some more reference to understand all this.

[sassanh]

Sure, this is the script: https://iterm2.com/utilities/imgcat Yes it is supposed to be outputting OSCs, I attach a sample output here

[leonerd]

I don't know whether https://github.com/eddieantonio/imgcat is the same one, but for that I notice that it only outputs OSCs when talking to a real iTerm2. When talking to some other terminal it just emits a grid of Unicode half-linedrawing characters, with foreground + background colours set, allowing it to paint pixels in colours regardless of what the terminal is.

How did you capture that sample output? Is it captured from within the shell inside nvim's :term?

[sassanh]

I ran this command in a terminal opened by :term in Neovim:

imgcat "<IMAGE_PATH>" > /tmp/sample_imgcat_output.osc

[leonerd]

Ah wait a second - the oscCallback function you defined earlier isn't even compatible with the required signature. The fallbacks structure requires:

  int (*osc)(int command, VTermStringFragment frag, void *user);

following the VTermStringFragment API. Perhaps that's the problem here?