Support 1337 escape sequence to show inline images

[cristynkells]

Details

• Browser and browser version: all
• OS version: all
• xterm.js version: 2.4.0

Feature request

Is there a way to view images from the command line?

by either clicking on the image (much like the linkify addon) or via a command:

$ cat vacation-beach-pic.png

[Tyriar]

Clicking on the link to open the image is possible via the an API that's experimental at the moment. It's experimental because it's likely to change but here is the current syntax:

// Add a link for a particular regex and handle when it's clicked.
xterm.registerLinkMatcher(/\w\.png/, function (mouseEvent, imageFile) => {
  // imageFile is the text of the link
});

The API should hopefully be finalized in the next couple of versions.

[cristynkells]

Hi Daniel, thank you for your speedy reply! Was wondering if there a way to display html on the fly through a set of control characters? Similar to how DomTerm uses “\e]72;\a” sequences ?

https://lwn.net/Articles/670078/

From: Daniel Imms [email protected] Reply-To: "sourcelair/xterm.js" [email protected] Date: Saturday, March 18, 2017 at 4:47 PM To: "sourcelair/xterm.js" [email protected] Cc: cristynkells [email protected], Author [email protected] Subject: Re: [sourcelair/xterm.js] how to view images in line (#614)

Clicking on the link to open the image is possible via the an API that's experimental at the moment. It's experimental because it's likely to change but here is the current syntax:

// Add a link for a particular regex and handle when it's clicked. xterm.registerLinkMatcher(/\w.png/, function (mouseEvent, imageFile) => {   // imageFile is the text of the link }); — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[Tyriar]

@cristynkells there isn't right now, standard xterm doesn't support anything like that so if we did want to support it it would certainly need to be in an addon.

[DonJayamanne]

@Tyriar

it would certainly need to be in an addon

Is this supported today? Right now I don't see how this can be done even with an add on. Basically I'm trying to investigate what it would take to build something with inline plots (something we already have in other IDE, PTVS, RTVS, Spyder, and the like e.g. https://ipython.org/ipython-doc/3/interactive/qtconsole.html).

[Tyriar]

@DonJayamanne since this issue I've become aware of a standard some terminals have implemented for displaying images. Because of this we could add it to the core library, it's probably a pretty hefty feature though in terms of making sure everything is correct, particularly relating to getting the scroll bar right as right now there's no such thing as variable height rows in the viewport.

[DonJayamanne]

implemented for displaying terminals

Yes, I found iTerm does this.

Thanks for getting back.

[ebraminio]

Hi there. I opened a related issue on VSCode and redirected here. I wonder if supporting iTerm2 image feature is a possibility as it can be used as easy as this,

printf "\n\033]1337;File=;inline=1:`cat a.png | base64`\a\n"`

And I already used it on a project.

[ztmr]

@ebraminio If we are going to draw graphics and possibly some more hypertext, I would really like to be able to specify coordinates where to render it, not just blindly dump it under the current cursor position... check #1176.

[rs]

@ztmr: You can use other termcap commands to move the cursor and display the image wherever you want. That is what I do in jplot to redraw the image in place.

[ztmr]

@rs: Oh, of course, how could I forget about CUP! :o)

[Tyriar]

Opps this was labelled incorrectly. This would be a great feature to get a contribution for, here are some of my thoughts:

• Do we need some pluggable way to support escape sequences?
• Maybe this should live inside an addon?
• Currently every single row is a fixed length, are images to be shown fixed to this grid as well or should the size be different? For example with row height=10px and image height=15px, should the image consume 15px or 20px on the terminal). If the latter we may want to tackle https://github.com/xtermjs/xterm.js/issues/1140 first.

[jerch]

+1 for custom escape sequences. Best place to start with those is to implement OSC and DCS subparser interfaces, where custom subparser can hook into the main parser. Also the main parser still has some escape code flaws and the test library still needs to be completed to cover all the basic sequences.

Btw DEC already had some terminal extensions for graphics and image loading (REGIS and SIXEL). No clue if it would make much sense to implement those, since the browser env of xterm.js has much better capabilities to draw such things.

[jerch]

SIXEL is supported by some programs (gnuplot) and some terminals (mlterm, xterm), there is even libsixel (https://github.com/saitoha/libsixel) for easy conversion. Seems not to be as dead as I thought...

Unsure about the OSC 1337 command, this seems to be supported by iterm2 only.

Edit: Also see jupyter console and https://github.com/liftoff/GateOne, they are both capable to show graphics, not sure though if they hack it around/on top of the terminal on a higher level. This might be a nice addon feature for xterm.js, still in doubt if this should be part of the terminal itself, as it raises many questions from data and state to output representation.

[soccermitchy]

From my point of view, Sixel support would be nice for a monitoring dashboard-style thing. Could easily show gauges and stuff in it - there are even browsers, iirc, that support rendering with it.

[Tyriar]

Here are some screenshot of how image placement/selection works in iterm2, not totally sure how the margins work yet.

[jerch]

I think SIXEL is the way to go for images, also made some progress with an early SIXEL parser (https://github.com/jerch/node-sixel). Once we are settled with the new buffer layout and marker/decorator rework this should be straight forward to integrate.

[PerBothner]

While sixel is a "standard" (and I think it is desirable to support it), it is a rather obsolete standard. It is not well suited to today's hardware, software, or workflow. Very few people have heard of it, and very few programs produce or consume it. It is extremely inefficient for photographic images (does not support many colors well), and it is constrained to images whose height is a multiple of 6.

The concept (if not necessarily the specifics) of the 1337 escape sequence is a lot easier for most people (and programs) to work with: a base64-encoding of standard current image-file formats: png, gif, jpeg, and optionally svg. I think that is the way to go. The 1337 syntax is a bit clunky (the inline option defaults to 0 - huh?), but it works. Is there some other escape sequence with similar functionality used by other terminals?

(While DomTerm doesn't support the 1337 escape sequence, it supports the same idea, except you have to wrap the data in an <img> or <svg> html element. Adding support for the 1337 escape sequence should be trivial.)

[jerch]

@PerBothner Imho the ticky part is #1901, as soon as we solved the question how to deal with the grid and layering for "foreign boxes" I see no reason why we should not support image formats that are natively supported by the browser env. But for that I'd love to see a broader discussion (on @egmontkob's platform?) and maybe rethink the sequence. For sixel the sequence is already there for 30ys, still the correct handling of the representation state is wonky (again #1901).

[PerBothner]

I fully agree with @jerch's most recent comment.

[PerBothner]

I created an issue on the freedesktop.org terminal-wg to try to standardize "simple image display". I hope we can get agreement on a suitable semi-portable escape sequence. Possibly the 1337 escape sequence (though it has some issues); possibly something else.

[sdegutis]

I think this can be broken down into two issues:

1. Handling of arbitrary escape code sequences, either internally or externally (via API). This is a useful feature for more than just images, and is orthogonal to the concept of what is shown or even how it's shown. It just needs a function callback API.

2. Hosting of foreign DOM content, in this case an image, but other content has the same properties: a DOM element with an x,y,w,h relative to a sub-grid of cells on the terminal's character grid.

Both of them would help me a lot, but in different ways. It might be worth creating two new issues for each.

[sdegutis]

• Currently every single row is a fixed length, are images to be shown fixed to this grid as well or should the size be different? For example with row height=10px and image height=15px, should the image consume 15px or 20px on the terminal). If the latter we may want to tackle #1140 first.

I just assumed the image would still be 15px and the extra 5px would be empty, so that the image would be "overlayed" over the rows. Considering cell width/height is so small, I don't see any drawbacks to the slight wasted pixel space of this solution.

[Tyriar]

Handling of arbitrary escape code sequences, either internally or externally (via API). This is a useful feature for more than just images, and is orthogonal to the concept of what is shown or even how it's shown. It just needs a function callback API.

Already done:

https://github.com/xtermjs/xterm.js/blob/bd0d267d972a1aab4777cddecf9bf8196d7aa44f/typings/xterm.d.ts#L1060-L1127

My comment is probably a little out of date, @jerch has been the main one looking at images with this, https://github.com/xtermjs/xterm.js/pull/2503 and https://gitlab.freedesktop.org/terminal-wg/specifications/issues/12

[sdegutis]

Ah right, yeah I even use that API. Whoops, should have remembered. Thanks.

[tracker1]

I do think it would be cool to support sixel as well as 1337 sequence... I do think that whatever is displayed should be potentially resized if too wide, and the space taken padded to a full line height.

May already be possible with the extension, not sure if custom handlers can hook into a given set of columns/rows and the height-width of either to custom render several rows?

Came into this after discovering sixel formats and noticing that it isn't available in the terminals I use most (one of which is vs code).

[jerch]

@tracker1 See #2503 for the current development state.

[trbot86]

I really hope this is still in active development. (Seems like a PR is stalled waiting for merge into xterm master?)

As a scientist who writes code to run experiments that produce graphs, the ability to visualize these graphs (in VSCode's xterm) without opening external software would be huge.

[jerch]

I really hope this is still in active development. (Seems like a PR is stalled waiting for merge into xterm master?)

Well, originally I intended to add the SIXEL part as sync code (which is almost feature complete), but the image decoding puts quite some pressure on the main thread, which we cannot simply ignore in bigger embedding apps like vscode as it would lead to bad lag spikes. The addon is basically waiting for #3222 to get finished, with that PR we can offload the hard work to a worker thread.

As a scientist who writes code to run experiments that produce graphs, the ability to visualize these graphs (in VSCode's xterm) without opening external software would be huge.

Yes scientific REPLs with direct graphics dumping caps will benefit alot from this.

[trbot86]

That makes perfect sense, and I'm really glad to hear it. Looks like you're down to just MacOS specific test failures, but if the skillset of a CS concurrency/multicore researcher would be useful, feel free to let me know.

[jerch]

Well #3222 will take abit more time, it changes several crucial aspects of the parser handlers which needs proper testing. Once it is there the adoption needs in #2503 should be small.

if the skillset of a CS concurrency/multicore researcher would be useful, feel free to let me know.

For #2503 I wonder, how much functionality could/should go to the workers in the end. I really would like to use OffscreenCanvas, but thats still unlikely to happen, as the API is far from stable across browser engines. If you have more insights/experience regarding this, fee free to discuss the internal addon shaping over there.