control characters unaccounted for

[deusaquilus]

I have some log files that have CSI characters which are showing up in my logs. Perhaps parsing should be added to remove these.

[ESC]SC[0Ktravis_fold:end:services
[ESC][0Ktravis_fold:start:postgresql
[ESC][0Ktravis_time:start:121e4528
[ESC][0K$ travis_setup_postgresql 9.6
Starting PostgreSQL v9.6[ESC][0K
 * Stopping PostgreSQL 9.2 database server       [ESC][80G 
[ESC][74G[ OK ]
 * Stopping PostgreSQL 9.3 database server       [ESC][80G 
[ESC][74G[ OK ]
 * Stopping PostgreSQL 9.4 database server       [ESC][80G 
[ESC][74G[ OK ]
 * Stopping PostgreSQL 9.5 database server       [ESC][80G 
[ESC][74G[ OK ]

I tried adding the following into ansi.sublime-settings but it did not work:

      {"scope": "_bold", "code": "\\x1b\\[0K", "color": "#ffffff"},
      {"scope": "_bold", "code": "\\x1b\\[K", "color": "#ffffff"},

[zerotag]

I'm sorry if this is not how you demonstrate this, but I have no experience in ANSI characters. This output (from docker-compose) has some characters which I didn't find anywhere, including on this issue, so I thought I would add to it.

Starting webserver ... 
Starting php-fpm   ... 

<0x1b>[2A<0x1b>[2K
Starting webserver ... done
<0x1b>[2B<0x1b>[1A<0x1b>[2K
Starting php-fpm   ... done
<0x1b>[1B

All I know is the <0x1b>[ is a CSI, are these the same as the issue?

[jfcherng]

The general CSI sequences is unsupported. Only 3/4-bit color are supported from what I can guess from the codes.

[DolphinDream]

The general CSI sequences is unsupported. Only 3/4-bit color are supported from what I can guess from the codes.

Not supported? then 1. why not? 2. Any workaround? (e.g. can the <0x1b> markers be replaced by some other supported ones?

[jfcherng]

Not supported? then 1. why not?

Fact: To color a region, you have to provide its scope for ST. There is no other way to do it currently.

For currently 4-bit support, we have at least (2^4)^2 = 256 scopes (actually is 24*18 = 432, see your user settings and the auto generated "Packages\User\ANSIescape\ansi.sublime-color-scheme"). (2^4) is bit depth. ^2 is the combinations of foreground and background. And technically 256*2 (actually 432*2) regexes replacements to be run. The performance is still acceptable while opening a large log file.

To support 8-bit, expected scopes count is at least (2^8)^2 = 65536 scopes and 65536*2 = 131072 regexes to be run. For 24-bit, I don't even think it's acceptable obviously. Even the 8-bit, the number is still unacceptable.

The only thing you can change is the number of regexes if you can figure out a new algorithm or plugin architecture, but the number of scopes is inevitable currently. I think the term "support" is not "cleaning anything not supported", right?

Another thought is to pick a similar color to "downgrade" 8-bit/24-bit colors to 4-bit ones. But I am not going to implement it since the current plugin works for me. Welcome to send a Pull Request though.

2. Any workaround? (e.g. can the <0x1b> markers be replaced by some other supported ones?

No. \x1b is hard-coded everywhere in the source codes.

[DolphinDream]

1. Any workaround? (e.g. can the <0x1b> markers be replaced by some other supported ones?

No. \x1b is hard-coded everywhere in the source codes.

I meant replace \x1b in the text file with one that the ST ansi filter does support.

[jfcherng]

I meant replace \x1b in the text file with one that the ST ansi filter does support.

\x1b is the only one that ansi supports. [ESC], \x1b and <0x1b> are the same thing. <0x1b>[2A is not working because it's not 3/4-bits.

Another thought is to pick a similar color to "downgrade" 8-bit/24-bit colors to 4-bit ones.

This can be a good solution actually if someone can build a 8bit-to-4bit map. My lame though is writing a script to calculate the "distance" among all 8-bit colors and all 4-bit ones and use "closest" ones as approximations.