Client-to-client formatting tag

[lopcode]

Inline formatting, as it stands, has a number of downsides. Some include:

• Clients that don't wish to display any formatting still have to support it, and strip it out manually
• Usage is 'documented' here but there is no formal spec and client implementations seem to vary a lot
• Only 16 colours are supported

I'm skeptical of inline formatting tools such as markdown, because clients have to understand them completely to be able to strip it. Users should also be able to type things like *statement* without it getting formatted, and adding an escaping burden to existing methods of expression feels odd.

I see an implementation as naturally being similar to "custom emoji" - a @formatting tag with a json array of formatting types and replacement indices and lengths, something like: @formatting=[{"type":"bold","idx":[[7,5]]}] which would produce "hello, world!".

We'd likely want types to include (as an idea, not limited to):

• Colour
• Italics
• Bold
• Underline
• Strikethrough

I'd like to start a conversation on getting some more formal client-to-client formatting tags specced up, and potentially seek a working group member to sponsor it. Also interested in people's considerations and concerns.

[lopcode]

An immediate concern I see here is ordering. Perhaps there would need to be some non-normative rules about the order in which to apply formatting and how they might overlap?

[MrBenC]

A few comments:

1. Is it really better to "front load" the formatting? While I understand you don't like inline formatting and you presented a good reason, who's to say it couldn't be stripped out at the IRCd level? Color stripping modes work well enough. Although that adds the additional burden of possibly using more non-printable characters, which could bring it's own set of problems. So, I could see why you may prefer to "front load" the formatting into a tag argument. However, this is by far the least of my concerns with this proposal (which isn't a bad idea, by the way).

2. For better or worse, I don't see the "old way" ending anytime soon and even if this becomes more concrete, the old way will probably continue to be used for awhile. Should IRCds "convert" and "translate" the old way into the new format, and vice versa, or leave the text as is and use both?

Pros to conversion/translation:

• The client would only have to accommodate one type on a given IRC connection.
• It may help speed up adoption of the new way if you don't have to wait for every client and bot to implement it. It may also prevent and reduce instances of more conservative IRCds and client/bot authors from taking a "wait and see" approach so this actually gets used.
• No concerns or questions about "precedence" (more on this below under the Cons of using both).

Cons to conversion/translation:

• This may be more difficult for IRCds to implement.
• Clients would have to support both types for the foreseeable, indefinite future but realistically, they will have to support it anyway regardless of the approach taken.

Pros to using both the old way and the new (no translation):

• It would be easier to implement from the IRCd point of view, but would be a nightmare for client authors and may result in inconsistencies.

Cons of using both:

• Clients would have to support both on all IRC connections. This also brings up a question regarding "doubling" and which would take precedence. Using the example posted above, let's also add the traditional "bold" characters around "or." If that occurs, does the bold formatting stop after the 'r' or after the 'd?' This is an implementation detail and problem that I believe would be best avoided by using conversion. If left up to individual client authors, this could lead to a lot of inconsistency regarding expectations of how other clients will see the message.

3. When you listed types of formatting to "include" I'm assuming you meant "including but not necessarily limited to." There needs to be room for potentially adding additional types. Just as examples that don't need to be in a first draft, this could also be formatting types such as double-strikethrough, double-underline, or "code" (I'm thinking monospace though GUI clients could choose to add a bubble around it). There's probably more but I think you get the idea.

A registry of types should be established, and clients should probably ignore and not render types that aren't implemented in that client. This would make the new spec extensible.

For colors I'd probably suggest a RGB hex code. As this is protocol level, I don't see a point in adding shortcuts like "red," "blue," "black," or "white."

As for the question of conversion/translation, I believe the better way is for IRCds should probably convert the old way into new way, and to the extent possible, the new way into the old, skipping the new formatting that obviously don't exist in the old way.

This is just my two cents on this. I think this has the potential to be pretty good, but some of the problems I mentioned above, and possibly others I may not have thought of, would need to be addressed first.

Ben

[lopcode]

Thanks for the feedback!

Front loading the formatting facilitates additions much better. Taking the examples of double underline and double strikethrough, if that were added as an inline formatting code, I doubt that clients would display anything other than an odd character as it stands.

I'm also not convinced that "compatibility considerations" like translation should block the development of a spec. Those discussions are important for sure, though.

I also conflated CTCP and inline formatting in the original post, so removed CTCP references.

If there's appetite for this then I'll take a look at solidifying the tag specifics, more thinking about general concerns for the moment.

[lopcode]

More thoughts: I agree about a registry, and yes I don't think the types I've listed aren't the limit of what could be added. I do think we'd need to come up with a set of sensible types for a "first pass".

Good point about the colours, we'd likely need an optional "parameter" field to store some value. Hex is probably fine but specifics of that aren't too important.

[DanielOaks]

With regards to only allowing 16 colours, I've added this hex colours section which details an existing method for conveying hex-based colours that's already present in some clients.

The "compatibility considerations" concern, while ordinarily able to be dismissed, is an issue here because literally everything supports inline codes for things like colour and bold. Servers will need to continue to do this translation forever, which I think instantly makes this solution less desirable than just doing something like standardising the current inline formatting characters and working forward from there.

It should be kept in mind that the method you use to let your client know to format some text (i.e. *this* for bold, _this_ for underline), and how it's conveyed on the wire are separate concerns. They're often conflated to push this sort of formatting, but you can do it just fine with current inline codes as well.

[DanielOaks]

I should add to the above comment – either servers will need to do translation between the methods forever, or clients will need to support both methods forever. Either option increases the complexity that the server/client need to put up with on a day-to-day basis.

As well, if clients are expected to understand both then clients will need to send both the inline formatting code as well as the frontloaded one to make sure that the client on the other end has the best chance of understanding it. If we get the servers to do translation then either they'll need to know whether the end client supports tagged colour codes using a cap or translate messages in the same way so both older and newer clients can interpret them. This seems crufty and ugly imo.

[lopcode]

The trouble is that I think there are fundamental problems with formatting strings inline to begin with, and I'd definitely be hesitant to extend them in any way right now, and expect very little in the way of sanity when clients get something inline that they're not expecting.

I agree about the difference wire format and client input. Markdown for input sounds desirable but markdown over the wire muddies the message you're trying to send, and as far as I know many IRC users prefer no formatting at all (which would have to get stripped out).

[lopcode]

I also don't agree about the "crufty" comment - is this not the point of advertising CAPs? Assuming the server translated it, if a client supports the "formatting" tag, then the server turns inline codes in to formatting json embedded in a tag. If not, don't mess with it.

[lopcode]

Another thought: ending up with old clients (read: mIRC) having no formatting capability sounds totally reasonable. I don't think we should expect both to be supported indefinitely, and there would definitely be an ircv3 push to use sane formatting metadata over inline codes.

[jwheare]

Re: compatibility. Why not have clients send both formatting tags AND old style codes for the styles that are supported, and then require them to strip out the old style codes on the way in if they see a formatting tag.

[DanielOaks]

I like the idea of keeping formatting out-of-band. I dislike the duplication of having to represent or parse the same thing in multiple ways, and think that this effort could be better spent getting clients to simply formalise a set of inline formatting characters and the parsing of those characters instead.

I'll probably leave my contributions to this idea here for now, since my inherent view on this is to not like it until I see a more concrete spec detailing it and the upsides/downsides.

[lopcode]

I agree about writing a spec, and between here and IRC I think I have enough considerations. I'll get on it in the near future so people can critique a spec rather than the idea.

[luxaritas]

I'll throw in my two (or ten, or twenty) cents.

Here are some possible formatting options that I wrote up a while back (via https://github.com/EteRNAgame/HTML-Chat/issues/5). Not all of these are necessarily critical, but in past conversations (including my usage of Slack and Discord) I've found them to be helpful:

• Italics
• Bold
• Underline
• Strikethrough
• Blockquotes
• Links (when the text is different than the URL, security considerations involved of course)
• Code/Preformatted (The latter requiring newlines)
• Newlines (I think this would be a better way to implement it than https://github.com/ircv3/ircv3-specifications/issues/208, since it's generally a formatting thing, ~~and I'm not sure if a batch approach would provide well for empty lines~~ [this would just be a blank line - thanks @jwheare])
• Bullets (relying on newline)
• Ordered/Numbered lists (relying on newline)
• Images (probably covered by https://github.com/ircv3/ircv3-specifications/pull/302, or at any rate out of scope since it's not text formatting)

One significant benefit I see from front-loading formatting is that vendor prefixes could be used, so any new or custom tags could be easily distinguished. Inline formatting has little concept of intrinsic meaning, ie 0x02 is just a number and *text* could mean either bold or italics (though at least it somewhat has some visual indication of meaning), whereas bold, draft/bold, or myserver/bold have very specific meanings. This provides little room for differing interpretations/clashes between implementations. For a distributed ecosystem like IRC, I think this approach of keeping different content distinct (ie text is just text, formatting specified separately with the ability to prefix) is extremely important.

Some downfalls of control characters:

• Using control characters also means that there is a limit to the number of possible formatting options. I doubt we'd run into that ceiling (afaik there's a number of control characters not really used for anything), but the ceiling is there none the less.
• Adding new control characters would also show random characters in clients just like dropping support might, unless it is server mediated (no longer a true client tag), or a bunch of control characters are reserved up front (in which case legacy clients would still show the boxes, but there wouldn't be an issue with new formatting characters used within this range since clients could strip the ones they don't understand).
• IRC is (from what I remember reading) designed to be human readable. Control characters are not human readable (if they are even visible they must be interpreted), connecting back to the point about lacking an intrinsic meaning.

As far as compatibility is concerned, I think it would probably be wise to maintain compatibility with the legacy formatting codes at least starting out, but I feel that the benefits of a more robust formatting mechanism would significantly outweigh the potential loss of simply dropping support for formatting in legacy clients after a while (in the couple of communities I've been in, I've rarely seen formatting being used, though someone here who's been involved more can overrule me).

As far as how to achieve backwards compatibility, I see two options (as @MrBenC mentioned):

1. The semi-more robust way, server-mediated. There would be a cap, where if it's requested, the server would translate all formatting to the new mechanism, and strip out the inline characters. If not requested, the server might initially translate all formatting (if possible) to the legacy codes in order to support and assumed legacy client. If treated as a temporary compatibility mechanism, it could deteriorate somewhat gradually/gracefully:

   • If a server stops providing the cap before clients stop requesting it, no translation would occur. Legacy clients would have no/crippled formatting capability. New clients would either:
     • No longer understand the codes, and therefore see some extra random boxes if someone with a legacy client tries to use formatting
     • A client has a two-stage mechanism, now opting to maintain support for incoming formatting until they feel it is no longer necessary
     • A client has a two-stage mechanism, now opting to strip the unsupported tags until they feel it is no longer necessary.
   • If client stops requesting the cap before servers stop providing it, the client would receive BOTH the new formatting and the legacy formatting from a server that does translation (the server would NOT strip out formatting client tags, they would simply provide the inline codes in addition). New clients would act the same way as if the server stopped first, while legacy clients would of course still receive and use the legacy characters.

2. The semi-more efficient way, duplication in clients (requiring only client involvement). Supporting clients would write both the tag and inline. If they receive a message with one mechanism or the other, they simply parse through it per normal. When both are received, inline characters should be stripped out (including only non-character information in the tags would me somewhat messy and would unnecessarily break support for clients who lack or have completely dropped support for inline codes). If treated as a temporary mechanism, a client would no longer transmit inline characters. Clients would act the same way as described in the server-mediated solution where the server stops support first.

With option one, you remove the duplication entirely, and legacy clients are guaranteed support as long as the server maintains support. With option two, you remove the need for server involvement entirely and simplify the spec. Frankly, don't see a significant benefit involving the server, especially imagining that most new clients would adopt a two-stage approach anyways, so they would still be potentially parsing through the formatting anyways even though over time older clients start losing formatting capability (which as I described before, doesn't particularly bother me). Similarly, if a client really doesn't want to support both, just not supporting inline codes doesn't feel like a huge drawback.

TL;DR, I'd argue have a mechanism to allow for clients to catch up, but I wouldn't cry if sooner or later old formatting just stops working. If implementations want to support it forever, great, but I don't find it necessary.

[dpyro]

Formatting

The thing about markdown is that it doesn't have to be stripped out to remain legible.

Does IRC really need extensible formatting? Most text messaging services that can technically support formatting (iMessage, WhatsApp, etc.) don't do it--likely for legibility reasons. IRC is different but it is very easy to destroy accessibility, legibility, and portability with excessive decoration.

Colors

IRC color formatting separates the foreground and background colors which create readability and accessibility issues. If the author creates formatted text using a client with a white background and the text is read on a client a black background, the text might not even be visible! This is even worse if the user is colorblind. I would suggest allowing a standard hex color code (like :#FF0000:) and have the client select the appropriate foreground and background color and/or ignoring it depending on the user's preferences such as here.

[M2Ys4U]

I see an implementation as naturally being similar to "custom emoji" - a @formatting tag with a json array of formatting types and replacement indices and lengths, something like: @formatting=[{"type":"bold","idx":[[7,5]]}] which would produce "hello, world!".

What are you indexing here, bytes?

What happens if you start or end formatting in the middle of a multi-byte sequence?

What happens if the receiving client (or server, or bouncer) performs normalisation on the string?

[jwheare]

Yeah it's bytes. If a sending client messes up the indexing and puts a boundary within a multi-byte sequence then you'd get garbage, but that's a non-issue, the client is buggy.

I'm not aware of any server or bouncer that does multibyte normalisation, is that something you've seen or a straw man? If the receiving client does it then it's surely capable of adjusting the indices appropriately.

What's perhaps a more likely scenario is if a message is sent with mirc style formatting codes as well as indexed formatting tags, and the server/channel strips the mirc codes.

So, a valid concern. Some potential mitigations that come to mind (of varying degrees of madness):

• Duplicate the full message text in a message tag that servers won't touch and that clients can use for index-based formatting. Effectively rendering +c etc useless, and gives control back to clients.
• Define a tag that servers can add to a message if they alter the text in any way. Not really enforceable.
• Include a checksum/hash in the formatting tag json to make sure the message is unaltered. Clients would lose all formatting in the case of a mismatch, but at least be left with a useable message.