Major Version 3.0.0

[WebCoder49]

If I've notified you, or you've just come across this, I'd love ❤️ it if you could give feedback (here or via email to [email protected]). However, feel free to take your time and don't feel you have to; I know you may be busy.

I'm thinking of rewriting most of the core of code-input and its architecture. The reasons are: architectural flaws, better modularity, and performance, all of which desire breaking changes:

Features

The current architecture that has existed since the start, where a code-input custom element that in reality contains a textarea is interacted with as if it were a textarea itself, requires quite a lot of glue code which can contain bugs, causes CSS problems, and isn't as easy to combine with other textarea-modifying code as it could be. Following code-input's existing focus on both using and mirroring the API of the native textarea for the majority of its functionality, a breaking-change architecture could lead to less core code being needed, and even better textarea support, by exposing a real textarea element to JavaScript and HTML then generating highlighted elements etc. based on interactions with it (e.g. a textarea class="code-input" element for usage, instead of code-input element). Plugin architecture is also problematic; more flexibility to add and remove plugins real-time would be useful (as suggested in #122).

Better modularity (while remaining as lightweight as possible and probably even more lightweight!) likely includes writing the library as ES modules (but other than that vanilla JavaScript) by default, separating the Plugin definition into another file so it doesn't need to be imported, separating templates, and bundling only the necessary features for a setup to a CDN. Encapsulation should be implemented and a public API exposed more strictly than just TypeScript definitions, for better classification of breaking changes. It also involves deleting deprecated, little-used functionality that is present for back-compatibility but is now largely causing bloat.

Performance involves only partially highlighting on input to reduce time complexity, as in #70. This isn't a breaking change but is quite a large change and to avoid bloat should be loaded as a separate module, suggesting better modularity which does require a breaking change.

I will also try to standardise the dialect of JavaScript being used for more portability across browsers, and not just use any JS/CSS feature.

I'm also considering releasing the new version under the MPL-2.0 license which allows plugins under any license, and use of the code in a project under any license (both like MIT) but preserves the availability of the library itself as open source by requiring edited versions of existing plugins and the core library (only those files) to be distributed with source code under an open source license. Do you have feedback on this?

The result should be a lighter, faster, and more flexible code-input.js library.

Maintenance

Security fixes should still be backported to version 2, but for other updates it shouldn't be too difficult to upgrade plugins to the new API, and usage in HTML is only one element so not much code to change. JavaScript usage shouldn't change much since the textarea interface is still a focus.

As always, before version 3, solving people's immediate problems in version 2 will be prioritised wherever practical.

Feedback

What do you all think? I'm unlikely to have time to implement this in code very soon, but feedback and/or patches to begin this are very much welcome, and in a future holiday I will have time. I'm working on a project right now that both wants this update at some point and is teaching me about concepts that would be helpful in the update.

Notifying those who have contributed code/docs and thus interacted with the architecture: @vanowm, @MattiasBuelens, @paulrosen, @andrelsm, @mitchcapper, @eyaler, @keithjgrant, @RobertBlackhart

Notifying some people relevant to: architecture @nikita51bot (from #122); modularity @paulrouget, @Bowrna; performance @andrelsm (yes, I know you said it probably isn't worth it but with a more modular architecture it will be possible and is quite a large hurdle to adoption; and with e.g. line-based highlighting could likely be quite versatile across all (regex-based) highlighters), @KaiWilke

Everyone else is also welcome to comment.

[WebCoder49]

Examples of specific issues that strictly require a new major version can be found here.

[eyaler]

@WebCoder49 Thanks for doing all this!

Following this issue, and the discussion on working closer to the textarea "metal", I got thinking whether we could just use textarea. In some projects I am working on, I would like to add a syntax highlighting feature. But afaiac, this is just a visual feature, and I am not happy to restructure my code around this (and this is before considering the bidirectional risks). I don't want to add this large "mirror" just to have different colors for differnt words. There is a textarea, and I am willing to let you add some CSS and JS to color it nicely.

So if we are already breaking stuff... here is a crude demo of a single element textarea with syntax highlighting. I actually wanted to do it with shaders, but this just uses background-image: linear-gradient with background-clip: text to add color, while accounting for the latter's various quirks in both Chromium and Firefox. I didn't test on Safari yet, and it currently only supports monospace, and does not support variable width tabs nor wrapping. I didn't look into performance at all. It uses highlight.js for parsing. Look ma, no spans: https://codepen.io/eyaler/pen/NPGOaJb

What do you think? Would you consider this a viable approach at least for some of the simpler use cases?

[WebCoder49]

@eyaler Thanks for this! I'll respond in detail when I have time. By the way, please could you give your feedback on this paragraph from above?:

I'm also considering releasing the new version under the MPL-2.0 license which allows plugins under any license, and use of the code in a project under any license (both like MIT) but preserves the availability of the library itself as open source by requiring edited versions of existing plugins and the core library (only those files) to be distributed with source code under an open source license. Do you have feedback on this?

[MattiasBuelens]

While we're throwing ideas around, have you considered using the CSS Custom Highlight API instead? Bramus made a demo on how it could be used to do syntax highlighting, and apparently browser support is quite good nowadays. You can't use a <textarea> yet though, so you'd have to switch to a <pre contenteditable> instead... And maybe use attachInternals() to have it participate in a surrounding <form>? 🤔

[WebCoder49]

Thanks for these ideas which have given me a lot of inspiration! Here's my evaluation, ready for feedback when/if you have time:

@eyaler wrote:

Following this issue, and the discussion on working closer to the textarea "metal", I got thinking whether we could just use textarea. In some projects I am working on, I would like to add a syntax highlighting feature.

But afaiac, this is just a visual feature, and I am not happy to restructure my code around this (and this is before considering the bidirectional risks).

My current plan of v3's architecture aims to prevent you from ever having to interact with a non-textarea API (AKA restructure your code) except when it's absolutely necessary. A page that works without highlighting should work with highlighting just by adding a JavaScript import and class to the textarea. This probably means custom elements won't be used because this only maps clearly to customized built-in elements, which WebKit doesn't support. (v2 of code-input-js aims to do the same, but instead of a textarea the HTML <code-input><textarea id="thetextarea"></textarea></code-input> is used, which then becomes <code-input id="thetextarea">(programatically-added content)</code-input>, allowing document.getElementById("thetextarea") to be used like a textarea all the time in JavaScript, but requiring a bit more JavaScript in the code-input.js file for redirecting the interface from code-input elements.)

I don't want to add this large "mirror" just to have different colors for different words. There is a textarea, and I am willing to let you add some CSS and JS to color it nicely.

I have concerns about removing the "mirror" because it partially requires rewriting syntax highlighter code rather than being syntax-highlighter-agnostic as code-input.js currently is, as well as Custom Highlight API-specific reasons added below. The "mirror" element will likely remain an option, but other methods may be available as alternatives if they provide reasons to be preferred in some cases (one, both or neither of 1. your linear-gradient approach; 2. @MattiasBuelens' Custom Highlight API described below).

So if we are already breaking stuff... here is a crude demo of a single element textarea with syntax highlighting. I actually wanted to do it with shaders, but this just uses background-image: linear-gradient with background-clip: text to add color, while accounting for the latter's various quirks in both Chromium and Firefox.

This is pretty cool and has several places where it may improve performance (e.g. less rendered DOM) and several where it may worsen it (e.g. more CSS parsing). Please see my request for performance tests below.

I didn't test on Safari yet,

and it currently only supports monospace, and does not support variable width tabs nor wrapping.

Wrapping is something I'd very much like to be available in major version 3, as is more flexibility with fonts (for more diverse uses than code), and tab alignment is pretty useful for code editors. Italic, bold, etc. in themes are also not supported by this. I think that if this highlight method is an option it cannot be the main one.

It uses highlight.js for parsing. Look ma, no spans: https://codepen.io/eyaler/pen/NPGOaJb

What do you think? Would you consider this a viable approach at least for some of the simpler use cases?

Someone carrying out performance tests would be helpful if you think this approach holds practicality despite its weaknesses (unfortunately I don't have time right now). If it proves faster it could be an optional alternative to the "mirror" approach (I like swappability of modules of code, but not maintaining both one option and another which is fundamentally different but without benefits).

@MattiasBuelens wrote:

While we're throwing ideas around, have you considered using the CSS Custom Highlight API instead?

This is an interesting, potentially higher-performance alternative to @eyaler's linear-gradient approach, which supports more different types of highlighting scheme. The main problem is that it doesn't look very easy to retain compatibility with general highlighters like highlight.js and Prism.js, and thus choice, without losing the performance benefits. This, again, would be an alternative

Bramus made a demo on how it could be used to do syntax highlighting, and apparently browser support is quite good nowadays. You can't use a <textarea> yet though, so you'd have to switch to a <pre contenteditable> instead...

The problems are:

• lack of plug-and-play compatibility with all non-editable highlighters (see above).
• the textarea JavaScript API might be expected by people as a gracefully-degraded fallback when code-input is unloadable or disabled.
• the browser support is still not satisfactory in my opinion (I hoped code-input.js v3 would have a formal standard of browser support and slightly more than v2 because my use of architecture/CSS etc. caused some limitations in browser support that could likely be fixed in a rewrite of the CSS. I'm generally for quite a lot of browser support, for environmental, accessibility and anti-planned-obsolescence reasons.)
• Firefox appears to have no support for text-decoration and text-shadow with this; text-decoration is required for a somewhat-WYSIWG markdown highlighting theme, and text-shadow is in Prism's default theme. This may be a good thing to implement at browser level, though its lack of implementation for a while suggests it is architecturally difficult.

And maybe use attachInternals() to have it participate in a surrounding <form>? 🤔

This is a good idea for one of the textarea interface features, although it feels a bit like returning to the reimplementing-expected-APIs-with-more-code approach the slimmer version aims to avoid.

[paulrouget]

So if we are already breaking stuff... here is a crude demo of a single element textarea with syntax highlighting.

That's dope. I built something very similar with mix-blend-mode: https://jsfiddle.net/zfo395ag/5 (and wasm + treesitter, but that's just extra-for-fun).

--

Generally speaking, code-input was interesting to me because "it's just a textarea". I hope it will stay that way.

[WebCoder49]

Thanks for your ideas, @paulrouget!

So if we are already breaking stuff... here is a crude demo of a single element textarea with syntax highlighting.

That's dope. I built something very similar with mix-blend-mode: https://jsfiddle.net/zfo395ag/5 (and wasm + treesitter, but that's just extra-for-fun).

I've looked into mix-blend-mode before, but it had some issues a bit like the two alternatives discussed above. I thought it may be useful in some cases due to it preventing duplicate Ctrl+F results in Firefox (a browser bug that has been fixed; I would personally initially focus development on the existing element-mirror approach unless mix-blend-mode has enough performance advantages): https://github.com/WebCoder49/code-input/issues/68#issuecomment-1859871881

Generally speaking, code-input was interesting to me because "it's just a textarea". I hope it will stay that way.

I personally think this is one of the main differentiators of code-input.js: having a familiar native interface. I think code-input.js should use a textarea by default unless someone can demonstrate that contenteditable has significant advantages.

[WebCoder49]

@eyaler, @MattiasBuelens and @paulrouget; just to confirm you've noticed this....?

By the way, please could you give your feedback on this paragraph from above?:

I'm also considering releasing the new version under the MPL-2.0 license which allows plugins under any license, and use of the code in a project under any license (both like MIT) but preserves the availability of the library itself as open source by requiring edited versions of existing plugins and the core library (only those files) to be distributed with source code under an open source license. Do you have feedback on this?

[eyaler]

@WebCoder49 I fixed my code to allow for both tabs and wrapping (strictly at container width), and it is now tested to also work for Firefox/Chrome/Safari on macOS and iOS, and most other glitches I've found are fixed.

I still need to look into performance, however please note that putting in highlight.js was just a quick hack for demoing, and there could already be use cases that want simple highlighting and do not need complex parsing.

I believe that bold/italic should be out of scope for this.

Perhaps mix-blend / background-blend could be useful to control glyph background color, as well as for cases where bg-clip: text is broken (e.g. translations in Chromium), and also reduce to slight ghost halo I currently have, but i really wanted to see how far i can go with the single element, and i am happy not to require an align background div (altough you might need that anyway for the overall bg color).

The last big issue is monospace... it does make sense for code. going variable-space is possible but would be much more compute intensive (and fragile). For monospace there could still be issues with missing glyphs, or with user defaults using different monospace fonts for different languages, so best to provide good defaults than rely on browser defaults.

Anyway, I now feel like this passes proof of concept, and as long as it is not found to be too fragile, I would be interested to standartize this option in a proper library as code-input, if you want it.

Regarding licensing, I don't have any special feedback. Personally I don't enforce copy-left, but I don't see harm having this option at a file-level with MPL.

[WebCoder49]

Thanks for your feedback and time, @eyaler!

Anyway, I now feel like this passes proof of concept, and as long as it is not found to be too fragile, I would be interested to standartize this option in a proper library as code-input, if you want it.

This is a good idea since you've been willing to help; I don't think it should be the only behaviour of code-input (the "mirror" should still be an option, for more styling options and better integration with some plugins) but as an option it would be good. Benefits include the caret-color following the highlighting scheme, immediate feedback of visible characters when typing, and potentially performance.