Newlines get ignored

[richardmtsr]

Any newline character seems to be ignored, leaving my content rendered all on a single line. The conversion to react / html components is working but due to the ignoring of new lines it doesn't display correctly.

Is there a setting for this?

[JiLiZART]

Seems like a bug

[mmichaelis]

As we also struggle with this, some thoughts here:

Architectural Challenge

With only having scratched the very surface of BBob, I see an architectural challenge in mapping, as there does not seem to be a "non-BBCode-tag" to element mapping approach. We would have some plain text content here, that in some contexts (thus, non-code-/non-preformatted-contexts) should get newlines transformed to an element.

<p> or <br>

The leading question for us was, if to wrap newlines into paragraphs and/or add <br> elements.

Unfortunately, there is a wild discussion in the net, which to prefer.

Some refer to behave similar to Markdown: Ignore newlines, unless either two in a row, which become a paragraph or, a newline after two spaces as last characters in a line that becomes a <br>.

Other solutions I found prefer plain newline to <br> mapping.

Conclusion of the Research

As it seems, most of the resources do not make up their mind on proper transformation of BBCode to HTML and vice versa when it comes to <br> and <p>. Rough tendency, without clear specification, seems to be to make any newline a <br> tag.

I would recommend adding some configuration options here, either by some "selected options" or to make it part of the preset-API how to deal with it.

Research

Some possible demo references:

• BBCode Text Editor

  Each newline character becomes a <br>.

• BBCode Editing | CKEditor 4 Documentation

  Same here: Newlines becomes <br>.

• BBcode to HTML converter

  Has an option to turn on newline to <br> mapping.

• SCEditor - A lightweight WYSIWYG HTML and BBCode editor - SCEditor

  Transforms newlines (Enter) to <div> and on Shift+Enter adds a <br>.

And some demo references doing it the other way round (HTML to BBCode)

• HTML to BBCode Converter | LambdaTest

  Seems to be rather bugged regarding <p> and <br>. Thus, ignores the first, and has some strange transformation for the second.

• HTML to BBCode Converter Online tool

  Transforms <p> and <br> just to single newlines.

And some more links:

• <br> tag in bbcode - x10Hosting: Free Hosting Community

  One comment states the <p> as the default mapping, but without references.

• Any Way to Force a Line Break in BBCode?

  Referring to some extra BBCode [br] to trigger line-breaks (that may also work for BBob due to its fallback parsing an unhandled BBCode-tag to corresponding HTML-Element on same name, unless you are working with allow-lists).

• The way bbcode handles new line breaks [#51152] | Drupal.org

  Very old comments talking about some "intelligent paragraph" handling to introduce/opt-in.

• BBCode - Wikipedia (en)

  Contains no hints at all.

• BBCode – Wikipedia (de)

  Refers to the "extra newline to paragraph" solution. This aligns with the solution found for Markdown.

• kefirbb/src/org/kefirsf/bb/default.xml at kefirbb-1.5 · kefirfromperm/kefirbb

  Also transforms all EOL-Newlines to plain <br> elements.

[Alteras1]

Thought I'd add my team's work to the discussion in case someone wants to go down the <br> route and save them a day's work.

We created a plugin that will walk the tree and insert <br> wherever there's an end of line. Our use case is a bit odd since we plan on using it alongside a bbcode tag [nobr] that will prevent the conversion of line breaks. Regardless, the code for the plugin can be generalized.

https://github.com/RpNation/bbcode/blob/main/bbcode-src/plugins/lineBreak.js

/**
 * Plugin that converts line breaks to `<br/>` tags.
 * To use, put as function similar to the presets.
 *
 * If a node is marked with `noLineBreakConversion`, then it'll skip the parsing the children
 *
 * @example
 * ```ts
 * const output = bbob([preset(), lineBreakPlugin()]).process(input, {render}).html
 * ```
 */
import { isEOL } from "@bbob/plugin-helper";

/**
 * Checks if input is an object
 * @param value input
 * @returns if value is an object
 */
const isObj = (value) => typeof value === "object";

/**
 * Walks the tree of nodes. Will add `br` tag to all `\n` in format that can be used in any renderer.
 * Preserves \n so that markdown-it doesn't try to treat everything like a block
 *
 * If a node has the property noLineBreakConversion is encountered, will skip parsing children.
 * @param t tree of nodes to be processed
 * @returns modified tree
 */
const walk = (t) => {
  const tree = t;

  if (Array.isArray(tree)) {
    for (let idx = 0; idx < tree.length; idx++) {
      const child = walk(tree[idx]);
      if (Array.isArray(child)) {
        tree.splice(idx, 1, ...child);
        idx += child.length - 1;
      } else {
        tree[idx] = child;
      }
    }
  } else if (tree && isObj(tree) && tree.content) {
    if (tree.disableLineBreakConversion) {
      // stop walk. children won't be parsed to have <br>
      return tree.tag ? tree : tree.content;
    }
    walk(tree.content);
  }

  if (isEOL(tree)) {
    return [{ tag: "br", content: null }, "\n"];
  }

  return tree;
};

/**
 * Converts `\n` to `<br/>` self closing tag. Supply this as the last plugin in the preset lists
 *
 * @example converts all line breaks to br
 * ```ts
 * const output = bbob([preset(), lineBreakPlugin()]).process(input, {render}).html
 * ```
 * @example will not convert line breaks inside [nobr]
 * ```ts
 * const nobr = (node: TagNode) => {return { disableLineBreakConversion: true, content: node.content }}; \\ tag in preset
 * ...
 * const output = bbob([preset(), lineBreakPlugin()]).process(input, {render}).html
 * ```
 * @returns plugin to be used in BBob process
 */
export const lineBreakPlugin = () => {
  return (tree) => walk(tree);
};

We made the deliberate choice to preserve \n alongside the <br> since our use case is a bit extreme where we need to make sure it doesn't break a downstream processing. Personally, I think this is better since the resulting HTML will be close to how <br> usage is. Either way, the return can be easily edited to something else if need be.

We just call the plugin after the preset. The intention is to run this as the final step before rendering. https://github.com/RpNation/bbcode/blob/main/bbcode-src/index.js#L19-L22

  bbob([preset(), lineBreakPlugin()]).process(code, {
    render,
    ...options,
  });

So as a bit of background, we're using BBob as our future bbcode engine for migration from Xenforo, which also does the <br> replacement route. The extra downstream processing is actually a markdown-it engine that we modified to prevent <p> tag output.

I think this is enough for fulfilling the <br> replacement option?

[mmichaelis]

@Alteras1, thanks for your workaround for <br> handling via extra plugin. We thought of a similar approach, as this is an obvious extension point.

We skipped though, as we need the result to be parsed as CKEditor 5 data view, which again works best having paragraphs instead. And this again requires more "context awareness" best represented in a preset.

Our solution is a custom preset, mostly a copy of the HTML5 Preset (thus, we reuse the defaultTags). This is, because we need to be context aware (e.g., do not add paragraphs within a code block).

Our solution mainly consists of these adaptations:

• process: Instead of the default-processor, that just walks the tree, we required an artificial root tag as intermediate state, so that we have "paragraph detection" on root level. Our customized function adds a [root] tag and later removes this prior to generating HTML.

• Rules per Paragraph Support: For each TagNode we want to support paragraph parsing within its direct contents, we added an extra rule (thus, for example, for td, th and quote - also for li as this is the result of the default list processing). Benefit: Within inline tags, we simply ignore newlines (just as before), as will get broken HtML5 otherwise (must not add a paragraph within a span). This also fixes an issue with the existing quote mapping for BBob, which just wraps everything into a paragraph - which is again invalid for lists, for example.

• Markdown Like Paragraph Parsing: In an extra function, we do all the paragraph parsing. We required defining "block-level tags" that must not be wrapped into paragraphs for valid HTML5. As a result, the following BBCode:

  [quote] Hello World!
  [list]
  [*] Item 1
  [/list]
  [/quote]
  

  would render to:

  <blockquote>
    <p>Hello World>!</p>
    <ul><!-- ... ---></ul>
  </blockquote>
  

  while the original HTML5 preset would have generated:

  <blockquote>
    <p>
      Hello World>!
      <ul><!-- ... ---></ul>
    </p>
  </blockquote>
  

Find more details in current snapshots:

• https://github.com/CoreMedia/ckeditor-plugins/blob/292f402d475e515c85d53b8e6b17da097262d491/packages/ckeditor5-bbcode/src/bbob/ckeditor5Preset.ts
• https://github.com/CoreMedia/ckeditor-plugins/blob/292f402d475e515c85d53b8e6b17da097262d491/packages/ckeditor5-bbcode/src/bbob/Paragraphs.ts

Or as part of this pull request, introducing BBCode support as CKEditor 5 Plugin: https://github.com/CoreMedia/ckeditor-plugins/pull/158 (possibly including more refactorings, as the PR is still in progress).

Intermediate Conclusion: Whatever the solution for this issue will be - we obviously need customization options depending on who is the consumer of the generated HTML.