fix: rework hyphenation

[carlobeltrame]

Fixes #3018, was broken since #2600 Fixes #1642 Fixes #2456 Fixes #2564 Fixes #2739 Enables better solutions for the following issues: Fixes #1238 Fixes #1380 Fixes #1416 Fixes #1662

I added each change in a separate commit, with accompanying tests to make sure the intention of the code change will not as easily be broken again in the future. Especially the textkit layout integration test took a while to write, but should be of great importance to detect unwanted changes in the future.

This is technically a breaking change for people who have previously used custom word wrapping functions, but one could argue this feature has been quite broken for a long time now, as described in #3018. I can also update the documentation website and possibly some of the examples once this is merged.

[changeset-bot[bot]]

🦋 Changeset detected

Latest commit: e343b3db72bf4599b9e8d8c511a3587ef0537705

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 13 packages

Name	Type
@react-pdf/pdfkit	Minor
@react-pdf/textkit	Minor
@react-pdf/font	Patch
@react-pdf/renderer	Patch
@react-pdf/layout	Patch
@react-pdf/render	Patch
@react-pdf/types	Patch
next-14	Patch
next-15	Patch
@react-pdf/vite-example	Patch
@react-pdf/e2e-node-cjs	Patch
@react-pdf/e2e-node-esm	Patch
@react-pdf/stylesheet	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[wojtekmaj]

Insane work. Looks good to me!

[diegomura]

@carlobeltrame great work man! Sorry for the late response here. Can you guide me on how (or if) this fixes solving hyphenation in different languages? I always thought we should export some hyphenation "language specifications" (if that even exists) that people can pass to Document to easily hyphenate with non english rules. Not to do now, but wonder if even possible. I been a bit away from this issue for some time 😅

[carlobeltrame]

@carlobeltrame great work man! Sorry for the late response here. Can you guide me on how (or if) this fixes solving hyphenation in different languages? I always thought we should export some hyphenation "language specifications" (if that even exists) that people can pass to Document to easily hyphenate with non english rules. Not to do now, but wonder if even possible. I been a bit away from this issue for some time 😅

@diegomura see also https://github.com/diegomura/react-pdf-site/pull/150 for a documentation PR which attempts to do just that.

After this PR here, a full example for German and for splitting at hyphens in the source text could look like this:

import { Font } from "@react-pdf/renderer";
import { hyphenateSync as hyphenateDe } from "hyphen/de";

const SOFT_HYPHEN = '\u00ad';
const hyphenationCallback = (word) => {
  // technically, null may be passed as argument
  if (word === null) return [];
  // take care to leave the soft hyphens in the split parts, so a hyphen is
  // rendered in the pdf if the string is split at that position
  const syllables = hyphenateDe(word).split(new RegExp(`(?<=${SOFT_HYPHEN})`));
  // also allow line splitting at dashes in the text, e.g. inside URLs
  return syllables.flatMap((syllable) => syllable.split(/(?<=-)/))
};
Font.registerHyphenationCallback(hyphenationCallback);

I don't think react-pdf should be responsible for providing the hyphenation rules for all languages. I think users should be responsible for selecting and installing only the necessary language-specific and domain-specific hyphenation dictionaries which they use. The other languages of the hyphen package or of other packages can then be removed using tree-shaking.

If react-pdf ever starts to include "presets" for hyphenation in common western languages, the customizable hyphenationCallback will still be necessary. Imagine if a company named "foobar" never wants to have its company name hyphenated in their pdfs, they could implement this using the hyphenationCallback.

Also, please note that I only have experience with western languages such as English, German, French, Italian, etc., which share pretty similar hyphenation strategies. I have no idea how hyphenation works in e.g. middle eastern or asian languages, and cannot consult on how to implement support for that. I think if there is a need for hyphenation in such a language, we need contributors from these regions to describe the required rules in an issue first.

[ybd-project]

Hello. I'm a Japanese speaker. First of all, thank you for the wonderful pull request.

Next, regarding hyphenation rules in Japanese, I personally think they are unnecessary. It's sufficient for each person to break lines at appropriate positions. Also, while hyphenation in English-speaking countries (including languages spoken in European countries) places a “-” at the end of a line, this is not necessary in Japanese. The only thing needed is line breaks at appropriate positions.

Currently, using React-PDF inserts hyphens in Japanese text, which is extremely disruptive. (Even following the steps to disable it doesn't remove them.) Personally, I would like to see this pull request merged as soon as possible.

I hope this is helpful. Thank you.

(Please note that this translation was generated by a translation tool and may contain unnatural expressions.)

[ybd-project]

〔Addendum〕Is the fact that entering meaningless characters (e.g., “aaaa”) doesn't cause a line break related to hyphenation?

[carlobeltrame]

〔Addendum〕Is the fact that entering meaningless characters (e.g., “aaaa”) doesn't cause a line break related to hyphenation?

Yes. The current version of react-pdf adds line breaks on

• whitespace (e.g. aaaa aaaa can be split)
• when the hyphen library recognizes an english word (e.g. the very long but valid english word pneumonoultramicroscopicsilicovolcanoconiosis can be split)
• when you define your own custom hyphenation callback

Since aaaaaaaaaaa has no spaces and is no english word, react-pdf keeps it together on one line by default.

[ybd-project]

<Text style={{ width: '100%' }}>No.3では生命の連続性第二章「遺伝の規則性と遺伝子」から出題されます。それと、これも大変重要な役割を持ちます。</Text>

When I run the code you implemented (part of the pull request), it behaves as shown in the image above.

This is not the desired behavior, and I find it aesthetically unpleasing. Ideally, I want it to behave like this. (I added code to achieve the look shown in this image.)

const JA_REGEX = /^[\p{scx=Hiragana}\p{scx=Katakana}\p{scx=Han}]+$/u;

const getNodes = (
  attributedString: AttributedString,
  { align }: Attributes,
  options: LayoutOptions,
): Node[] => {
  let start = 0;

  const hyphenWidth = 5;
  const softHyphen = '\u00ad';

  // Here!
  attributedString.syllables = attributedString.syllables
    .map((s) => (JA_REGEX.test(s) ? s.split('') : s))
    .flat();

  ...

(/textkit/src/engines/linebreaker/index.ts : getNodes function)

[!NOTE] This issue occurs because Japanese syllables are not being recognized correctly. In the image above, the text is not separated by punctuation such as “。” (equivalent to the English period).