purebasic icon indicating copy to clipboard operation
purebasic copied to clipboard

Help-Sources Encoding Workaround Proposal

Open tajmone opened this issue 11 months ago • 0 comments

Currently the Help documentation is subject to corrupted encoding source files (see #152 and #135) due to the fact that DocMaker expects the .txt sources to be encoded in ISO-8859-1 (aka Latin1), but Git is treating them as UTF-8 files, resulting in non-ASCII characters being wrongly encoded (especially in French and German).

Since it's impossible to enforce ISO-8859-1 encoding on the source files, and pending an update to DocMaker to treat input source files as being UTF-8 encoded, I have a proposal for a workaround to the current situation.

We could store the user-edited sources files in UTF-8 encoding in dedicated source folders (with the added _utf8 suffix) mirroring the current source folders, and then have a script convert them to ISO-8859-1/Latin1 via a tool like iconv:

  • Documentation/English/ — Auto-converted to ISO-8859-1 (ignored by Git).
  • Documentation/English_utf8/ — Editable English sources.
  • Documentation/French/ — Auto-converted to ISO-8859-1 (ignored by Git).
  • Documentation/French_utf8/ — Editable French sources.
  • Documentation/German/ — Auto-converted to ISO-8859-1 (ignored by Git).
  • Documentation/German_utf8/ — Editable German sources.

With the above solution, editors and contributors would be able to edit the source documents in the *_utf8/ folders, and at build-time the various build scripts would take care of converting the UTF-8 .txt sources to ISO-8859-1 into the folders which are used by DocMaker (i.e. the current ones).

Since the current Help source folders will become ignored in the repository, this won't burden the repository with duplicate files (only local clones will end up hosting dual versions of each Help source, i.e. in UTF-8 and Latin1).

Personally, I would have hoped that DocMaker could have been updated to handle UTF-8 sources, but since its source haven't been opened that's not a viable option from third party contributors.

tajmone avatar Sep 19 '23 08:09 tajmone