Failed to load AsciiDoc document - incompatible encodings: UTF-8 and US-ASCII

[marcelhuberfoo]

trafficstars

Unfortunately I cannot preview a file (UTF-8) I am currently editing as it seems the generated temporary file is written in plain US ASCII and it cannot include a file written in UTF-8 :(

I am using the latest plugin: Asciidoctor Editor 2.1.1 de.jcup.asciidoctoreditor.feature.group Albert Tregnaghi

Encodings of the files as detected by the OS:

~/Develop/DatEng $  file /tmp/asciidoctor-editor-temp/project_DataEngineeringExercises-70643087/1607348072_hidden-editorfile_README.adoc
/tmp/asciidoctor-editor-temp/project_DataEngineeringExercises-70643087/1607348072_hidden-editorfile_README.adoc: ASCII text
~/Develop/DatEng $  file /home/Develop/DatEng/StoredProcedures/README.adoc
/home/Develop/DatEng/StoredProcedures/README.adoc: UTF-8 Unicode text

Contents of generated file:

// origin :/home/Develop/DatEng/StoredProcedures/README.adoc
// editor :1607348072
// basedir:/home/Develop/DatEng
include::StoredProcedures/README.adoc[]

Here is the log:

>> Triggered ASP server start at port:4444
11:43:13.225 [main] INFO  d.j.a.s.a.service.AsciidoctorService - Starting, will create asciidoctorj instance
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.headius.backport9.modules.Modules (file:/home/m1huber/.eclipse-asciidoctor-editor/libs/2.1.0/asp-server-asciidoctorj-dist.jar) to method sun.nio.ch.NativeThread.signal(long)
WARNING: Please consider reporting this to the maintainers of com.headius.backport9.modules.Modules
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
11:43:15.092 [main] INFO  d.j.a.s.a.service.AsciidoctorService - Created instance
11:43:15.092 [main] INFO  d.j.a.s.a.service.AsciidoctorService - Starting warmup phase for asciidoctor
11:43:15.521 [main] INFO  d.j.a.s.a.service.AsciidoctorService - Warmup done
11:43:15.522 [main] INFO  d.jcup.asp.server.core.CoreAspServer - Server starting at port:4444
11:43:15.527 [main] INFO  d.jcup.asp.server.core.CoreAspServer - Server secret :xxxxxxxxxxxxxxxxxxxxxxx
>> ASP Server has been started successfully
ASP: Processing file:/home/Develop/DatEng/StoredProcedures/README.adoc
ASP client->server :{"version":"0.0.0","command":"is_alive"}
ASP server->client :{}
ASP client->server :{"version":"0.0.0","command":"convert_file","source_filepath":"/tmp/asciidoctor-editor-temp/project_DataEngineeringExercises-70643087/1607348072_hidden-editorfile_README.adoc","options":{"header_footer":false,"to_dir":"/tmp/asciidoctor-editor-temp/project_DataEngineeringExercises-70643087","sourcemap":"true","base_dir":"/home/Develop/DatEng","safe":0,"backend":"html5","attributes":{"showtitle":true,"source-highlighter":"rouge","eclipse-editor-basedir":"/home/Develop/DatEng","icons":"font","coderay-css":"style","env":"eclipse","env-eclipse":"","angproj_url":"{ci_server_baseurl}ifs/AngProj","author":"Marcel Huber","backend":"revealjs","bankdb_url":"{ci_project_dirref}Databases/bank","ci_artifacts_prefix":"-/jobs/artifacts/","ci_commit_link":"pass:q,m[link:{ci_project_url}/../../commit/]","ci_commit_ref_name":"{ci_commit_tag}","ci_project_artifacts_browse":"{ci_project_artifactsurl}{ci_commit_ref_name}/browse/","ci_project_artifacts_download":"{ci_project_artifactsurl}{ci_commit_ref_name}/raw/","ci_project_artifacts_view":"{ci_project_artifactsurl}{ci_commit_ref_name}/file/","ci_project_artifactsurl":"{ci_project_url}{ci_artifacts_prefix}","ci_project_url":"/../","ci_server_baseurl":"{ci_project_url}../../../","current_dir_prefix":"OR-Mapper_JPA/","description":"Exercises to learn object relational mapping using JDBC and postgresql","email":"marcel.huber@...","ex_db_name":"dvdstore_500mb","ex_number":"0","example-caption":"Lösungsvorschlag","imagesdir":"/tmp/asciidoctor-editor-temp/project_DataEngineeringExercises-70643087/images","imagesoutdir":"/tmp/asciidoctor-editor-temp/project_DataEngineeringExercises-70643087/images","issue_prefix":"issues/","keywords":"database, sql, postgresql, jdbc, jpa, orm, java","lang":"de","media":"prepress | screen | print","mr_prefix":"merge_requests/","myexerciseprefix":"Übungen;{sp}","mysubtitle":": Data Engineering Frühlingssemester 2020","outfilesuffix":".pdf","pdf-page-layout":"landscape","pdf-page-size":"letter  # 6in x 10.66in (16:9-like)","query_name":"employees-managed-by-leto","query_number":"0","queryoutputfile":"ColumnStore.MonetDB.log","queryoutputfilepg":"ColumnStore.PostgreSQL.log","revdate":"June 15, 2018","revnumber":"00","revremark":"<pass by attribute>","rouge-style":"pastie","solutions-text":"pass:q[ #*_Musterlösung_*#]","stem":"latexmath","subsub":"1","subtopic":"a","title-logo-image":"image:Brainspace_Logo.png[]","title-page-background-image":"image:Brainspace_Logo.png[]","title-slide-background-image":"70s.jpg","title-slide-transition":"zoom","title-slide-transition-speed":"fast","toc-title":"Inhaltsverzeichnis","outdir":"/tmp/asciidoctor-editor-temp/project_DataEngineeringExercises-70643087"}}}
Mar 20, 2020 11:43:17 AM org.asciidoctor.jruby.internal.JRubyAsciidoctor convertFile
SEVERE: (CompatibilityError) asciidoctor: FAILED: /tmp/asciidoctor-editor-temp/project_DataEngineeringExercises-70643087/1607348072_hidden-editorfile_README.adoc: Failed to load AsciiDoc document - incompatible encodings: UTF-8 and US-ASCII
11:43:17.929 [main] ERROR d.j.a.s.a.s.ConvertLocalFileServiceImpl - Was not able to convert
org.asciidoctor.jruby.internal.AsciidoctorCoreException: org.jruby.exceptions.EncodingError$CompatibilityError: (CompatibilityError) asciidoctor: FAILED: /tmp/asciidoctor-editor-temp/project_DataEngineeringExercises-70643087/1607348072_hidden-editorfile_README.adoc: Failed to load AsciiDoc document - incompatible encodings: UTF-8 and US-ASCII
	at org.asciidoctor.jruby.internal.JRubyAsciidoctor.convertFile(JRubyAsciidoctor.java:400)
	at org.asciidoctor.jruby.internal.JRubyAsciidoctor.convertFile(JRubyAsciidoctor.java:366)
	at de.jcup.asp.server.asciidoctorj.service.ConvertLocalFileServiceImpl.handleConvert(ConvertLocalFileServiceImpl.java:81)
	at de.jcup.asp.server.asciidoctorj.service.ConvertLocalFileServiceImpl.convertFile(ConvertLocalFileServiceImpl.java:55)
	at de.jcup.asp.server.asciidoctorj.AsciiDoctorJServerClientRequestHandler.handleRequest(AsciiDoctorJServerClientRequestHandler.java:50)
	at de.jcup.asp.server.core.CoreAspServer.waitForClient(CoreAspServer.java:121)
	at de.jcup.asp.server.core.CoreAspServer.start(CoreAspServer.java:81)
	at de.jcup.asp.server.asciidoctorj.AsciidoctorJServer.start(AsciidoctorJServer.java:40)
	at de.jcup.asp.server.asciidoctorj.AsciidoctorJServerApplication.main(AsciidoctorJServerApplication.java:46)
Caused by: org.jruby.exceptions.EncodingError$CompatibilityError: (CompatibilityError) asciidoctor: FAILED: /tmp/asciidoctor-editor-temp/project_DataEngineeringExercises-70643087/1607348072_hidden-editorfile_README.adoc: Failed to load AsciiDoc document - incompatible encodings: UTF-8 and US-ASCII
ASP server->client :{"version":"1.1.1","error":"org.jruby.exceptions.EncodingError$CompatibilityError: (CompatibilityError) asciidoctor: FAILED: /tmp/asciidoctor-editor-temp/project_DataEngineeringExercises-70643087/1607348072_hidden-editorfile_README.adoc: Failed to load AsciiDoc document - incompatible encodings: UTF-8 and US-ASCII"}
Command convert_file failed.Failed:org.jruby.exceptions.EncodingError$CompatibilityError: (CompatibilityError) asciidoctor: FAILED: /tmp/asciidoctor-editor-temp/project_DataEngineeringExercises-70643087/1607348072_hidden-editorfile_README.adoc: Failed to load AsciiDoc document - incompatible encodings: UTF-8 and US-ASCII

[de-jcup]

Hello, having no such problems:

and leads to

So I need your assistance here, to provide a fix:

Problem evaluation / bug fixing

Please do me a favor:

1. fork https://github.com/de-jcup/eclipse-asciidoctor-testprojects

2. check which of the test projects are like your own setup (e.g. https://github.com/de-jcup/eclipse-asciidoctor-testprojects/tree/master/asciidoctor-testproject2-simple-imagedirset)

3. create there a subfolder "issue-301" and add there your problematic files, or simular ones (e.g. your README.adoc and the include etc.)

4. check that this fails ...

5. push to your forked repo

6. create a PR

I will merge the testproject PR . After this It should be much easier to evaluate the problem.

Additional information

• What exact OS are you using ? (looks like Linux...)
• Did you start your java vm with a dedicated charset ?
• If not Windows (cp1252...) which is your standard charset ?

Workaround

In the mean time you can switch to native asciidoctor instance as a workaround (see https://github.com/de-jcup/eclipse-asciidoctor-editor/wiki/FAQ#how-can-i-use-installed-native-asciidoctor)

[marcelhuberfoo]

@de-jcup Thank you for your quick reply. I will follow your steps and prepare for the PR. I already tried the workaround which showed me another problem regarding asciidoctor-block comments (////) which do not get parsed correctly if I am not mistaken. But this will also be contained in the forked repo containing my examples then.

[marcelhuberfoo]

Yes, I am using Linux, Archlinux. Default charset is UTF-8 according to these language related env vars:

LANG=en_US.UTF-8
LC_MEASUREMENT=de_CH.UTF-8
LC_MONETARY=de_CH.UTF-8
LC_NUMERIC=de_CH.UTF-8
LC_PAPER=de_CH.UTF-8
LC_TIME=de_CH.UTF-8

Jave anvironment:

~/Develop/DatEng $ /usr/bin/java -version
openjdk version "13.0.2" 2020-01-14
OpenJDK Runtime Environment (build 13.0.2+8)
OpenJDK 64-Bit Server VM (build 13.0.2+8, mixed mode)

My Eclipse has no special charset spec when looking at the ps entry:

user   518761  518726  5 11:02 pts/0    00:11:15 /usr/bin/java -Xms1g -Xmx4g -Dorg.eclipse.swt.browser.DefaultType=webkit -Dorg.eclipse.swt.browser.UseWebKitGTK=true --add-modules=ALL-SYSTEM -jar /usr/lib/cevelop//plugins/org.eclipse.equinox.launcher_1.5.600.v20191014-2022.jar -os linux -ws gtk -arch x86_64 -showsplash -launcher /usr/lib/cevelop/cevelop -name Cevelop --launcher.library /usr/lib/cevelop//plugins/org.eclipse.equinox.launcher.gtk.linux.x86_64_1.1.1100.v20190907-0426/eclipse_1801.so -startup /usr/lib/cevelop//plugins/org.eclipse.equinox.launcher_1.5.600.v20191014-2022.jar --launcher.overrideVmargs -exitdata 98003 -vm /usr/bin/java -vmargs -Xms1g -Xmx4g -Dorg.eclipse.swt.browser.DefaultType=webkit -Dorg.eclipse.swt.browser.UseWebKitGTK=true --add-modules=ALL-SYSTEM -jar /usr/lib/cevelop//plugins/org.eclipse.equinox.launcher_1.5.600.v20191014-2022.jar

All the rest follows in the fork/PR.

[de-jcup]

Thanks for https://github.com/de-jcup/eclipse-asciidoctor-testprojects/pull/5 - after merging your variant and opening it, I also got the problem:

After inspection of data I found the problem: Inside ADocMeta.asc you defined variable mysubtitle with an content starting with a leading colon. Removing the leading colon fixed the problem:

see https://raw.githubusercontent.com/de-jcup/eclipse-asciidoctor-testprojects/master/asciidoctor-testproject3-simple-noimagedirset/issue-301/reduced-no-include/README.adoc for details.

Switching between the two different variable definitions inside this test file will switch also between working and error.

[de-jcup]

@marcelhuberfoo I think just changing your variable definition should fix the problem