yard
yard copied to clipboard
lsegal
Unicode characters breaks YARD generation
Unicode characters in the Ruby or README files is breaking YARD with incompatible character encodings: ASCII-8BIT and UTF-8 (Encoding::CompatibilityError
When I used the superscript 3 character (³) in my files, YARD will fail to generate correct output.
I have verified that I saved the files with UTF-8 encoding by inspecting the files with a hex-editor:
Steps to reproduce
.yardopts
--title "test³"
--markup markdown
*.rb
-
README.md
hello.rb
module Hello
# hi³ there.
def self.hi
345
end
end
README.md
# Hello World³
Actual Output
C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/templates/default/layout/html/layout.erb:21:in `_erb_cache_5': incompatible character encodings: ASCII-8BIT and UTF-8 (Encoding::CompatibilityError)
from C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/lib/yard/templates/template.rb:289:in `erb'
from C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/templates/default/layout/html/setup.rb:62:in `layout'
from C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/lib/yard/templates/template.rb:369:in `render_section'
from C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/lib/yard/templates/template.rb:261:in `block (2 levels) in run'
from C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/lib/yard/templates/template.rb:258:in `each'
from C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/lib/yard/templates/template.rb:258:in `block in run'
from C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/lib/yard/templates/template.rb:400:in `add_options'
from C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/lib/yard/templates/template.rb:257:in `run'
from C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/lib/yard/templates/template.rb:136:in `run'
from C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/templates/default/fulldoc/html/setup.rb:52:in `block in serialize_index'
from C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/lib/yard/templates/engine.rb:123:in `block in with_serializer'
from C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/lib/yard/logging.rb:82:in `capture'
from C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/lib/yard/templates/engine.rb:121:in `with_serializer'
from C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/templates/default/fulldoc/html/setup.rb:51:in `serialize_index'
from C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/templates/default/fulldoc/html/setup.rb:68:in `serialize_file'
from C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/templates/default/fulldoc/html/setup.rb:11:in `block in init'
from C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/templates/default/fulldoc/html/setup.rb:10:in `each'
from C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/templates/default/fulldoc/html/setup.rb:10:in `each_with_index'
from C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/templates/default/fulldoc/html/setup.rb:10:in `init'
from C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/lib/yard/templates/template.rb:195:in `initialize'
from C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/lib/yard/templates/template.rb:131:in `new'
from C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/lib/yard/templates/template.rb:136:in `run'
from C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/lib/yard/templates/engine.rb:105:in `generate'
from C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/lib/yard/cli/yardoc.rb:357:in `run_generate'
from C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/lib/yard/cli/yardoc.rb:267:in `run'
from C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/lib/yard/cli/command.rb:14:in `run'
from C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/lib/yard/cli/command_parser.rb:72:in `run'
from C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/lib/yard/cli/command_parser.rb:54:in `run'
from C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/bin/yard:13:in `<top (required)>'
from C:/Ruby32-x64/bin/yard:32:in `load'
from C:/Ruby32-x64/bin/yard:32:in `<main>'
If I remove the superscript 3 from the README the output becomes:
C:\Users\tthomas2\Desktop\yard-bug>yard doc --debug
[debug]: Parsing ["*.rb"] with `ruby` parser
[debug]: Parsing hello.rb
[debug]: Serializing to .yardoc/objects/root.dat
[debug]: Re-generating object ...
[debug]: Re-generating object Hello...
[debug]: Generating asset js/jquery.js
[debug]: Serializing to doc/js/jquery.js
[debug]: Generating asset js/app.js
[debug]: Serializing to doc/js/app.js
[debug]: Generating asset js/full_list.js
[debug]: Serializing to doc/js/full_list.js
[debug]: Generating asset css/style.css
[debug]: Serializing to doc/css/style.css
[debug]: Generating asset css/common.css
[debug]: Serializing to doc/css/common.css
[debug]: Generating asset css/full_list.css
[debug]: Serializing to doc/css/full_list.css
[debug]: Generating asset class_list.html
[debug]: Serializing to doc/class_list.html
[debug]: Generating asset method_list.html
[debug]: Serializing to doc/method_list.html
[debug]: Generating asset file_list.html
[debug]: Serializing to doc/file_list.html
[debug]: Generating asset frames.html
[debug]: Serializing to doc/frames.html
[debug]: Serializing to doc/_index.html
[debug]: Serializing to doc/index.html
[debug]: Serializing to doc/file.README.html
[debug]: Serializing to doc/top-level-namespace.html
[error]: Exception occurred while generating 'Hello.html'
[error]: CompatibilityError: incompatible character encodings: ASCII-8BIT and UTF-8
[error]: Stack trace:
C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/templates/default/layout/html/layout.erb:21:in `_erb_cache_5'
C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/lib/yard/templates/template.rb:289:in `erb'
C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/templates/default/layout/html/setup.rb:62:in `layout'
C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/lib/yard/templates/template.rb:369:in `render_section'
C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/lib/yard/templates/template.rb:261:in `block (2 levels) in run'
C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/lib/yard/templates/template.rb:258:in `each'
Files: 1
Modules: 1 ( 1 undocumented)
Classes: 0 ( 0 undocumented)
Constants: 0 ( 0 undocumented)
Attributes: 0 ( 0 undocumented)
Methods: 1 ( 0 undocumented)
50.00% documented
The documentation "completes" but with an inline error message that doesn't stop the process. This time it's the superscript 3 in hello.rb that breaks.
I'm wondering if YARD is loading text from templates as ASCII and thus failing to combine UTF-8 and ASCII strings.
Expected Output
Expected YARD to generate the documentation without errors.
Environment details:
- OS: Windows 10
- Ruby version (
ruby -v): ruby 3.2.2 (2023-03-30 revision e51014f9c0) [x64-mingw-ucrt] - YARD version (
yard -v): yard 0.9.34
I have read the Contributing Guide.
I can't seem to reproduce this on ruby 3.3.x with yard 0.9.35. Is this still an issue for you?
