File traversal is recursive, which could result in SystemStackError

[fsateler]

trafficstars

On a large project, it is easy to see lots of (debug) messages like this:

[debug]: Missing object Item in file `app/forms/item/assign_form.rb', moving it to the back of the line.

These are caused by idioms of the type Item::AssignForm, as opposed to

module Item
  class AssignForm

Grepping for that phrase, yields this code in ensure_loaded!:

https://github.com/lsegal/yard/blob/359006641260eef1fe6d28f5c43c7c98d40f257d/lib/yard/handlers/base.rb#L566-L570

In particular, parser.parse_remaining_files causes a recursion:

On sufficiently large projects, this causes a SystemStackError.

Steps to reproduce

% seq 1 10000 | while read i ; do
echo "module Foo::My${i} ; end" > my_${i}.rb
done
% bundle exec yard doc --debug '*.rb'

Actual Output

<snip>
[debug]: Parsing my_79.rb
[debug]: Missing object Foo in file `my_79.rb', moving it to the back of the line.
[debug]: Parsing my_80.rb
[debug]: Missing object Foo in file `my_80.rb', moving it to the back of the line.
[debug]: Parsing my_81.rb
[debug]: Missing object Foo in file `my_81.rb', moving it to the back of the line.
[debug]: Parsing my_82.rb
[debug]: Missing object Foo in file `my_82.rb', moving it to the back of the line.
[debug]: Parsing my_83.rb
<snip>
[debug]: Missing object Foo in file `my_556.rb', moving it to the back of the line.
[debug]: Parsing my_557.rb
[debug]: Missing object Foo in file `my_557.rb', moving it to the back of the line.
[debug]: Parsing my_558.rb
[debug]: Missing object Foo in file `my_558.rb', moving it to the back of the line.
[debug]: Parsing my_559.rb
[debug]: Missing object Foo in file `my_559.rb', moving it to the back of the line.
[debug]: Parsing my_560.rb
<snip>
bundler: failed to load command: yard (<home>/<gems>/ruby/2.6.0/bin/yard)
Traceback (most recent call last):
        10483: from <home>/.rbenv/versions/2.6.6/bin/bundle:23:in `<main>'
        10482: from <home>/.rbenv/versions/2.6.6/bin/bundle:23:in `load'
        10481: from <home>/.rbenv/versions/2.6.6/lib/ruby/gems/2.6.0/gems/bundler-2.2.11/exe/bundle:37:in `<top (required)>'
        10480: from <home>/.rbenv/versions/2.6.6/lib/ruby/gems/2.6.0/gems/bundler-2.2.11/lib/bundler/friendly_errors.rb:130:in `with_friendly_errors'
        10479: from <home>/.rbenv/versions/2.6.6/lib/ruby/gems/2.6.0/gems/bundler-2.2.11/exe/bundle:49:in `block in <top (required)>'
        10478: from <home>/.rbenv/versions/2.6.6/lib/ruby/gems/2.6.0/gems/bundler-2.2.11/lib/bundler/cli.rb:24:in `start'
        10477: from <home>/.rbenv/versions/2.6.6/lib/ruby/gems/2.6.0/gems/bundler-2.2.11/lib/bundler/vendor/thor/lib/thor/base.rb:485:in `start'
        10476: from <home>/.rbenv/versions/2.6.6/lib/ruby/gems/2.6.0/gems/bundler-2.2.11/lib/bundler/cli.rb:30:in `dispatch'
         ... 10471 levels...
            4: from <home>/<gems>/ruby/2.6.0/gems/yard-0.9.25/lib/yard/parser/ruby/ruby_parser.rb:237:in `visit_event'
            3: from <home>/<gems>/ruby/2.6.0/gems/yard-0.9.25/lib/yard/parser/ruby/ast_node.rb:64:in `source_range'
            2: from <home>/<gems>/ruby/2.6.0/gems/yard-0.9.25/lib/yard/parser/ruby/ast_node.rb:345:in `reset_line_info'
            1: from <home>/<gems>/ruby/2.6.0/gems/yard-0.9.25/lib/yard/parser/ruby/ast_node.rb:200:in `children'
<home>/<gems>/ruby/2.6.0/gems/yard-0.9.25/lib/yard/parser/ruby/ast_node.rb:200:in `select': stack level too deep (SystemStackError)

Expected Output

Not a crash.

Environment details:

• OS: Linux
• Ruby version (ruby -v): ruby 2.6.6p146 (2020-03-31 revision 67876) [x86_64-linux]
• YARD version (yard -v): yard 0.9.25
• Relevant software dependency/versions:

I have read the Contributing Guide.

[QuentinLemCode]

+1 on this issue Can you merge the linked PR please ? It successfully fix the issue on my project

[lsegal]

@QuentinLemCode the temporary workaround here is to use the module Item; class AssignForm syntax, or simply create an app/forms/item.rb in which you define the outer module:

# app/forms/item.rb:
module Item; end

This would be fully compatible with Rails / Zeitwerk naming conventions and should not create any issues with your existing codebase except for the few extra files.

You could also create a single lib/extras/modules.rb file that contains all of these toplevel modules and included at the top of your .yardopts (given the naming, it would not be Zeitwerk compatible and thus never loaded by Rails, only YARD):

# .yardopts
lib/extras/modules.rb
lib
app

This would only need to be done for widely used namespaces, and only if the first suggestion is not possible.

FWIW, rubocop can auto-fix/auto-format the class Item::AssignForm syntax for you, so there's really no "good" reason to avoid this option for codebases that require it; it is also the default style.