Source file/line set to None for nodes created by sphinx-needs

[staalb]

trafficstars

The issue

Some time ago I was struggling with an issue which caused sphinx to crash with an exception when running the spellchecker (sphinxcontrib.spelling) on a document tree using sphinx-needs (https://github.com/sphinx-doc/sphinx/pull/11183). At that moment I didn't realize this only happened for nodes created by sphinx-needs. I proposed to solve the issue in completely the wrong way, so the fix was rightly rejected.

Another other related problem is that the spellchecker doesn't report the file name/line number when a spelling error is found inside a needs block:

None:None: : Spell check: impedant: ["impudent", "important", "impedance"]: in reset when the CPU is not started (GPIO = high impedant).

Instead of spelling errors in other parts of a document:

doc/design.rst:1081: : Spell check: emiter: ["emitter", "miter", "emote"]: maximum emiter voltage is the base voltage + 0,7V.

My analysis so far

This issue is caused by the fact that docutils.utils.get_source_line(node) returns None for (some or all) nodes which are created by sphinx-needs. get_source_line uses the parent attribute to walk up the node hierarchy until it finaly finds a node with source/line info set. It turns out that the nodes for which get_source_line returns None don't have the parent attribute set.

I think I found a piece of code that might be related or even causing this issue. The function find_and_replace_node_content (in functions/function.py creates new nodes and manually inserts them into a parent node, without using the (as far as I understand) proper methods in the docutils.nodes.Node class. If for example docutils.nodes.Node.append() would have been used to insert the node, the setup_child() method would have set the parent property, along with the document, source and line properties.

I tried to replace the two lines that set node.children = new_children with node.extend(new_children). Unfortunately, that caused an exception which I do not understand yet:

Traceback (most recent call last):
  File "/home/bart/.local/share/virtualenvs/doc-gK3R_GBk/lib/python3.11/site-packages/sphinx/events.py", line 96, in emit
    results.append(listener.handler(self.app, *args))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bart/dev/3s/doc/doc/conf.py", line 158, in patched_process_caller
    result = orig_process_caller(app, doctree, fromdocname)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bart/.local/share/virtualenvs/doc-gK3R_GBk/lib/python3.11/site-packages/sphinx_needs/needs.py", line 298, in process_caller
    check_func(app, doctree, fromdocname, current_nodes[check_node])
  File "/home/bart/.local/share/virtualenvs/doc-gK3R_GBk/lib/python3.11/site-packages/sphinx_needs/roles/need_ref.py", line 154, in process_need_ref
    node_need_ref.replace_self(new_node_ref)
  File "/home/bart/.local/share/virtualenvs/doc-gK3R_GBk/lib/python3.11/site-packages/docutils/nodes.py", line 1011, in replace_self
    self.parent.replace(self, new)
  File "/home/bart/.local/share/virtualenvs/doc-gK3R_GBk/lib/python3.11/site-packages/docutils/nodes.py", line 984, in replace
    index = self.index(old)
            ^^^^^^^^^^^^^^^
  File "/home/bart/.local/share/virtualenvs/doc-gK3R_GBk/lib/python3.11/site-packages/docutils/nodes.py", line 737, in index
    return self.children.index(item, start, stop)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: <NeedRef: <emphasis...><emphasis...>> is not in list

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/bart/.local/share/virtualenvs/doc-gK3R_GBk/lib/python3.11/site-packages/sphinx/cmd/build.py", line 298, in build_main
    app.build(args.force_all, args.filenames)
  File "/home/bart/.local/share/virtualenvs/doc-gK3R_GBk/lib/python3.11/site-packages/sphinx/application.py", line 354, in build
    self.builder.build_update()
  File "/home/bart/.local/share/virtualenvs/doc-gK3R_GBk/lib/python3.11/site-packages/sphinx/builders/__init__.py", line 290, in build_update
    self.build(['__all__'], to_build)
  File "/home/bart/.local/share/virtualenvs/doc-gK3R_GBk/lib/python3.11/site-packages/sphinx/builders/__init__.py", line 363, in build
    self.write(docnames, list(updated_docnames), method)
  File "/home/bart/.local/share/virtualenvs/doc-gK3R_GBk/lib/python3.11/site-packages/sphinx/builders/__init__.py", line 571, in write
    self._write_serial(sorted(docnames))
  File "/home/bart/.local/share/virtualenvs/doc-gK3R_GBk/lib/python3.11/site-packages/sphinx/builders/__init__.py", line 578, in _write_serial
    doctree = self.env.get_and_resolve_doctree(docname, self)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bart/.local/share/virtualenvs/doc-gK3R_GBk/lib/python3.11/site-packages/sphinx/environment/__init__.py", line 635, in get_and_resolve_doctree
    self.apply_post_transforms(doctree, docname)
  File "/home/bart/.local/share/virtualenvs/doc-gK3R_GBk/lib/python3.11/site-packages/sphinx/environment/__init__.py", line 693, in apply_post_transforms
    self.events.emit('doctree-resolved', doctree, docname)
  File "/home/bart/.local/share/virtualenvs/doc-gK3R_GBk/lib/python3.11/site-packages/sphinx/events.py", line 107, in emit
    raise ExtensionError(__("Handler %r for event %r threw an exception") %
sphinx.errors.ExtensionError: Handler <function patched_process_creator.<locals>.patched_process_caller at 0x7f74a5639ee0> for event 'doctree-resolved' threw an exception (exception: <NeedRef: <emphasis...><emphasis...>> is not in list)

Btw, there seems to be another small mistake in the code too with this fragment: https://github.com/useblocks/sphinx-needs/blob/c27bc6449c0d2a106d953f4ec9e3051a9dc4500b/sphinx_needs/functions/functions.py#L141-L144

Shouldn't it be like this instead (the last line is not indented):

for child in node.children:
    new_child = find_and_replace_node_content(child, env, need)
    new_children.append(new_child)
node.children = new_children

[chrisjsewell]

Note this should be fixed now in current master (since all nodes created by directives are now set with source information), although I haven't yet checked specifically with sphinxcontrib.spelling