Fails to build PDF when 'req' directive contains nested directives

[danieleades]

I'm finding that while I can build HTML output (make html) the build fails with an extension error when using the PDF target (make latexpdf)

is this expected?

resolving references...
failed

Extension error:
Handler <function process_needextract at 0x7f77f5ac34c0> for event 'doctree-resolved' threw an exception
...

[danieleades]

the error occurs when the .. req:: directive contains other directives. In my case, i'm using both-

.. literalinclude:: some-file.feature
        :language: gherkin

and

.. uml:: some-diagram.plantuml

like so-

:orphan:

.. req:: Here is a requirement
    :id: REQ_001

    .. uml:: some-diagram.plantuml

    .. literalinclude:: some-file.feature
        :language: gherkin

both work fine when building HTML but fail when building PDF

[danieleades]

this is a high priority for us as-

• we use *.feature files to record requirements, where possible, as we use the same files to define acceptance tests
• we have a requirement to generate PDFs which capture the requirements and other elements of the documentation.

[danieleades]

on further investigation, it looks like any directive nested inside a .. req:: directive throws an error when attempting to build a PDF

[danwos]

In general it should work. There is no "HTML only" support planned, so every builder must work.

Can you please provide some more information about the used versions (python, sphinx, sphinx-needs).

Also there was an update in the last days (0.6.0), if you use this one, can you please check an older version. I will try to reproduce the error and fix it.

[danwos]

A quick check, if the sphinx-needs docs can be build for latexpdf was positive. No errors and a readable pdf-file. It's also strange that the errors seems to happen in process_needextract, which is not used by your example code.

[danieleades]

ah getting warmer!

my requirements are stored in separate files (1 requirement = 1 file) which are not included in any toctree. then i have a needtable or needextract in the index. If i don't have a needtable or needextract i don't get the error.

a minimal reproduction looks like this-

• index.rst
• requirements
  • index.rst
  • req.rst

where requirements/index.rst contains

Requirements
------------

.. needtable::

and requirements/req1.rst contains

:orphan:

.. req:: Here is a requirement
    :id: REQ_001

    .. note::

        this is a note

package versions are as follows-

python = "^3.8" Sphinx = "^3.4.3" sphinxcontrib-needs = "^0.6.0" sphinxcontrib-plantuml = "^0.19"

[danieleades]

i can push a reproduction to github if that's not clear enough?

[danwos]

Yeha, that would be helpful. I think I got your point. But before misinterpreting something, a shared code-base would be great :) Thanks!

[danieleades]

Yeha, that would be helpful. I think I got your point. But before misinterpreting something, a shared code-base would be great :) Thanks!

done - https://github.com/danieleades/sphinx-needs-issue-166

check the readme for build instructions

[danieleades]

as an aside, i notice that sphinxcontrib-plantuml is a dependency of sphinxcontrib-needs but isn't installed automagically by Poetry

that is, if it's not installed, building any target fails and complains that it couldn't be found. Does it need to be added to this project's dependencies?

[danieleades]

anything else I can do to help progress this one? Is there a way to generate extra debug output?

[danwos]

Sorry, I'm a little busy these days... Try to get it done asap, but not sure when this will be possible.

[danieleades]

the error occurs in needlist.py -> process_needlist(...)

this bit of code throws an exception-

# Create a reference
            if not need_info["hide"]:
                ref = nodes.reference('', '')
                ref['refdocname'] = need_info['docname']
                ref['refuri'] = app.builder.get_relative_uri(
                    fromdocname, need_info['docname'])
                ref['refuri'] += '#' + need_info['target_node']['refid']
                ref.append(title)
                para += ref
            else:
                para += title

the problematic line is

ref['refuri'] = app.builder.get_relative_uri(
                    fromdocname, need_info['docname'])

get_relatve_uri is defined in sphinx.builders.latex, and looks like this-

    def get_target_uri(self, docname: str, typ: str = None) -> str:
        if docname not in self.docnames:
            raise NoUri(docname, typ)
        else:
            return '%' + docname

    def get_relative_uri(self, from_: str, to: str, typ: str = None) -> str:
        # ignore source path
        return self.get_target_uri(to, typ)

this is raising a NoUri exception because docname not in self.docnames

does this indicate that this method is being called before the builder has been initialised with LaTeXBuilder.assemble_doctree?

[danieleades]

any movement on this one? This prevents us from adopting sphinx-needs, as we have a hard requirement that we must be able to generate PDFs.

[danwos]

Took (a late, sorry) look into it. And thanks for hunting this bug down to the responsible line of code.

For me it looks like that this is a bug/feature in the latexbuilder or in your general setup. Reason: Solution would be to have a accurate self.docnames var, which seems to be created and handled by the related builder. Nothing Sphinx-Needs has access to.

Maybe self.docnames is created by the two builders html and latexpdf in different ways. html: It contains all found .rst files latexpdf It contains rst-files referenced by totctree only. But this is just a guess.

I'm also afraid that other reference-related features would also not work: links, need_extract, filters, ... Everything were a link needs to be calculated.

So I would more search the problem in the latexpdf builder or use a workaround, like putting all files in a toctree. To keep it easy in your case, you maybe could use "globbing":

.. toctree::
   :glob:

   requirements/*

https://www.sphinx-doc.org/en/master/usage/restructuredtext/directives.html

I close this, but feel free to reopen if you see it different.

[danieleades]

I wonder if manually calling 'discover' would make any difference? https://www.sphinx-doc.org/en/master/extdev/projectapi.html#sphinx.project.Project

Thanks for investigating!

[davecap]

I'm encountering this issue while using rinohtype to generate PDFs. I'm wondering if anyone was able to get the above workaround to work?

[danwos]

Can you please tell me some more details about your project setup? Are there any problems, if you perform a html build? Are you having as well rst-files, which are not included into the project e.g. by using .. toctree::?

If the last point is not the case for your project, maybe you hit a new bug. In this case, may I ask you to create a new bug report? Also a small project example may be helpful, as I have no experiences with rinohtype.

[davecap]

I think my situation is quite similar to the above report. I did have several files which were not included in the toctree. When I added them I ran into further crashes unrelated to needs. My workaround is to generate single page html which works fine and print that to PDF either manually or via a small python script (pyppeteer).

I'm just curious if there's a cleaner way to get it working without adding extra stuff to the toctree that I don't want in my output file.

How do other people make PDFs of their docs?

[danieleades]

there is an API for manually discovering the documents in a tree. Since sphinx-needs relies on this information, I wonder if it should be using this API?

see https://github.com/useblocks/sphinxcontrib-needs/issues/166#issuecomment-834663093

[twodrops]

We generate single page HTML and then use https://wkhtmltopdf.org/ to generate PDF

[danwos]

Just checked it and discover() is simply available via app.project.discover() and returns a set.

I'm not sure if calling this command already does the trick, as I expect that the latexpdf builder is doing something similar already.

I reopen the issue to get a final solution or at least a satisfying answer.

Let me repeat, what we already know about the problem and a possible solution:

• The problem happens only, if there are rst-files, which are not referenced by a toctree. So officially they are not part of the docs (sphinx point of view)
• However, these rst-files get rendered as well. Because they are inside the source_folder and they got not excluded e.g. via exclude_pattern.
• Sphinx-Needs crashes, when it tries to create a reference to a need from one of these rst-files.
• The reason for the crash are missing entries in self.docnames
• app.project.discover() may be a solution to figure out all needed rst-files and maybe update self.docnames by hand.

I will need some time to make some tests. Any support is welcome :)

[danieleades]

looking at the code, it looks like discover does update the docnames attribute - https://github.com/sphinx-doc/sphinx/blob/fe47bf46cbb6a615b01d6c5dc9fdcac58c940f69/sphinx/project.py#L41

[davecap]

Great, that looks promising.

Just wondering, @twodrops which template do you use for your single page html?

[twodrops]

@davecap We created our own template with our company layout