ValueError: malformed node or string

[sobolevn]

Hi! I am running pydocstyle as a part of flake8-docstrings plugin which is a plugin for wemake-python-styleguide.

Original issue: https://github.com/wemake-services/wemake-python-styleguide/issues/579

One of my users reported this bug:

multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/usr/lib/python3.7/multiprocessing/pool.py", line 121, in worker
result = (True, func(*args, **kwds))
File "/usr/lib/python3.7/multiprocessing/pool.py", line 44, in mapstar
return list(map(*args))
File "/home/ilya/.virtualenvs/ds-backend/lib/python3.7/site-packages/flake8/checker.py", line 669, in _run_checks
return checker.run_checks()
File "/home/ilya/.virtualenvs/ds-backend/lib/python3.7/site-packages/flake8/checker.py", line 608, in run_checks
self.run_ast_checks()
File "/home/ilya/.virtualenvs/ds-backend/lib/python3.7/site-packages/flake8/checker.py", line 504, in run_ast_checks
for (line_number, offset, text, check) in runner:
File "/home/ilya/.virtualenvs/ds-backend/lib/python3.7/site-packages/flake8_docstrings.py", line 89, in run
for error in self._check_source():
File "/home/ilya/.virtualenvs/ds-backend/lib/python3.7/site-packages/flake8_docstrings.py", line 78, in _check_source
ignore_decorators=None,
File "/home/ilya/.virtualenvs/ds-backend/lib/python3.7/site-packages/pydocstyle/checker.py", line 75, in check_source
definition.docstring)
File "/home/ilya/.virtualenvs/ds-backend/lib/python3.7/site-packages/pydocstyle/checker.py", line 114, in check_docstring_missing
docstring and is_blank(ast.literal_eval(docstring))):
File "/usr/lib/python3.7/ast.py", line 91, in literal_eval
return _convert(node_or_string)
File "/usr/lib/python3.7/ast.py", line 90, in _convert
return _convert_signed_num(node)
File "/usr/lib/python3.7/ast.py", line 63, in _convert_signed_num
return _convert_num(node)
File "/usr/lib/python3.7/ast.py", line 55, in _convert_num
raise ValueError('malformed node or string: ' + repr(node))
ValueError: malformed node or string: <_ast.JoinedStr object at 0x7f1d8db27358>
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/ilya/.virtualenvs/ds-backend/bin/flake8", line 10, in 
sys.exit(main())
File "/home/ilya/.virtualenvs/ds-backend/lib/python3.7/site-packages/flake8/main/cli.py", line 18, in main
app.run(argv)
File "/home/ilya/.virtualenvs/ds-backend/lib/python3.7/site-packages/flake8/main/application.py", line 394, in run
self._run(argv)
File "/home/ilya/.virtualenvs/ds-backend/lib/python3.7/site-packages/flake8/main/application.py", line 382, in _run
self.run_checks()
File "/home/ilya/.virtualenvs/ds-backend/lib/python3.7/site-packages/flake8/main/application.py", line 301, in run_checks
self.file_checker_manager.run()
File "/home/ilya/.virtualenvs/ds-backend/lib/python3.7/site-packages/flake8/checker.py", line 328, in run
self.run_parallel()
File "/home/ilya/.virtualenvs/ds-backend/lib/python3.7/site-packages/flake8/checker.py", line 292, in run_parallel
for ret in pool_map:
File "/usr/lib/python3.7/multiprocessing/pool.py", line 354, in 
return (item for chunk in result for item in chunk)
File "/usr/lib/python3.7/multiprocessing/pool.py", line 748, in next
raise value
ValueError: malformed node or string: <_ast.JoinedStr object at 0x7f1d8db27358>

This happens here: https://github.com/PyCQA/pydocstyle/blob/master/src/pydocstyle/checker.py#L159

Reproduction code:

def function():
    f"""Template #{locals}"""
    ...

[sobolevn]

I would say that something like this should be done:

def safe_literal_eval(node) -> Optional[str]:
     try:
         return ast.literal_eval(node)
     except ValueError:
         return None  # happens when literal eval cannot process node

And then check that instead of ast.literal_eval this function is used and None value is checked.

[lordmauve]

Docstrings may not be f-strings. So actually the better fix is to specifically look for attempting to use a docstring as an f-string and issue a new error code.

[sobolevn]

@lordmauve agreed, new rules are always a good idea 🙂

[bra-fsn]

Any progress on this? It's quite puzzling to run into this, because there's no indication on what/where the error is, just a big fat python backtrace.

[s-weigand]

Just came across this error, since we autogenerate some docstrings with f-strings using a decorator. I could resolve the issue in our case by using --ignore-decorators.

pydocstyle <source_dir> --ignore-decorators=<name_of_decorator_gerating_docstring>

[sambhav]

Any progress on this? It's quite puzzling to run into this, because there's no indication on what/where the error is, just a big fat python backtrace.

I plan to pick it up and release a fix this weekend :)

[sambhav]

I have https://github.com/PyCQA/pydocstyle/pull/381 updated. Waiting for a review so that we can merge it.

[leungyukshing]

Is this bug fixed?

[vlad0337187]

still present