Pynguin encounters error when running against webpy

[JamZYu]

We are evaluating our project pyster(also a Python test generator) with Pynguin and found that Pynguin throws the following exception when running against webpy:

pynguin --algorithm WSPY --project_path public_repo/webpy --module_name web.db --output_path ./public_repo/webpy --budget 600
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/networkx/classes/digraph.py", line 805, in successors
    return iter(self._succ[n])
KeyError: ProgramGraphNode(index=1, basic_block=[<FOR_ITER arg=[<LOAD_CONST arg=ExecutionTracer lineno=369>, <LOAD_METHOD arg='executed_bool_predicate' lineno=369>, <LOAD_CONST arg=False lineno=369>, <LOAD_CONST arg=45 lineno=369>, <CALL_METHOD arg=2 lineno=369>, <POP_TOP lineno=369>, <JUMP_ABSOLUTE arg=[<LOAD_GLOBAL arg='SQLQuery' lineno=375>, <LOAD_METHOD arg='join' lineno=375>, <LOAD_FAST arg='result' lineno=375>, <LOAD_CONST arg='' lineno=375>, <CALL_METHOD arg=2 lineno=375>, <RETURN_VALUE lineno=375>] lineno=369>] lineno=369>])

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/bin/pynguin", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.8/site-packages/pynguin/cli.py", line 136, in main
    return generator.run().value
  File "/usr/local/lib/python3.8/site-packages/pynguin/generator.py", line 101, in run
    return self._run()
  File "/usr/local/lib/python3.8/site-packages/pynguin/generator.py", line 224, in _run
    if (setup_result := self._setup_and_check()) is None:
  File "/usr/local/lib/python3.8/site-packages/pynguin/generator.py", line 186, in _setup_and_check
    if not self._load_sut():
  File "/usr/local/lib/python3.8/site-packages/pynguin/generator.py", line 141, in _load_sut
    importlib.import_module(config.INSTANCE.module_name)
  File "/usr/local/Cellar/[email protected]/3.8.6/Frameworks/Python.framework/Versions/3.8/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "/usr/local/lib/python3.8/site-packages/pynguin/instrumentation/machinery.py", line 32, in exec_module
    super().exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 779, in exec_module
  File "/usr/local/lib/python3.8/site-packages/pynguin/instrumentation/machinery.py", line 49, in get_code
    return instrumentation.instrument_module(to_instrument)
  File "/usr/local/lib/python3.8/site-packages/pynguin/instrumentation/branch_distance.py", line 326, in instrument_module
    return self._instrument_code_recursive(module_code)
  File "/usr/local/lib/python3.8/site-packages/pynguin/instrumentation/branch_distance.py", line 114, in _instrument_code_recursive
    return self._instrument_inner_code_objects(
  File "/usr/local/lib/python3.8/site-packages/pynguin/instrumentation/branch_distance.py", line 75, in _instrument_inner_code_objects
    self._instrument_code_recursive(
  File "/usr/local/lib/python3.8/site-packages/pynguin/instrumentation/branch_distance.py", line 113, in _instrument_code_recursive
    self._instrument_cfg(cfg, code_object_id)
  File "/usr/local/lib/python3.8/site-packages/pynguin/instrumentation/branch_distance.py", line 130, in _instrument_cfg
    predicate_id = self._instrument_node(
  File "/usr/local/lib/python3.8/site-packages/pynguin/instrumentation/branch_distance.py", line 172, in _instrument_node
    predicate_id = self._instrument_for_loop(
  File "/usr/local/lib/python3.8/site-packages/pynguin/instrumentation/branch_distance.py", line 423, in _instrument_for_loop
    for successor in dominator_tree.get_transitive_successors(node):
  File "/usr/local/lib/python3.8/site-packages/pynguin/analyses/controlflow/programgraph.py", line 188, in get_transitive_successors
    return self._get_transitive_successors(node, set())
  File "/usr/local/lib/python3.8/site-packages/pynguin/analyses/controlflow/programgraph.py", line 192, in _get_transitive_successors
    for successor_node in self.get_successors(node):
  File "/usr/local/lib/python3.8/site-packages/pynguin/analyses/controlflow/programgraph.py", line 129, in get_successors
    for successor in self._graph.successors(node):
  File "/usr/local/lib/python3.8/site-packages/networkx/classes/digraph.py", line 807, in successors
    raise NetworkXError(f"The node {n} is not in the digraph.") from e
networkx.exception.NetworkXError: The node ProgramGraphNode(1) is not in the digraph.

[Wooza]

Hi @JamZYu,

thank you for using Pynguin in your evaluation and for reporting this bug. After looking into to the stacktrace and webpy's source, I think that the reparam method in the db module causes this error. The method contains dead code (lines 365-375), which in turn causes problems when Pynguin tries to instrument the method's bytecode. We might make Pynguin more resilient against dead code in future releases.

A simple fix to continue your evaluation is to remove the dead code from the module and everything should work again. You should also think about opening a ticket with webpy as this dead code seems to be an oversight on their side.

@stephanlukasczyk FYI.

[MarineChap]

hello, I am popping on this issue because I am having the same type of errors very often. I don't understand how you saw the problem was dead code (lines 365-375). Below the interesting part of the error, how do you decrypted the error to know more of where is the problem here exactly ?

KeyError: ProgramGraphNode(index=22, basic_block=[<JUMP_FORWARD arg=[<LOAD_FAST arg='next_visit' lineno=206>, <LOAD_FAST arg='C' lineno=206>, <COMPARE_OP arg=<Compare.EQ: 2> lineno=206>, <POP_JUMP_IF_FALSE arg=[<LOAD_FAST arg='C' lineno=214>, <STORE_FAST arg='next_arm' lineno=214>] lineno=206>] lineno=201>])

Thanks

[stephanlukasczyk]

Hi @MarineChap ,

I guess the conclusion of dead code was done by looking at the source code, not only on the byte code/the program-graph nodes generated by Pynguin. When you look at the code fragment, linked by @Wooza in his answer, it reads like:

def reparam(string_, dictionary):
    """
    Takes a string and a dictionary and interpolates the string
    using values from the dictionary. Returns an `SQLQuery` for the result.
        >>> reparam("s = $s", dict(s=True))
        <sql: "s = 't'">
        >>> reparam("s IN $s", dict(s=[1, 2]))
        <sql: 's IN (1, 2)'>
    """
    return SafeEval().safeeval(string_, dictionary)

    dictionary = dictionary.copy()  # eval mucks with it
    # disable builtins to avoid risk for remote code execution.
    dictionary["__builtins__"] = object()
    result = []
    for live, chunk in _interpolate(string_):
        if live:
            v = eval(chunk, dictionary)
            result.append(sqlquote(v))
        else:
            result.append(chunk)
    return SQLQuery.join(result, "")

Obviously, everything after the first return statement cannot be reached by the execution, since return leaves the method; thus, everything following that line is dead code, since it can never be reached.

Pynguin is not robust enough, which is something we will deal with in a future release. Nevertheless, I suggest that this dead code should also be removed from the webpy project. Or at least the code segment should be carefully revisited to avoid any unexpected behaviour.

[MarineChap]

Ah okay... There is nothing as obvious as this in my code, that's why I asked this. I guess it is likely due to the use of 'yield' at the end of the methods concerned by the errors. Thanks for the answer.

[stephanlukasczyk]

You're welcome. Would you mind to share the code you're trying to run Pynguin on? From what you wrote it sounds like you want to generate tests for a generator method? Please note that Pynguin is in a quite early stage thus this might be functionality we need to add to it. Hence, it would be great to also have some real-world examples to see whether a future implementation can deal with them.

[MarineChap]

Sorry I forget to answer this. Yes definitely the test of a method with generator within was maybe too much. More reasons to have a way to specify don't do test on this method (see my other open issue) ;)

Code example: https://bitbucket.org/kloostermannerflab/fklab-controller-lab/src/master/hive/implementation/w_alter_task.py

Also, while I am here, I have some server communication in this project, is it possible to mock some part of the code with Pynguin ?

Good luck for your next development.