[BUG] - pf.flow.test failed with PythonLoadError: Failed to load python module from file ... (NameError) name 'Document' is not defined

[aaronjolson]

Describe the bug I have a large promptflow project. I have been building it out using the VScode promptflow extension. Today I copied over some nodes from a separate project into the first one. I copied them over by modifying 3 files in MyProject/flow/.promptflow/

• flow.layout.json
• flow.tools.json
• flow.uihint.json

And this one in the MyProject/flow dir

• flow.dag.yaml

I also copied over all associated tool scripts and dependencies. From inside of the files, nothing appears wrong. after restarting VSCode, then I go to promtpoflow -> flow -> open to bring up the UI, everything also checks out according to the UI, no errors are shown there.

When I run the test command

python -m promptflow._cli._pf.entry flow test --flow path/to/flow --user-agent "prompt-flow-extension/1.16.1 (darwin; arm64) VSCode/1.88.1"

however, I get the following error, pf.flow.test failed with PythonLoadError: Failed to load python module from file '/full/path/to/tool/file.py' : (NameError) name 'Document' is not defined

So, it appears there is a disconnect between the configuration of the UI and however these tool files get linked together under the hood.

How To Reproduce the bug Steps to reproduce the behavior, how frequent can you experience the bug: This error can be consistently reproduced 2 different ways, either by attempting to add new nodes to an existing system through means that are not the "add new node" button in the UI, or attempting to use the "Regenerate with advanced options" button on an existing/working tool.

Method 1 modify 3 files in ./flow/.promptflow/ .

• flow.layout.json
• flow.tools.json
• flow.uihint.json

And this one in the ./flow dir

• flow.dag.yaml

Modify them correctly so the UI does not complain. The easiest way to do this is to copy existing nodes from another project. Then restart VSCode so the project reloads.

Method 2 It is also possible to re-produce in the UI an alternate way. In the extension UI, DAG view, select a tool, click on the "Regenerate with advanced options". ./flow/ for the first option ./flow/venv/bin/python for the second option (it varies depending on how you have your system set up) It will then attempt to regenerate the tool metadata, and fail. I suspect these two errors are related. There are issues between the UI and metadata. If you restart VSCode at this point, the UI errors will disappear, but the tool will remain broken when attempting to run the pf test commands and generating this error

pf.flow.test failed with PythonLoadError: Failed to load python module from file '/full/path/to/tool/file.py' : (NameError) name 'Document' is not defined

when running

python -m promptflow._cli._pf.entry flow test --flow path/to/flow --user-agent "prompt-flow-extension/1.16.1 (darwin; arm64) VSCode/1.88.1"

Expected behavior I expect the flow to run without throwing a 'Document' is not defined error. If there is some kind of master config file somewhere, there should be an easy way to rebuild it from the UI.

Screenshots If applicable, add screenshots to help explain your problem.

Running Information(please complete the following information):

• Promptflow Package Version using pf -v: [e.g. 0.0.102309906]
• Operating System: [e.g. Ubuntu 20.04, Windows 11]
• Python Version using python --version: [e.g. python==3.10.12]

promptflow			 1.8.0
promptflow-tracing		 1.8.0
promptflow-core			 1.8.0
promptflow-devkit		 1.8.0

MacOS Sonoma 14.4.1

Python 3.10.6

Additional context Add any other context about the problem here.

[lalala123123]

Hi @aaronjolson , when executing pf test, the working directory will be switched to the flow folder. Is the file path in the error log is the user code? If so, can you confirm whether Doctumentmodel in the script in the error log is used correctly?

[aaronjolson]

@lalala123123 - The working directory is supposed to be the flow folder, so that switch should not be an issue. The code and tools all live in the flow directory. The project structure looks like

ProjectName/
    flow/
        tools/
           tool_foo.py
           tool_bar.py

Again, in the UI, I see no errors, everything looks correct until I try to run the test command. Can you provide more information about the Document model you are referring to?

[aaronjolson]

A little more detail on the error

(venv) ➜  flow /Users/me/myproject/flow/venv/bin/python -m promptflow._cli._pf.entry flow test --flow /Users/me/myproject/flow --user-agent "prompt-flow-extension/1.16.1 (darwin; arm64) VSCode/1.88.1"
[2024-04-17 10:57:12,457][promptflow][DEBUG] - preparing home directory with default value.
[2024-04-17 10:57:13,101][promptflow][DEBUG] - PFClient init with kwargs: {'config': {}}
[2024-04-17 10:57:13,196][promptflow.contracts.flow][DEBUG] - Try loading connection names for node results_aggregation.
[2024-04-17 10:57:13,197][promptflow.contracts.flow][DEBUG] - Node results_aggregation doesn't reference any connection.
[2024-04-17 10:57:13,197][promptflow.contracts.flow][DEBUG] - Try loading connection names for node intent_extractor.
[2024-04-17 10:57:13,676][promptflow][DEBUG] - Command Namespace(version=False, action='flow', flow='/Users/me/myproject/flow', node=None, variant=None, interactive=False, input=None, inputs=None, environment_variables=None, multi_modal=False, ui=False, config=None, detail=None, skip_open_browser=False, verbose=False, debug=False, user_agent='prompt-flow-extension/1.16.1 (darwin; arm64) VSCode/1.88.1', sub_action='test') execute failed. Failed to load python module from file '/Users/me/myproject/flow/tools/intent_extractor.py': (NameError) name 'Document' is not defined
[2024-04-17 10:57:13,676][promptflow][INFO] - Command ran in 0.672 seconds (init: 0.097, invoke: 0.575)

[github-actions[bot]]

Hi, we're sending this friendly reminder because we haven't heard back from you in 30 days. We need more information about this issue to help address it. Please be sure to give us your input. If we don't hear back from you within 7 days of this comment, the issue will be automatically closed. Thank you!