[BUG] promptflow batch run not start execution when calling pf.run() in a notebook from another notebook

[Maggieli99]

Describe the bug I was trying to do a promptflow batch run using the code below:

from promptflow import PFClient
pf = PFClient()
batch_run = pf.run(
        flow = flow_directory,
        data = jsonl_data,
        stream = True,
        column_mapping = {column mappings}

The code is written in notebook A.ipynb, and I need to run notebook A.ipynb in notebook B.ipynb using the code %run ./A.ipynb. When I tried to run B.ipynb, as it proceeded to run pf.run(), I got a logs.txt as usual which shows the log for the promptflow batch run. Usually in logs.txt, I will first see 3 lines like

2024-07-23 18:37:31 -0700 44536 execution.bulk INFO Current thread is not main thread, skip signal handler registration in BatchEngine. 2024-07-23 18:37:31 -0700 44536 execution.bulk INFO Current system's available memory is 2568.66015625MB, memory consumption of current process is 215.2890625MB, estimated available worker count is 2568.66015625/215.2890625 = 11 2024-07-23 18:37:31 -0700 44536 execution.bulk INFO Set process count to 3 by taking the minimum value among the factors of {'default_worker_count': 4, 'row_count': 3, 'estimated_worker_count_based_on_memory_usage': 11}.

And then see each line starting execution like

2024-07-23 18:37:36 -0700 44536 execution.bulk INFO Process name(SpawnProcess-39)-Process id(49008)-Line number(0) start execution. ...

But this time, in logs.txt, I only see the first 3 lines, and I never see a single line start execution even after running an hour. I do not see error messages, it's just the batch run never start execution and runs forever.

How To Reproduce the bug Steps to reproduce the behavior, how frequent can you experience the bug:

1. Prepare a promptflow and a dataset for batch run.
2. Run the promptflow in a notebook A using pf.run() as described in the above section.
3. Then in another notebook B, using %run ./A.ipynb to run notebook A.
4. You will notice the run ever ends. If you go to the logs.txt to this batch run, you will only see the first 3 lines as I described in the above section, and will not see logs about a line starting execution.

Expected behavior The expected behavior is that the batch run finished in a reasonable amount of time. And during the batch run, in the logs.txt, we can see logs popping up about a line starting or ending execution.

Running Information(please complete the following information):

• Promptflow Package Version using pf -v: 1.11.0
• Operating System: Windows 11
• Python Version using python --version: e.g. python==3.12.3

[Hhhilulu]

Hi, @Maggieli99, I followed the reproduction steps you provided to run the batch, but the issue did not reoccur.

Here is my code: simple_flow_with_python_tool.zip