promptflow icon indicating copy to clipboard operation
promptflow copied to clipboard
verified publisher icon microsoft

Execute multi-LLM promptflow with local runtime, flow execution failure - low timeout/retry settings

Open corticalstack opened this issue 1 year ago • 4 comments

Describe the bug

  • Running a promptflow of several LLM nodes - each prompts Azure GPT-4 model deployment
  • Most nodes complete (showing good connection to model endpoint). A node fails, causing the flow execution to fail
  • The GPT-4 pay-as-you-go service is slow. When running this series of prompts using vanlilla code, many retry attempts are necessary before sufficient bandwidth and tokens quoata available to complete
  • Running this promptflow locally on dev environment via the VS Code extension (i.e. not on an AML runtime compute instance), prompt flow view, expaning the flows, finding my flow, then clicking Test. Perhaps this timeput issue/limitation will always be present in Test mode, but not when running batch?

How To Reproduce the bug

  • Run a flow with several more complex prompts with an Azure GPT-4 model deployement endpoint. From VS Code promptflow "test" feature.

Expected behavior Flow should complete, even if there has to be many retries and long execution time. If there anywhere, for example in the promptflow configuration, to customize timeout/retry?

Screenshots

Running Information(please complete the following information):

promptflow 1.6.0       
promptflow-tools 1.3.0 
Python 3.11.4

2024-03-11 15:48:23 +0100   54475 execution.flow     INFO     Node accommodation completes.
2024-03-11 15:48:23 +0100   54475 execution.flow     INFO     Executing node accommodation_sanitizer. node run id: 5ac699a0-ced6-46c2-b0b1-1345860f8a30_accommodation_sanitizer_0
2024-03-11 15:48:23 +0100   54475 execution.flow     INFO     Node accommodation_sanitizer completes.
2024-03-11 15:48:56 +0100   54475 execution          WARNING  local_cuisine in line 0 has been running for 300 seconds, stacktrace of thread 140659658823360:
  File "/root/developments/lhg/oneai/lhg-oneai-destination-finder-batch-gen-content-promptflow/.venv/lib/python3.11/site-packages/promptflow/tools/common.py", line 207, in wrapper
    return func(*args, **kwargs)
  File "/root/developments/lhg/oneai/lhg-oneai-destination-finder-batch-gen-content-promptflow/.venv/lib/python3.11/site-packages/promptflow/tools/aoai.py", line 157, in chat
    completion = self._client.chat.completions.create(**params)
  File "/root/developments/lhg/oneai/lhg-oneai-destination-finder-batch-gen-content-promptflow/.venv/lib/python3.11/site-packages/promptflow/_core/openai_injector.py", line 88, in wrapper
    return f(*args, **kwargs)
  File "/root/developments/lhg/oneai/lhg-oneai-destination-finder-batch-gen-content-promptflow/.venv/lib/python3.11/site-packages/promptflow/_core/tracer.py", line 528, in wrapped
    output = func(*args, **kwargs)
  File "/root/developments/lhg/oneai/lhg-oneai-destination-finder-batch-gen-content-promptflow/.venv/lib/python3.11/site-packages/openai/_utils/_utils.py", line 275, in wrapper
    return func(*args, **kwargs)
  File "/root/developments/lhg/oneai/lhg-oneai-destination-finder-batch-gen-content-promptflow/.venv/lib/python3.11/site-packages/openai/resources/chat/completions.py", line 663, in create
    return self._post(
  File "/root/developments/lhg/oneai/lhg-oneai-destination-finder-batch-gen-content-promptflow/.venv/lib/python3.11/site-packages/openai/_base_client.py", line 1200, in post
    return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))
  File "/root/developments/lhg/oneai/lhg-oneai-destination-finder-batch-gen-content-promptflow/.venv/lib/python3.11/site-packages/openai/_base_client.py", line 889, in request
    return self._request(
  File "/root/developments/lhg/oneai/lhg-oneai-destination-finder-batch-gen-content-promptflow/.venv/lib/python3.11/site-packages/openai/_base_client.py", line 918, in _request
    response = self._client.send(
  File "/root/developments/lhg/oneai/lhg-oneai-destination-finder-batch-gen-content-promptflow/.venv/lib/python3.11/site-packages/httpx/_client.py", line 914, in send
    response = self._send_handling_auth(
  File "/root/developments/lhg/oneai/lhg-oneai-destination-finder-batch-gen-content-promptflow/.venv/lib/python3.11/site-packages/httpx/_client.py", line 942, in _send_handling_auth
    response = self._send_handling_redirects(
  File "/root/developments/lhg/oneai/lhg-oneai-destination-finder-batch-gen-content-promptflow/.venv/lib/python3.11/site-packages/httpx/_client.py", line 979, in _send_handling_redirects
    response = self._send_single_request(request)
  File "/root/developments/lhg/oneai/lhg-oneai-destination-finder-batch-gen-content-promptflow/.venv/lib/python3.11/site-packages/httpx/_client.py", line 1015, in _send_single_request
    response = transport.handle_request(request)
  File "/root/developments/lhg/oneai/lhg-oneai-destination-finder-batch-gen-content-promptflow/.venv/lib/python3.11/site-packages/httpx/_transports/default.py", line 233, in handle_request
    resp = self._pool.handle_request(req)
  File "/root/developments/lhg/oneai/lhg-oneai-destination-finder-batch-gen-content-promptflow/.venv/lib/python3.11/site-packages/httpcore/_sync/connection_pool.py", line 196, in handle_request
    response = connection.handle_request(
  File "/root/developments/lhg/oneai/lhg-oneai-destination-finder-batch-gen-content-promptflow/.venv/lib/python3.11/site-packages/httpcore/_sync/connection.py", line 101, in handle_request
    return self._connection.handle_request(request)
  File "/root/developments/lhg/oneai/lhg-oneai-destination-finder-batch-gen-content-promptflow/.venv/lib/python3.11/site-packages/httpcore/_sync/http11.py", line 113, in handle_request
    ) = self._receive_response_headers(**kwargs)
  File "/root/developments/lhg/oneai/lhg-oneai-destination-finder-batch-gen-content-promptflow/.venv/lib/python3.11/site-packages/httpcore/_sync/http11.py", line 186, in _receive_response_headers
    event = self._receive_event(timeout=timeout)
  File "/root/developments/lhg/oneai/lhg-oneai-destination-finder-batch-gen-content-promptflow/.venv/lib/python3.11/site-packages/httpcore/_sync/http11.py", line 224, in _receive_event
    data = self._network_stream.read(
  File "/root/developments/lhg/oneai/lhg-oneai-destination-finder-batch-gen-content-promptflow/.venv/lib/python3.11/site-packages/httpcore/_backends/sync.py", line 126, in read
    return self._sock.recv(max_bytes)
  File "/usr/lib/python3.11/ssl.py", line 1263, in recv
    return self.read(buflen)
  File "/usr/lib/python3.11/ssl.py", line 1136, in read
    return self._sslobj.read(len)

2024-03-11 15:49:05 +0100   54475 execution          WARNING  [local_cuisine in line 0 (index starts from 0)] stderr> Exception occurs: APIConnectionError: Connection error.
2024-03-11 15:49:05 +0100   54475 execution          ERROR    Node local_cuisine in line 0 failed. Exception: OpenAI API hits APIConnectionError: Connection error. [Error reference: https://platform.openai.com/docs/guides/error-codes/api-errors].
Traceback (most recent call last):
  File "/root/developments/lhg/oneai/lhg-oneai-destination-finder-batch-gen-content-promptflow/.venv/lib/python3.11/site-packages/httpx/_transports/default.py", line 69, in map_httpcore_exceptions
    yield
  File "/root/developments/lhg/oneai/lhg-oneai-destination-finder-batch-gen-content-promptflow/.venv/lib/python3.11/site-packages/httpx/_transports/default.py", line 233, in handle_request
    resp = self._pool.handle_request(req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/developments/lhg/oneai/lhg-oneai-destination-finder-batch-gen-content-promptflow/.venv/lib/python3.11/site-packages/httpcore/_sync/connection_pool.py", line 216, in handle_request
    raise exc from None
  File "/root/developments/lhg/oneai/lhg-oneai-destination-finder-batch-gen-content-promptflow/.venv/lib/python3.11/site-packages/httpcore/_sync/connection_pool.py", line 196, in handle_request
    response = connection.handle_request(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/developments/lhg/oneai/lhg-oneai-destination-finder-batch-gen-content-promptflow/.venv/lib/python3.11/site-packages/httpcore/_sync/connection.py", line 101, in handle_request
    return self._connection.handle_request(request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/developments/lhg/oneai/lhg-oneai-destination-finder-batch-gen-content-promptflow/.venv/lib/python3.11/site-packages/httpcore/_sync/http11.py", line 143, in handle_request
    raise exc
  File "/root/developments/lhg/oneai/lhg-oneai-destination-finder-batch-gen-content-promptflow/.venv/lib/python3.11/site-packages/httpcore/_sync/http11.py", line 113, in handle_request
    ) = self._receive_response_headers(**kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/developments/lhg/oneai/lhg-oneai-destination-finder-batch-gen-content-promptflow/.venv/lib/python3.11/site-packages/httpcore/_sync/http11.py", line 186, in _receive_response_headers
    event = self._receive_event(timeout=timeout)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/developments/lhg/oneai/lhg-oneai-destination-finder-batch-gen-content-promptflow/.venv/lib/python3.11/site-packages/httpcore/_sync/http11.py", line 238, in _receive_event
    raise RemoteProtocolError(msg)
httpcore.RemoteProtocolError: Server disconnected without sending a response.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/root/developments/lhg/oneai/lhg-oneai-destination-finder-batch-gen-content-promptflow/.venv/lib/python3.11/site-packages/openai/_base_client.py", line 918, in _request
    response = self._client.send(
               ^^^^^^^^^^^^^^^^^^
  File "/root/developments/lhg/oneai/lhg-oneai-destination-finder-batch-gen-content-promptflow/.venv/lib/python3.11/site-packages/httpx/_client.py", line 914, in send
    response = self._send_handling_auth(
               ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/developments/lhg/oneai/lhg-oneai-destination-finder-batch-gen-content-promptflow/.venv/lib/python3.11/site-packages/httpx/_client.py", line 942, in _send_handling_auth
    response = self._send_handling_redirects(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/developments/lhg/oneai/lhg-oneai-destination-finder-batch-gen-content-promptflow/.venv/lib/python3.11/site-packages/httpx/_client.py", line 979, in _send_handling_redirects
    response = self._send_single_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/developments/lhg/oneai/lhg-oneai-destination-finder-batch-gen-content-promptflow/.venv/lib/python3.11/site-packages/httpx/_client.py", line 1015, in _send_single_request
    response = transport.handle_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/developments/lhg/oneai/lhg-oneai-destination-finder-batch-gen-content-promptflow/.venv/lib/python3.11/site-packages/httpx/_transports/default.py", line 232, in handle_request
    with map_httpcore_exceptions():
  File "/usr/lib/python3.11/contextlib.py", line 155, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/root/developments/lhg/oneai/lhg-oneai-destination-finder-batch-gen-content-promptflow/.venv/lib/python3.11/site-packages/httpx/_transports/default.py", line 86, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.RemoteProtocolError: Server disconnected without sending a response.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/root/developments/lhg/oneai/lhg-oneai-destination-finder-batch-gen-content-promptflow/.venv/lib/python3.11/site-packages/promptflow/tools/common.py", line 207, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/root/developments/lhg/oneai/lhg-oneai-destination-finder-batch-gen-content-promptflow/.venv/lib/python3.11/site-packages/promptflow/tools/aoai.py", line 157, in chat
    completion = self._client.chat.completions.create(**params)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/developments/lhg/oneai/lhg-oneai-destination-finder-batch-gen-content-promptflow/.venv/lib/python3.11/site-packages/promptflow/_core/openai_injector.py", line 88, in wrapper
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
  File "/root/developments/lhg/oneai/lhg-oneai-destination-finder-batch-gen-content-promptflow/.venv/lib/python3.11/site-packages/promptflow/_core/tracer.py", line 528, in wrapped
    output = func(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^
  File "/root/developments/lhg/oneai/lhg-oneai-destination-finder-batch-gen-content-promptflow/.venv/lib/python3.11/site-packages/openai/_utils/_utils.py", line 275, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/root/developments/lhg/oneai/lhg-oneai-destination-finder-batch-gen-content-promptflow/.venv/lib/python3.11/site-packages/openai/resources/chat/completions.py", line 663, in create
    return self._post(
           ^^^^^^^^^^^
  File "/root/developments/lhg/oneai/lhg-oneai-destination-finder-batch-gen-content-promptflow/.venv/lib/python3.11/site-packages/openai/_base_client.py", line 1200, in post
    return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/developments/lhg/oneai/lhg-oneai-destination-finder-batch-gen-content-promptflow/.venv/lib/python3.11/site-packages/openai/_base_client.py", line 889, in request
    return self._request(
           ^^^^^^^^^^^^^^
  File "/root/developments/lhg/oneai/lhg-oneai-destination-finder-batch-gen-content-promptflow/.venv/lib/python3.11/site-packages/openai/_base_client.py", line 952, in _request
    raise APIConnectionError(request=request) from err
openai.APIConnectionError: Connection error.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/developments/lhg/oneai/lhg-oneai-destination-finder-batch-gen-content-promptflow/.venv/lib/python3.11/site-packages/promptflow/_core/flow_execution_context.py", line 88, in invoke_tool
    result = self._invoke_tool_with_timer(node, f, kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/developments/lhg/oneai/lhg-oneai-destination-finder-batch-gen-content-promptflow/.venv/lib/python3.11/site-packages/promptflow/_core/flow_execution_context.py", line 196, in _invoke_tool_with_timer
    raise e
  File "/root/developments/lhg/oneai/lhg-oneai-destination-finder-batch-gen-content-promptflow/.venv/lib/python3.11/site-packages/promptflow/_core/flow_execution_context.py", line 190, in _invoke_tool_with_timer
    return f(**kwargs)
           ^^^^^^^^^^^
  File "/root/developments/lhg/oneai/lhg-oneai-destination-finder-batch-gen-content-promptflow/.venv/lib/python3.11/site-packages/promptflow/_core/tracer.py", line 528, in wrapped
    output = func(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^
  File "/root/developments/lhg/oneai/lhg-oneai-destination-finder-batch-gen-content-promptflow/.venv/lib/python3.11/site-packages/promptflow/tools/common.py", line 217, in wrapper
    raise WrappedOpenAIError(e)
promptflow.tools.exception.WrappedOpenAIError: OpenAI API hits APIConnectionError: Connection error. [Error reference: https://platform.openai.com/docs/guides/error-codes/api-errors]
2024-03-11 15:49:05 +0100   54475 execution.flow     ERROR    Flow execution has failed. Cancelling all running nodes: local_cuisine.
pf.flow.test failed with UserErrorException: Exception: OpenAI API hits APIConnectionError: Connection error. [Error reference: https://platform.openai.com/docs/guides/error-codes/api-errors]

corticalstack avatar Mar 11 '24 15:03 corticalstack

According to the promptflow troubleshooting guide here:

"..the behavior is expected because your prompt might generate a long response that takes longer than the interactive mode's upper threshold. In this situation, we recommend trying Bulk test because this mode doesn't have a timeout setting."

So I created a 1-line .jsonl and mapped data to input node, and ran as batch. Still timed out after 600 seconds:

File "/usr/lib/python3.11/ssl.py", line 1263, in recv
  return self.read(buflen)
File "/usr/lib/python3.11/ssl.py", line 1136, in read
  return self._sslobj.read(len)

2024-03-11 17:26:25 +0100   72627 execution          WARNING  [sightseeing in line 0 (index starts from 0)] stderr> Exception occurs: APITimeoutError: Request timed out.
2024-03-11 17:26:25 +0100   72627 execution          WARNING  [sightseeing in line 0 (index starts from 0)] stderr> APITimeoutError #0, but no Retry-After header, Back off 3 seconds for retry.
2024-03-11 17:26:29 +0100   72446 execution          ERROR    1/1 flow run failed, indexes: [0], exception of index 0: Line 0 execution timeout for exceeding 600 seconds

corticalstack avatar Mar 11 '24 16:03 corticalstack

Hi, we're sending this friendly reminder because we haven't heard back from you in 30 days. We need more information about this issue to help address it. Please be sure to give us your input. If we don't hear back from you within 7 days of this comment, the issue will be automatically closed. Thank you!

github-actions[bot] avatar Apr 11 '24 21:04 github-actions[bot]

Any update?

corticalstack avatar Apr 17 '24 18:04 corticalstack

Hi, we're sending this friendly reminder because we haven't heard back from you in 30 days. We need more information about this issue to help address it. Please be sure to give us your input. If we don't hear back from you within 7 days of this comment, the issue will be automatically closed. Thank you!

github-actions[bot] avatar May 17 '24 21:05 github-actions[bot]

bump. curious on this as well

lbalceda avatar May 22 '24 06:05 lbalceda

Hi, we're sending this friendly reminder because we haven't heard back from you in 30 days. We need more information about this issue to help address it. Please be sure to give us your input. If we don't hear back from you within 7 days of this comment, the issue will be automatically closed. Thank you!

github-actions[bot] avatar Jun 21 '24 21:06 github-actions[bot]