[Feature]: (API Calls | Integrations | System Calls) Need to be Supported in the PromptStep in Task

[HamadaSalhab]

trafficstars

🔖 Feature description

The function tools are already supported in the PromptStep.

Here's how it's implemented:

• Tools are being formatted here to the format that's acceptable by OpenAI before being sent with the prompt.
• If the model decides to call the tool, the finish_reason inside the output of the StepOutcome will be set to tool_calls. We're catching that here.
• In the function tool type, we're entering a wait_for_input step because the developer would have to execute the tool on his/her end and send the response back to us (this is not needed for this issue as the other 3 tool types should run without the interference of the developer).
• After that, we're sending the results back as an OpenAI-accepted format (the dev sends them formatted already), then we resume the execution.

For the case of the 3 additional tools, we would have to run them on our end. This includes:

1- Formatting the tools to an OpenAI-accepted tool format. 2- When catching the tool_calls, we should get the tool type from the tool name (because the type would be returned as function from the model, but it might be something else). 3- We would turn the tool call from the format given to us to the format accepted by the tool type depending on what type this tool is. 4- Run the execution activity depending on the tool type. 5- Format the output of the activity to the OpenAI-accepted format for the tool_call response.

Note: We shouldn't enter a wait_for_input step in the case of the other 3 tool types.

Let me know if there's any questions.

🎤 Why is this feature needed ?

No response

✌️ How do you aim to achieve this?

No response

🔄️ Additional Information

No response

👀 Have you spent some time to check if this feature request has been raised before?

• [X] I checked and didn't find similar issue

Are you willing to submit PR?

None