Assistant: `getPlot` tool not working with openai(-compatible) providers

[sharon-wang]

System details:

Positron and OS details:

Positron Version: 2025.12.0 (user setup) build 114 Code - OSS Version: 1.106.0 Commit: d49b1e96dfbb289f3982c5f861b75e442a1484dc Date: 2025-11-21T07:35:34.718Z Electron: 37.7.0 Chromium: 138.0.7204.251 Node.js: 22.20.0 V8: 13.8.258.32-electron.0 OS: Windows_NT x64 10.0.26200

Session details:

Not working with OpenAI and OpenAI-compatible providers with any model

Describe the issue:

This issue seems to have occurred with the 1.106 upstream merge. Models from the Anthropic provider continue to work correctly, but OpenAI models have lost the ability to use the getPlot tool.

Example Case

OpenAI → GPT-4.1

Error Messages

When using OpenAI → GPT-4.1, the following error appears:

Steps to reproduce the issue:

1. Create a plot in Positron
2. Ask the Assistant (using OpenAI provider) about the plot
3. Observe that the getPlot tool is not utilized and error messages appear

Expected or desired behavior:

It uses getPlot tool to see the plot and tell me about it.

[sharon-wang]

There are a few issues with the getPlot tool. For OpenAI/OpenAI-compatible, we get errors like:

[OpenAI] [gpt-5]' Error in chat response: {"error":{"message":"An assistant message with 'tool_calls' must be followed by tool messages responding to each 'tool_call_id'. The following tool_call_ids did not have response messages: call_cRT7Y2vmMHgqL8ocujCgtqMT","type":"invalid_request_error","param":"messages.[8].role","code":null}}

For Snowflake, the model sometimes errors with the above (GPT-5), but for Claude 4 Sonnet, the model may not make a tool call at all. Further investigation is needed.

We are likely a bit out-of-compliance with OpenAI-compatible endpoints by using the Vercel AI SDK v4, as some of the response structures and properties have changed. We may want to pursue upgrading to v5 to become compliant with OpenAI-compatible endpoints.

[sharon-wang]

Some helpful notes from Winston!

I was looking at the types for the OpenAI Typescript SDK. I believe this is the type for a tool call result that's sent to the server for the Completions API: https://github.com/openai/openai-node/blob/4f2b63f776d0af0b3859183dbb1d0156510ffb2b/src/resources/chat/completions/completions.ts#L1413-L1428

export interface ChatCompletionToolMessageParam {
  content: string | Array<ChatCompletionContentPartText>;
  role: 'tool';
  tool_call_id: string;
}

And for the Responses API: https://github.com/openai/openai-node/blob/4f2b63f776d0af0b3859183dbb1d0156510ffb2b/src/resources/responses/responses.ts#L2417-L2444

export interface ResponseFunctionToolCallOutputItem {
  id: string;
  call_id: string;
  output: string | Array<ResponseInputText | ResponseInputImage | ResponseInputFile>;
  type: 'function_call_output';
  status?: 'in_progress' | 'completed' | 'incomplete';
}

Notably, for the Completions API, it only accepts text parts, but for the Responses API, it supports text, images, and files (which I think means just PDFs at this point).

---

So based on this, the getplot tool should start working for OpenAI once https://github.com/posit-dev/positron/pull/10903 is in because it uses the Responses API, but we'll need a different solution for OpenAI-compatible providers (e.g. Snowflake, OpenRouter) which use the Completions API and thus can't accept images.

To get around this, we can replace the tool call message with an assistant text message, and replace the tool result with a user message that has an image for providers using the Completions API.