ai icon indicating copy to clipboard operation
ai copied to clipboard

Bedrock adapter crashes if model hallucinates tool call with $ in the name

Open MaxAller opened this issue 1 month ago • 1 comments

Description

Sometimes, LLM hallucinate tool calls. When using Bedrock + Claude Sonnet 4.5, there are certain patterns that seem especially prone to hallucination -- if you give it download capabilities but not read capabilities, it's still going to try to read, but make up a tool, usually something like $READFILE. One solution to this is to simply give it read capabilities, but what happens in the meantime is the provider will actually crash out.

If you look at the ToolSpecification docs, it specifically says tool names must match [a-zA-Z0-9_-]+, yet the LLM is prone to generating names that contain $ for some reason. This causes Bedrock to throw an error.

Because this is an outright crash, even experimental_repairToolCall isn't invoked; it seems that the conversation history gets corrupted when this happens, but I confess I don't 100% understand the error path.


Expected

  1. Give Claude Sonnet 4.5 over Amazon Bedrock a download tool and prompt it to download a file. It will often try to read the file.
  2. experimental_repairToolCall will be called with a NoSuchToolError exception if a non-existent read operation is attempted.

Actual

  1. Same as above.
  2. Amazon Bedrock provider crashes instead.

Exception

Here, it hallucinated a $CHAT_READ_TOOL tool:

APICallError [AI_APICallError]: undefined: 1 validation error detected: Value at 'messages.5.member.content.2.member.toolUse.name' failed to satisfy constraint: Member must satisfy regular expression pattern: [a-zA-Z0-9_-]+
    at file:///<app>/node_modules/@ai-sdk/provider-utils/dist/index.mjs:804:14
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
    at async postToApi (file:///<app>/node_modules/@ai-sdk/provider-utils/dist/index.mjs:656:28)
    at async BedrockChatLanguageModel.doStream (file:///<app>/node_modules/@ai-sdk/amazon-bedrock/dist/index.mjs:923:50)
    at async fn (file:///<app>/node_modules/ai/dist/index.mjs:4917:27)
    at async file:///<app>/node_modules/ai/dist/index.mjs:1511:22
    at async _retryWithExponentialBackoff (file:///<app>/node_modules/ai/dist/index.mjs:1662:12)
    at async streamStep (file:///<app>/node_modules/ai/dist/index.mjs:4873:15)
    at async Object.flush (file:///<app>/node_modules/ai/dist/index.mjs:5189:23) {
  cause: undefined,
  url: 'https://bedrock-runtime.us-west-2.amazonaws.com/model/global.anthropic.claude-sonnet-4-5-20250929-v1%3A0/converse-stream',
  requestBodyValues: {
    system: [ [Object] ],
    messages: [ [Object], [Object], [Object], [Object], [Object], [Object] ],
    additionalModelRequestFields: undefined,
    inferenceConfig: { maxTokens: 4096 },
    toolConfig: { tools: [Array], toolChoice: [Object] }
  },
  statusCode: 400,
  responseHeaders: {
    connection: 'keep-alive',
    'content-length': '194',
    'content-type': 'application/json',
    date: 'Thu, 13 Nov 2025 14:14:36 GMT',
    'x-amzn-errortype': 'ValidationException:http://internal.amazon.com/coral/com.amazon.bedrock/',
    'x-amzn-requestid': '78750a60-60ed-4bb9-b147-330f215b18f2'
  },
  responseBody: `{"message":"1 validation error detected: Value at 'messages.5.member.content.2.member.toolUse.name' failed to satisfy constraint: Member must satisfy regular expression pattern: [a-zA-Z0-9_-]+"}`,                                                                                                                         
  isRetryable: false,
  data: {
    message: "1 validation error detected: Value at 'messages.5.member.content.2.member.toolUse.name' failed to satisfy constraint: Member must satisfy regular expression pattern: [a-zA-Z0-9_-]+"                                                                                                                                           
  },
  [Symbol(vercel.ai.error)]: true,
  [Symbol(vercel.ai.error.AI_APICallError)]: true
}
undefined: 1 validation error detected: Value at 'messages.5.member.content.2.member.toolUse.name' failed to satisfy constraint: Member must satisfy regular expression pattern: [a-zA-Z0-9_-]+             

Repro script

If you put this in the amazon-bedrock directory and set up your AWS credentials and region as environment variables, you can run the following:

import { bedrock } from './dist/index.mjs';
import { generateText, tool } from '../ai/dist/index.mjs';
import { z } from 'zod';

console.log('Reproducing Bedrock tool name validation bug...\n');

const model = bedrock('global.anthropic.claude-sonnet-4-5-20250929-v1:0', {
  region: 'us-west-2',
});

try {
  console.log('Sending conversation with $READFILE in history + active tools...');
  
  const result = await generateText({
    model,
    messages: [
      {
        role: 'user',
        content: [{ type: 'text', text: 'Read /tmp/data.txt' }],
      },
      {
        role: 'assistant',
        content: [{
          type: 'tool-call',
          toolCallId: 'call_123',
          toolName: '$READFILE',
          input: {},
        }],
      },
      {
        role: 'tool',
        content: [{
          type: 'tool-result',
          toolCallId: 'call_123',
          toolName: '$READFILE',
          output: { type: 'text', value: 'Tool not found' },
        }],
      },
      {
        role: 'user',
        content: [{ type: 'text', text: 'Try something else.' }],
      },
    ],
    tools: {
      answer: tool({
        description: 'Provide answer',
        inputSchema: z.object({
          text: z.string(),
        }),
        execute: async ({ text }) => text,
      }),
    },
  });

  console.log('\n✓ SUCCESS: Request completed without crashing');
  console.log('  This means the fix IS applied (tool names are sanitized)');

} catch (error) {
  console.error('\n✗ CRASHED with error:');
  console.error(`  ${error.message}\n`);
  
  if (error.message.includes('[a-zA-Z0-9_-]+')) {
    console.error('Tool name crash');
    console.error('  Bedrock rejected "$READFILE" in conversation history\n');
  }
  
  process.exit(1);
}

which for me prints out

Reproducing Bedrock tool name validation bug...

Sending conversation with $READFILE in history + active tools...

✗ CRASHED with error:
  1 validation error detected: Value at 'messages.2.member.content.1.member.toolUse.name' failed to satisfy constraint: Member must satisfy regular expression pattern: [a-zA-Z0-9_-]+

Tool name crash
  Bedrock rejected "$READFILE" in conversation history

Proposed Fix

Sanitize tool names first -- strip out invalid characters. This will still ultimately fail since this only happens with hallucinated tool names anyway, but at least it'll properly error instead of crash.

I'm not sure if this is the optimal fix -- it'd be nice to trigger the tool call repair before we even perform the call -- but it does work in my local testing.

AI SDK Version

  • ai: 5.0.89
  • @ai-sdk/amazon-bedrock: 3.0.52

Code of Conduct

  • [x] I agree to follow this project's Code of Conduct

MaxAller avatar Nov 13 '25 14:11 MaxAller

I can send a PR for this but would appreciate any directional pointers, otherwise I'll send one for my proposed fix.

MaxAller avatar Nov 13 '25 17:11 MaxAller

DocumentBlock has similar restrictions

Und3rf10w avatar Dec 05 '25 06:12 Und3rf10w