anthropic pdf

[narengogi]

Author Description

so couple of things in this PR, I've added two new parameters in the chat completions request body file content part

// file content part
{
  type: 'file';
  file?: {
    file_data?: string;
    file_id?: string;
    file_name?: string;
    file_url?: string; // new field
    mime_type?: string; // new field
  };
}

I've tested with anthropic, vertex anthropic and bedrock anthropic note: vertex anthropic does not support plain text and pdf urls, bedrock does not support plain text, and only supports s3urls

here are request bodies to test pdf url

{
    "model": "claude-3-5-sonnet-20240620",
    "max_tokens": 200,
    "stream": false,
    "messages": 
        }
    ]
}

pdf base64

{
    "model": "claude-3-5-sonnet-20240620",
    "max_tokens": 200,
    "stream": false,
    "messages": 
        }
    ]
}

plain text document

{
    "model": "claude-3-5-sonnet-20240620",
    "max_tokens": 200,
    "stream": false,
    "messages": 
        }
    ]
}

Summary By MatterAI

🔄 What Changed

This PR adds support for PDF files in Anthropic and Bedrock providers by introducing two new parameters in the chat completions request body file content part: file_url and mime_type. The implementation supports both PDF URLs and base64-encoded PDFs, as well as plain text documents.

🔍 Impact of the Change

This enhancement allows users to send PDF documents to Anthropic and Bedrock models for analysis. Different providers have varying support: Vertex Anthropic doesn't support plain text and PDF URLs, while Bedrock only supports S3 URLs for PDFs.

📁 Total Files Changed

3 files changed with 120 additions and 3 deletions:

• src/providers/anthropic/chatComplete.ts: Added PDF handling for Anthropic
• src/providers/bedrock/chatComplete.ts: Added PDF handling for Bedrock
• src/types/requestBody.ts: Updated content type interfaces

🧪 Test Added

Manual testing was performed with Anthropic, Vertex Anthropic, and Bedrock Anthropic providers using various request bodies (PDF URL, PDF base64, plain text).

🔒 Security Vulnerabilities

N/A

Type of Change

• [x] New feature (non-breaking change which adds functionality)
• [ ] Bug fix (non-breaking change which fixes an issue)
• [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
• [ ] Documentation update
• [ ] Refactoring (no functional changes)

How Has This Been Tested?

• [ ] Unit Tests
• [ ] Integration Tests
• [x] Manual Testing

Screenshots (if applicable)

N/A

Checklist

• [x] My code follows the style guidelines of this project
• [x] I have performed a self-review of my own code
• [ ] I have commented my code, particularly in hard-to-understand areas
• [ ] I have made corresponding changes to the documentation
• [ ] My changes generate no new warnings
• [ ] I have added tests that prove my fix is effective or that my feature works
• [ ] New and existing unit tests pass locally with my changes

Related Issues

N/A

Quality Recommendations

1. Add error handling for unsupported file types or formats

2. Add validation for file_url and file_data to ensure they are properly formatted

3. Consider adding unit tests to verify the PDF handling functionality

4. Add documentation comments for the new interfaces and functions

Sequence Diagram

sequenceDiagram
    participant Client
    participant Gateway
    participant AnthropicProvider
    participant BedrockProvider
    
    Client->>Gateway: POST /chat/completions
    Note over Client,Gateway: Request with file_url or file_data
    
    alt Anthropic Provider
        Gateway->>AnthropicProvider: transformAndAppendFileContentItem()
        Note over AnthropicProvider: Process file content based on type
        
        alt PDF URL
            AnthropicProvider-->>AnthropicProvider: Create AnthropicUrlPdfContentItem
            Note over AnthropicProvider: { type: 'document', source: { type: 'url', url: file_url } }
        else PDF Base64
            AnthropicProvider-->>AnthropicProvider: Create AnthropicBase64PdfContentItem
            Note over AnthropicProvider: { type: 'document', source: { type: 'base64', data: file_data, media_type: mime_type } }
        else Plain Text
            AnthropicProvider-->>AnthropicProvider: Create AnthropicPlainTextContentItem
            Note over AnthropicProvider: { type: 'document', source: { type: 'text', data: file_data, media_type: mime_type } }
        end
    else Bedrock Provider
        Gateway->>BedrockProvider: getMessageContent()
        Note over BedrockProvider: Process file content based on type
        
        alt File URL (S3)
            BedrockProvider-->>BedrockProvider: Create document with s3Location
            Note over BedrockProvider: { document: { format: fileFormat, name: UUID, source: { s3Location: { uri: file_url } } } }
        else File Data
            BedrockProvider-->>BedrockProvider: Create document with bytes
            Note over BedrockProvider: { document: { format: fileFormat, name: UUID, source: { bytes: file_data } } }
        end
    end
    
    Gateway-->>Client: Return completion response

https://github.com/Portkey-AI/gateway/issues/1075