Vercel AI SDK with Ollama - production

[jakobhoeg]

Description

I've stumbled accross this blog post stating that the Vercel AI SDK can be used in conjunction with Ollama. It works on my local dev server, but I would assume that it also could work on a Vercel production site. That is not the case however (unless I am doing something wrong).

This is my api route (/api/chat/route.ts):

import OpenAI from 'openai';
import { OpenAIStream, StreamingTextResponse } from 'ai';

// Create an OpenAI API client (that's edge friendly!)
const openai = new OpenAI({
  baseURL: 'http://localhost:11434/v1',
  apiKey: 'ollama',
});

// IMPORTANT! Set the runtime to edge
export const runtime = 'edge';

export async function POST(req: Request) {
  const { messages, selectedModel } = await req.json();

  try {
    const response = await openai.chat.completions.create({
      model: 'gemma:2b',
      stream: true,
      messages,
    });

    // Convert the response into a friendly text-stream
  const stream = OpenAIStream(response);

  // Respond with the stream
  return new StreamingTextResponse(stream);
  } catch (error) {
    console.error('Error:', error);
  }

}

I've tried multiple setups, and I keep getting these errors in production (Vercel logs):

Error: 403 error code: 1003
    at (node_modules/openai/error.mjs:46:19)
    at (node_modules/openai/core.mjs:256:24)
    at (node_modules/openai/core.mjs:299:29)
    at (src/app/api/chat/route.ts:17:21)
    at (node_modules/next/dist/esm/server/future/route-modules/app-route/module.js:189:36)
    at (node_modules/next/dist/esm/server/future/route-modules/app-route/module.js:128:25)
    at (node_modules/next/dist/esm/server/future/route-modules/app-route/module.js:251:29)
    at (node_modules/next/dist/esm/server/web/edge-route-module-wrapper.js:81:20)
    at (node_modules/next/dist/esm/server/web/adapter.js:157:15) {
  status: 403,
  headers: {
  cache-control: 'private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0',
  cf-connection-close: 'close',
  connection: 'keep-alive',
  content-length: '16',
  content-type: 'text/plain; charset=UTF-8',
  date: 'Mon, 18 Mar 2024 15:00:45 GMT',
  expires: 'Thu, 01 Jan 1970 00:00:01 GMT',
  referrer-policy: 'same-origin',
  x-frame-options: 'SAMEORIGIN'
},
  error: undefined,
  code: undefined,
  param: undefined,
  type: undefined
}

AND this:

[POST] /api/chat reason=EDGE_FUNCTION_INVOCATION_FAILED, status=500, user_error=true - this one actually shows as a 405 eror on Vercel, despite saying status=500.

Code example

No response

Additional context

No response

[xycjscs]

base url should aim to an exposed address. Vercel would not access its own localhost.

However, even you change it something else. It still won’t work. I encountered a similar problem. Vercel log shows 504 gateway problem. Actually it never visited my GPU machine.

[xycjscs]

I guess it related to origin problems.

[aetaix]

It's a same origin problem, replace our baseURL with http://127.0.0.1:11434/v1

const openai = new OpenAI({
  baseURL: 'http://127.0.0.1:11434/v1 ',
  apiKey: 'ollama',
});