feat(ai): llm provider

[kallebysantos]

What kind of change does this PR introduce?

feature, refactor

What is the current behaviour?

Current the Session only supports self-hosted Ollama or some OpenAI like provider - with no way to specify the API key.

What is the new behaviour?

This PR applies some refactors in the ai module to support an unified LLM provider API, this way it can be easily extended to new providers as well exporting a more standardised output format.

Improved typescript support

The ai module was huge refactored to provide better ts hints that dynamically changes based on the selected type:

examples

using type gte-small:

using type ollama:

using type openaicompatible:

Automatically infer AsyncGenerator type when stream: true

All providers, ollama & openai

Improved error handling support

In order to ensure error checking, the ai module was been refactored to follow Result pattern - Go like. It means that while Session.run() the returned value will be a tuple array of [success, error], this result is compatible with TS pattern matching, so it provides a completely LSP feedback.

examples

Non stream

Result type def

Checking error automatically validates the success part

Stream

When stream: true the first result will handle errors that may occur before create the AsyncGenerator. Then the incoming message will be a result as well, this way users can apply error handling while streaming.

Result type def

Streaming type def

Common response and Usage metrics

Since all LLM providers must implement a common interface they now also shared a unified response object.

response definitions

Success part

// This object represents a success LLM output, where inner is the raw value returned by the provider
export interface ILLMProviderOutput<T = object> {
  value?: string; // An unified way to get the generated value
  usage: {
    inputTokens: number;
    outputTokens: number;
    totalTokens: number;
  };
  inner: T;
}

Error part

// This object represents an error output, where inner is the original error value
export type SessionError<T = object | string> = {
  message: string;
  inner: T;
};

Tested OpenAI compatible providers

• OpenAI
• TogetherAI

missing

• Anthropic: I don't have free account for it
• Grok: I don't have free account for it
• Amazon Bedrock: I don't have free account for it

ideas

• Gemini: maybe implement a custom provider for this one

[kallebysantos]

Hi guys 💚

I know that I did things different here, by introducing internal .ts files. I notice that upstream repo is already coding ts instead of js and this PR was like a "experiment" of it and it really worked - I could get a great dev exp from this typescript support.

Since I did it, I wondering that may be nice if we have tsc bundler, this way we could easily generate .d.ts files to release - Instead of writing everything inside the globals.d.ts file.

I don't have too much exp on theses bundlers/releases but I'm sure that is possible to do. I would like to learn/do it, but I need some suggestion/approve before I go further.

My idea is to have something like that:

1. call tsc to look into ext/**/index.d.ts - or anything else u guys prefer to name it
2. inside this file we just re-export our internal types in a public namespace
3. then bundle it in a dist/edge-runtime.d.ts - This one will be then release to be used as

import "jsr:@supabase/functions-js/edge-runtime.d.ts";

I believe that native .ts files for future feats/refactors can enforce our dev exp, at same time we export theses types defs to user land.

---

cc @laktek, @nyannyacha

[github-actions[bot]]

🔕 This PR is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 7 days.

[kallebysantos]

bot reply

[github-actions[bot]]

🔕 This PR is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 7 days.

[kallebysantos]

bot reply