[Feature] Define interface for ModelProvider to ease contributing new providers

[zpg6]

Adding a new provider is a bit scattered across a few files and the steps aren't clear. It should be more interface-driven so that we can scale up adding many models (and lightweight assistants, agents, etc)... here's an example:

interface ModelProvider {
  // Initialize provider with authentication and configuration
  initialize(config: ProviderConfig): Promise<void>;
  
  // Send a chat request and get streamed response
  sendChat(params: ChatRequestParams): Promise<ChatResponse>;
  
  // Fill-in-middle completion (if supported)
  sendFIM?(params: FIMRequestParams): Promise<FIMResponse>;
  
  // List available models (if supported)
  listModels?(): Promise<ModelInfo[]>;
  
  // Abort ongoing requests
  abort(): void;
  
  // Check if a specific capability is supported
  supportsCapability(capability: ProviderCapability): boolean;
}

@andrewpareles: if this is of interest I'd like to work on it :)

[zpg6]

Also a fan of

type ModelProvider = {
  initialize: (config: ProviderConfig) => Promise<void>;
  sendChat: (params: ChatRequestParams) => Promise<ChatResponse>;
  sendFIM?: (params: FIMRequestParams) => Promise<FIMResponse>;
  listModels?: () => Promise<ModelInfo[]>;
  abort: () => void;
  supportsCapability: (capability: ProviderCapability) => boolean;
};

with

const createSomethingProvider = (ctx) => {
  return {
    initialize: async (providerConfig: ProviderConfig) => { 
        /* implementation */ 
    },
    sendChat: async (params: ChatRequestParams) => { 
        /* implementation */ 
    },
    abort: () => { 
        /* implementation */ 
    },
    supportsCapability: (capability: ProviderCapability) => { 
        /* implementation */ 
    }
  } satisfies ModelProvider;
};

[zpg6]

Some of the pending/requested additions for providers include Databricks, AWS Bedrock, and Copilot.

Others that will be asked for probably: Perplexity, IBM, HuggingFace.

I personally would need Azure AI Inference SDK (how you access their non-OpenAI models) which is OpenAI-like. I think other OpenAI-like frameworks like Letta (MemGPT) or AutoGen Agents could fit in nicely too.

[zpg6]

@andrewpareles when possible please share your thoughts on this proposal. If it's not aligned with your planning no worries!

[andrewpareles]

Hey @zpg, so sorry I missed this! I really like this idea. Just want to make sure the abstraction is right -- want to talk about it briefly via audio? Hard to talk by text. We have weekly meetups Thurs @ 5:30p PT, or can happy to do a quick call to go over everything (can you message me on Discord?)

[vrtnis]

hey @zpg6, this is a neat proposal! just wondering what was the outcome of the discussion on the direction here? any chance you might be able to join tomorrow’s call to discuss?

For context, I’m looking into resolving #672 and #596 and helping connect the dots across others to support the roadmap. Both of these run into limitations with the current provider setup.. esp. around supporting Azure-compatible endpoints with custom headers and Vertex-hosted Claude (non-OpenAI-style routing unsurprisingly).

i'd advocate for making the ProviderConfig passed to initialize() flexible enough to support (to the extent possible):

• custom domains and headers (e.g. Azure OAI-compatible endpoints with nonstandard x-api-key headers)

• partner routes like claude on vertex, which diverge significantly from oai’s schema and url shape

that said, the current setup does cover a majority of cases, so it would make sense to define a clean extension mechanism (e.g. provider subclasses or route templates) to handle these edge cases without bloating the core abstraction - would be great to hear your thoughts.

[zpg6]

@vrtnis Great points, looking forward to discussing more in today's call!

[andrewpareles]

A couple relevant files: modelCapabilities.ts, sendLLMMessage.impl.ts, and potentially convertToLLMMessageService.ts and sendLLMMessage.ts (I think that's it!)