feat: Agentic chains

[cvauclair]

• [x] I have looked for existing issues (including closed) about this

Feature Request

Add a new high level building block to Rig: the agent chain!

Motivation

Agentic chains (or agent chains, agent pipelines, etc.) have become a cornerstone of building LLM powered applications. Chains are flexible constructs that can be used to chain together multiple agents in a multistep process, define some agentic control flow based or simply to add data processing steps before and after prompting.

Whereas Rig Agents can be considered highly configurable blackboxes, chains make the prompting process explicit and declarative. They would therefore make a great addition to the library.

Proposal

Chains should provide a declarative interface for developers to define some arbitrary flow of operations. In fact, at their most general, chains are simply data processing pipelines where some steps happen to involve AI elements. As such, Rig chains should be implement as general processing pipeline, with some additional quality-of-life helpers for common LLM-related operations (e.g.: prompting a model/agent, RAGging documents, etc.). See the code snippet below for a proof of concept.

Chains would be analogous to Rust Iterators (or Streams) with two key differences:

1. They do not hold values (in other words, whereas Iterators have an associated type Item, chains would have two associated types Input and Output)
2. They operate on a single value at a time

// Proof of concept
let chain = chain::new()
    // Retrieve top document from the index and return it with the prompt
    .lookup(index, 2)

    // Format the prompt with the context documents if the previous step 
    // was successful
    .map_ok(|(query, docs): (_, Vec<String>)| {
        format!(
            "User question: {}\n\nWord definitions:\n{}",
            query,
            docs.join("\n")
        )
    })
    
    // Prompt the agent
    .prompt(&agent);

let response = chain.call("What does \"glarb-glarb\" mean?").await?;

Alternatives

Alternatives explored included leveraging existing Rust Iterators and/or Streams to implement Rig chains, but these proved to be inadequate.