feature(dspy): generate a graphical representation of modules

[fsndzomga]

The ModuleGraph class is a utility for generating visual representations of Python modules and their components using the graphviz library. It enables users to visualize the structure and interactions of modules, making it easier to understand complex systems.

Key Features:

Dynamic Graph Generation: The ModuleGraph class dynamically generates a graph representation of Python modules, including their submodules and components.

Dependency Visualization: It visualizes dependencies between modules, submodules, and components, helping users understand how different parts of the system interact with each other.

Graphviz Integration: Uses the graphviz library to create and render graphs in various formats (e.g., PNG), allowing for customization and easy sharing of visualizations.

Error Handling: Provides informative error messages if the graphviz library is not installed, guiding users on how to install the required dependency (graphviz) via pip.

Modular Design: The class is designed with modularity in mind, making it easy to extend and customize for different use cases and module structures.

Usage Example:

import dspy
import os
from dotenv import load_dotenv

load_dotenv()

# Configuration of dspy models
llm = dspy.OpenAI(
    model='gpt-3.5-turbo',
    api_key=os.environ['OPENAI_API_KEY'],
    max_tokens=100
)

colbertv2_wiki = dspy.ColBERTv2(url='http://20.102.90.50:2017/wiki17_abstracts')

dspy.settings.configure(lm=llm, rm=colbertv2_wiki)

class GenerateAnswer(dspy.Signature):
  "Answer with long and detailled answers"
  context = dspy.InputField(desc="may content relevant facts")
  question = dspy.InputField()
  answer = dspy.OutputField(desc="often between 10 and 50 words")

class RAG(dspy.Module):
  def __init__(self, num_passages=3):
    super().__init__()
    self.retrieve = dspy.Retrieve(k=num_passages)
    self.generate_answer = dspy.ChainOfThought(GenerateAnswer)

  def forward(self, question):
    context = self.retrieve(question).passages
    prediction = self.generate_answer(context=context, question=question)
    return dspy.Prediction(context=context, answer=prediction.answer)

rag_system = RAG()
graph = ModuleGraph("RAG", rag_system)

graph.render_graph()

Here is the graph generate in this case:

[arnavsinghvi11]

Thanks @fsndzomga, this looks great! Left some minor comments and is ready to merge after that.

It would be great to have documentation for it as well, potentially detailing the motivations behind use cases and applications for graphical representations added to the Community Examples if you would like to!

[arnavsinghvi11]

@fsndzomga just bumping this!

[arnavsinghvi11]

Hi @fsndzomga , just bumping the comments again as the PR is not yet mergeable. would love to merge this once those are resolved!

[fsndzomga]

Hi @fsndzomga , just bumping the comments again as the PR is not yet mergeable. would love to merge this once those are resolved!

Hi @arnavsinghvi11, sorry, this completely slipped my mind. I just shipped the first version. I will further enhance the capabilities later.

[arnavsinghvi11]

Thanks @fsndzomga ! Merging this, and feel free to open another PR with additional changes and documentation!