Draft: Support Programmatic API

[Mini256]

Goal

• NOT a API SDK, compared to the HTTP API, the programmatic API can further extend the basic workflow of Autoflow by callbacks / hooks and other methods

API Design

Create New Knowledge Base

from autoflow import Autoflow, IndexMethod

autoflow = Autoflow(db_engine)

kb = autoflow.create_knowledge_base(
  name="TiDB's Docs",
  index_methods=[IndexMethod.VECTOR_SEARCH, IndexMethod.KNOWLEDGE_GRAPH]
)

Add/Import Docuemnts into Knowledge Base

Option 1: Add document directly

kb.add_document(Document(name="Introduction TiDB", content="xxxx"))

Option 2: Import document from files.

with open("path/to/doc.md") as file:
  kb.import_document_from_file(file)

Option 3: Import document from other source

metadata_extractor = CustomMetadataExtractor()
ds = DataSource(
  type="github",
  name="TiDB Docs",
  config={
     "repository_url": "https://github.com/pingcap/docs",
     include=["*.md"]
  },
  metadata_extractor=metadata_extractor
)
kb.import_document_from_data_source(ds)

Retrieve Documents

Knowledge Base Level

result = kb.retrieve_chunks(query="What is TiDB?")

Global Level

result = autoflow.retrieve_chunks(query="What is TiDB?")

Retrieve Knowledge Graph

Knowledge Base Level

knowledge_graph = kb.retrieve_knowledge_graph(
    query="Does TiDB support Vector Search?"
)

Global Level

knowledge_graph = autoflow.retrieve_knowledge_graph(
    query="Does TiDB support Vector Search?"
)

Chat with Knowledge Base

chat_engine = ChatEngine(llm, knowledge_bases=[kb], config={})
chat_engine.chat(user_question="What is TiDB", chat_history=[])

stackvm = StackVM(llm)
stackvm.register_tool(...)
stackvm.register_tool(...)
chat_engine = PlanableChatEngine(stackvm=stackvm)
chat_engine.chat(goal="What is TiDB")

[sykp241095]

How about using wiki to design the whole APIs? There we can design the API directories and each API usage & parameters.

[Mini256]

How about using wiki to design the whole APIs?

I don't think we need a wiki. Wiki is not a suitable tool for discussion and review. In this issue, we will only discuss the overall structure, and the specific API parameters should be discussed in the sub issues and PR.

[c4pt0r]

Can you add some low-level storage API, like Put(K,V) / Get(K,V) / Query(K)