[Documentation]: Improve the Guides chapter

[Mark90]

Context

As part of the initiative to improve the WFO documentation, we want to divide the documentation in 3 chapters:

1. Workflow Orchestrator (Architecture, the "Why")
2. Guides (the "How")
3. Reference (the "What" - verbose summary/list of all options)

This has partially been done, but the chapters themselves still need to be restuctured and fleshed out by adding new sections or moving existing sections from other chapters.

This ticket's main focus is the Guides chapter.

Goals

• To restructure the Guides chapter so that it is clear ƒor who these docs are meant: first time users, experienced users, operators.
• To add sections with missing information and/or move sections from other chapters that belong in Guides.
• To add more Orchestrator-UI documentation

Current structure

At the time of writing the chapter has this structure:

• Getting Started
  • Prerequisites
  • Application
    • Preparing source folder
    • Base application
    • Agentic application
  • Workflows
  • Docker
  • Orchestrator UI
• Product Modeling
  • Importing Existing Products
  • Backfilling Existing Subscriptions
• Creating Tasks and Schedules
• Scaling the Orchestrator
• Pausing the Orchestrator
• Upgrade Guides
  • 2.x
  • 3.x
  • 4.x

Proposed new structure

• Beginner (first time seeing the project or not having a WFO implementation yet)
  • Example-orchestrator link/showcase (choice to bootstrap own Implementation or follow workshop)
  • Bootstrap WFO
    • Orchestrator-core implementation
    • Orchestrator-UI implementation
    • Postgres / pgvector
    • Redis
  • Example orchestrator workshop (from https://workfloworchestrator.org/orchestrator-core/workshops/advanced/overview/)
• Intermediate (having a (basic) WFO implementation)
  • Developing products and workflows
    • Creating a product
      • Bootstrap product
        • Generate template files through YAML
        • Copy existing product/workflow files (how to register everything)
      • Design forms (part of https://workfloworchestrator.org/orchestrator-core/reference-docs/forms/)
      • Product Modeling (import/backfill existing subscriptions)
    • Updating a product (new fields/blocks/workflows)
      • Files to edit
      • CLI commands to generate migrations
    • Removing a product/workflow
      • Delete versus soft-delete
      • Files to edit/delete
      • CLI commands to generate migrations
  • Developing tasks
    • Scheduler
  • Bootstrap WFO intermediate
    • Scaling the orchestrator (threadpool v celery)
    • Authentication
  • Upgrade guides
    • Orchestrator-core
      • 2.x / 3.x
    • Orchestrator-UI
• Operating (on an existing WFO implementation; these sections could be structured better)
  • Deployment practices (docker, k8s)
  • Compatibility matrix (Orchestrator-core / Orchestrator-UI)
  • Database management
    • Creating and restoring postgres backups
    • Running migrations (link to ref-docs for db CLI commands)
    • Manipulate data with orchestrator-shell (expert usage)
  • Pausing the orchestrator
  • Handling failed/stuck workflows/tasks
  • Interacting with the scheduler (link to ref-docs for scheduler CLI commands)
• Developing / Contributing
  • Orchestrator-core
    • Contributing sections (from current https://workfloworchestrator.org/orchestrator-core/contributing/development/)
  • Orchestrator-UI
  • nwa-stdlib, oauth2-lib, pydantic-forms
  • Versioning (https://workfloworchestrator.org/orchestrator-core/reference-docs/python/)

Key points to keep in mind

• Refrain from explaining why things are the way they are -> This should be in the Architecture chapter.
• Refrain from summarizing all possible ways to do or what an API Endpoint/UI Component/Database table is for -> This should be in the Reference Documentation chapter.
• Look through the other chapters and identify sections that belong in Guides
• When moving existing sections, make sure to keep URL's intact by adding the required redirects: mappings in mkdocs.yml
• When adding a new section that requires a significant amount of work, please add a stub implementation and create an additional github ticket to fill it in more detail - then link to that ticket from the documentation.

[eenblam]

This feels like a big chunk of work, but I think the overall structure reads like a proper book on WFO! I think it's a worthwhile direction to move in.

I think if we do pursue this, we should probably commit in some substantial way to the design before making big sweeping changes. For example, we could move this whole structure into the mkdocs file and just comment out the parts that haven't been created yet, so it's obvious where things should go later. I can imagine this being a bit controversial, though.

When adding a new section that requires a significant amount of work, please add a stub implementation and create an additional github ticket to fill it in more detail - then link to that ticket from the documentation.

I think this will be helpful not just for us tracking, but also for users confused by abbreviated documentation. It communicates that there IS more for them to know, and it gives them a place to ask for clarification, which should help us know what docs are most in demand by users.

[aughera]

I love the structure with the separation of different learner levels!

Should we move "Handling failed/stuck workflows/tasks" to at least the Intermediate level?

[Mark90]

Thanks for the feedback guys.

Should we move "Handling failed/stuck workflows/tasks" to at least the Intermediate level?

Hmm, I saw this as an administrative/operative action for a deployed WFO stack. The "Intermediate" section contains material for development on an WFO stack. Maybe that aspect should be emphasized more in the section title. But I'm open to other structures, feel free to post suggestions :)