orchestrator-core icon indicating copy to clipboard operation
orchestrator-core copied to clipboard

[Documentation]: Ensure scheduler cli commands initialize DB connection + document setting up celery client

Open Mark90 opened this issue 2 months ago • 0 comments

The solution I suggested in #1084 -splitting off the API initialization from main.py to wsgi.py- has as consequences that

  1. main.py scheduler commands execute without a database connection being present. This can be solved by users by importing and calling init_database() in their main.py, but the better solution is to ensure that scheduler cli commands do this
  2. main.py scheduler commands execute without a celery client being initialized. This cannot be solved in the core because setting up celery has to be done on the implementation side. It should be added to the documentation instead.

Setting up celery client

Add file with celery client:

# tasks_client.py

from celery import Celery

from orchestrator.settings import app_settings
from settings import backend, broker

# Use the celery_client from the API or Scheduler to put tasks on the Celery queue
celery_client = Celery(
    app_settings.SERVICE_NAME,
    broker=broker,
    backend=backend,
    include=["orchestrator.services.tasks"],
)

celery_client.conf.update(
    result_expires=3600,
)

Update main.py (below example is from surf, consolidate with minimum exapmle in docs)

# main.py

import typer
from orchestrator import app_settings
from orchestrator.cli.main import app as core_cli
from orchestrator.db import init_database
from orchestrator.services.tasks import initialise_celery
from surf import load_surf_cli
from surf.tasks_client import celery_client


def init_cli_app() -> typer.Typer:
    init_database(app_settings)
    initialise_celery(celery_client)
    load_surf_cli(core_cli)
    return core_cli()


if __name__ == "__main__":
    init_cli_app()

Update wsgi.py to import tasks_client.celery_client and run initialise_celery(celery_client)

Mark90 avatar Oct 29 '25 17:10 Mark90