orchestrator-core
orchestrator-core copied to clipboard
[Documentation]: Ensure scheduler cli commands initialize DB connection + document setting up celery client
The solution I suggested in #1084 -splitting off the API initialization from main.py to wsgi.py- has as consequences that
main.py schedulercommands execute without a database connection being present. This can be solved by users by importing and callinginit_database()in their main.py, but the better solution is to ensure that scheduler cli commands do thismain.py schedulercommands execute without a celery client being initialized. This cannot be solved in the core because setting up celery has to be done on the implementation side. It should be added to the documentation instead.
Setting up celery client
Add file with celery client:
# tasks_client.py
from celery import Celery
from orchestrator.settings import app_settings
from settings import backend, broker
# Use the celery_client from the API or Scheduler to put tasks on the Celery queue
celery_client = Celery(
app_settings.SERVICE_NAME,
broker=broker,
backend=backend,
include=["orchestrator.services.tasks"],
)
celery_client.conf.update(
result_expires=3600,
)
Update main.py (below example is from surf, consolidate with minimum exapmle in docs)
# main.py
import typer
from orchestrator import app_settings
from orchestrator.cli.main import app as core_cli
from orchestrator.db import init_database
from orchestrator.services.tasks import initialise_celery
from surf import load_surf_cli
from surf.tasks_client import celery_client
def init_cli_app() -> typer.Typer:
init_database(app_settings)
initialise_celery(celery_client)
load_surf_cli(core_cli)
return core_cli()
if __name__ == "__main__":
init_cli_app()
Update wsgi.py to import tasks_client.celery_client and run initialise_celery(celery_client)