Separate http api and job processing in app-service (k8s)

[Rory-Powell]

Describe the feature request Currently both the http api and job (currently limited to automations) processing are performed in the same app-service instance. To enable better scaling for both concerns we want to enable app-service to be run in different modes.

COS

• Introduce an environment variables:
  • DISABLE_JOB_PROCESSING: number (1/0)
    • Default: 0
    • Turn off bull job processing.
    • Writing to bull should still be enabled and if the api is enabled it should still be possible to view the bull dashboard.
  • DISABLE_HTTP_API: number (1/0)
    • Default: 0
    • Turns off the koa http api
• Add new kubernetes deployment / service - app-service-jobs
  • Ideally we can create a common definition for app-service that is re-used by app-service-jobs, see: https://faun.pub/dry-helm-charts-for-micro-services-db3a1d6ecb80
  • Update the environment of each deployment as:
    • app-service: DISABLE_JOB_PROCESSING=1, DISABLE_HTTP_API=0
    • app-service-jobs: DISABLE_JOB_PROCESSING=0, DISABLE_HTTP_API=1

Additional considerations

• We should further investigate the shutdown process of app-service-jobs. If there are any running jobs and shutdown is called, we should not let kubernetes terminate the instance until it safely finishes executing all jobs.
• The liveness probe for app-service-jobs should be removed, or alternatively a very minimal /health endpoint should be added. As above on shutdown this endpoint should not fail until all jobs have been safely executed

[Rory-Powell]

app-service portion of this task completed in: https://github.com/Budibase/budibase-pro/compare/labday/app-service-jobs?expand=1 https://github.com/Budibase/budibase/compare/labday/app-service-jobs?expand=1

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity.