edx-django-utils icon indicating copy to clipboard operation
edx-django-utils copied to clipboard

Published 20 hours ago • verified publisher icon openedx

Feat add dd and otel celery tasks

Open connorhaugh opened this issue 1 year ago • 1 comments

Description:

Adds an initialize_celery_monitoring util to allow celery tasks to be monitored by the Open Telemetry and Datadog providers. edx-platform PR to follow.

JIRA:

https://github.com/edx/edx-arch-experiments/issues/584

Testing instructions:

  1. Set this package to be installed locally, check out this PR
    1. Run your edx-platform off this branch: https://github.com/openedx/edx-platform/pull/34673 to use the implementation required from the package.
  3. set your OPENEDX_TELEMETRY django setting to be [edx_django_utils.monitoring.DataDogBackend]
  4. follow https://2u-internal.atlassian.net/wiki/spaces/~840928901/pages/813793298/OpenTelemetry+New+Relic+and+Datadog for devstack setup for DD, but set up the DD agent on CMS, not lms
  5. Follow https://2u-internal.atlassian.net/wiki/spaces/ENGAGE/pages/395673715/How+to+configure+Celery+worker+on+LMS+CMS+Devstack to set up celery workers on cms.
  6. Export a course
  7. View the export on DD with a celery task

Reviewers:

  • [ ] tag reviewer

Merge checklist:

  • [ ] All reviewers approved
  • [x] CI build is green
  • [x] Version bumped
  • [x] Changelog record added
  • [x] Documentation updated (not only docstrings)
  • [ ] Commits are squashed

Post merge:

  • [ ] Create a tag
  • [ ] Check new version is pushed to PyPi after tag-triggered build is finished.
  • [ ] Delete working branch (if not needed anymore)

Author concerns:

List any concerns about this PR - inelegant solutions, hacks, quick-and-dirty implementations, concerns about migrations, etc.

connorhaugh avatar Apr 29 '24 18:04 connorhaugh

Based on Tim's comment, I went and looked into if we were getting what we needed from celery by default.

To do that, I looked at the logs over the past month.

Do the logs related to celery increase or decrease in volume when we enabled DD in edx-platform? The answer is it increased in volume from april 22-27rd, which was the window for edx-platform having celery on in dd. I have been really struggling to explain why that is the case, however, given the code that enables the patch by default. Either way, I'm making improvements to the code and we should maybe plan on needing it.

connorhaugh avatar May 09 '24 12:05 connorhaugh

@timmc-edx: FYI: I'm closing this PR. I assume at this point we do not need this.

robrap avatar Jun 07 '24 17:06 robrap