Task result metadata

[pdrivomHW]

Description

It would be very useful for django-tasks to support a built-in task caching mechanism to allow developers to track the progress and store metadata of running tasks. This is particularly helpful for long-running tasks where the client needs to poll for updates or retrieve the result later.

We've implemented a simple version of this using a TaskCache model and utility functions to associate a task with a cache entry, update its progress, and retrieve results.

Example Use Case

# Create a cache entry before enqueuing
cache_id = init_task_cache()
result = some_long_task.enqueue(args, cache_id)
bind_task_cache(cache_id, result.id)

# Inside the task
@task()
def some_long_task(args, cache_id=None):
    set_task_cache(cache_id, "progress", 0)
    # do some work...
    set_task_cache(cache_id, "progress", 50)
    # more work...
    set_task_cache(cache_id, "progress", 100)

Example Implementation

# models.py
class TaskCache(models.Model):
    id = models.UUIDField(primary_key=True, default=uuid.uuid4, editable=False)
    task_id = models.UUIDField(blank=True, null=True)
    cache_data = models.JSONField(default=dict)

# utils.py
def init_task_cache():
    task_cache = TaskCache.objects.create(cache_data={})
    return str(task_cache.id)

def bind_task_cache(cache_id, task_id):
    task_cache = TaskCache.objects.get(id=cache_id)
    task_cache.task_id = task_id
    task_cache.save()

def set_task_cache(cache_id, key, value):
    task_cache = TaskCache.objects.get(id=cache_id)
    task_cache.cache_data[key] = value
    task_cache.save()

def get_task_cache_by_task_id(task_id, key):
    task_cache = TaskCache.objects.get(task_id=task_id)
    return task_cache.cache_data.get(key)

Benefits

• Enables tracking task progress (e.g. for frontend progress bars)
• Stores metadata or partial results across async boundaries
• Simplifies debugging and monitoring of background jobs
• Provides a standardized pattern for cache/task binding

Proposal

Add optional support to django-tasks for task-level caching. This could include:

• A pluggable base cache model or interface
• Hooks or decorators to bind tasks to a cache record
• Option to use Django’s cache framework or a model-based backend

[RealOrangeOne]

I don't think this needs to be a cache specifically, but some kind of additional metadata on a task would be useful.

Currently, running function doesn't have access to the task object, which makes it more complex. That might be the first step in resolving this.

[stuartmaxwell]

I've used this pattern with django-rq to display progress bars for long-running tasks. django-rq uses the RQ library to do this, and it's documented briefly here: https://python-rq.org/docs/jobs/#storing-arbitrary-data-on-jobs

It's useful for storing the status of the job and then passing it to some JavaScript via a template.

    ...
    return {
        "status": job.get_status(),
        "current_status": job.meta.get("current_status", ""),
        "progress": job.meta.get("progress", ""),
        "time": job.meta.get("time", ""),
    }

Would love to have this feature implemented here.

[RealOrangeOne]

Now that tasks have context (#174 ), adding scratch metadata like this is a logical next step. I agree it has a lot of uses, particularly around progress.