celery-batches icon indicating copy to clipboard operation
celery-batches copied to clipboard

flower support

Open orlov-ai opened this issue 2 years ago • 2 comments

Hello! How can I use it with flower? "create_task" works fine with flower, but I can't see "count_click" tasks on flower. What could I do wrong? It feels like I'm using wrong namespaces or starting arguments.

from .celery import celery_app
import time

from celery_batches import Batches, SimpleRequest
from collections import Counter

@celery_app.task(name="create_task")
def create_task(task_type):
    time.sleep(int(task_type) * 2)
    return True


@celery_app.task(base=Batches, flush_every=5, flush_interval=2)
def count_click(requests: list[SimpleRequest]):
    """Count the number of times each URL is requested."""
    count = Counter(request.kwargs["click"] for request in requests)
    for url, count in count.items():
        print(f">>> Clicks: {url} -> {count}")
    for request in requests:
        celery_app.backend.mark_as_done(request.id, 'hi', request=request)

A working example would be appreciated.

orlov-ai avatar Jul 23 '22 02:07 orlov-ai

celery-batches unfortunately doesn't provide all the signals that normal Celery tasks do (see #33 for an example). I suspect this is why you're not seeing it appear properly in flower. Does flower have a list of signals it depends on?

clokep avatar Jul 25 '22 12:07 clokep

Thinking a bit more about this, I wonder what the appropriate events to send are. For example if we call a task 30 times with a batch size of 10; should we report that each task request started, finished, etc. or should we only report each batch? I suspect the latter makes more sense.

clokep avatar Jan 08 '24 20:01 clokep