Events ghosted to the listeners

[dannwise]

Hello there,

For some reason, when dispatching an event from within any given class method or from a listener's handle method, the event doesn't get captured by its corresponding listeners.

Does this happen by design or it is an issue?

[radding]

Hello! I believe that we are talking on LinkedIn about this issue.

Do you have example code I could look at? This seems to be a bug in the implementation. Do you have reproduction steps?

[dannwise]

Hey @radding, you're correct. I made a sample project to show you the issue since the real project has a lot of business logic that would only be in the way for this purpose. I'll be giving detailed info below but you may also find the project here: https://github.com/dannwise/ghost

Before anything, I feel the need to say that I'm using a view here only for the purpose of reproducing the issue without too much work, but this happens consistently inside any class method. I've already tried to use a custom manager class to trigger the event's dispatch, as well as calling it inside model's save() method and other view's methods. I have also tried chaining events by calling the Dispatch() of an Event B from within the handle() method of a listener subscribed to an Event A. In all the described scenarios, the result is the same.

Django Version: 3.0.6 Python Version: 3.6.5 No other libraries installed besides celery and its dependencies; django-event-system and gevent.

App Structure:

ghost ├── ghost │ ├── __init__.py │ ├── asgi.py │ ├── settings.py │ ├── urls.py │ └── wsgi.py ├── ghosted │ ├── __init__.py │ ├── admin.py │ ├── (...) │ ├── events.py │ ├── listeners.py │ └── views.py ├── templates │ └── index.html ├── manage.py

ghost - urls.py

from django.contrib import admin
from django.urls import path, include
from ghosted.views import IndexView

urlpatterns = [
    path('admin/', admin.site.urls),
    path('', IndexView.as_view(), name='index'),
]

ghosted - events.py

from events import Event


class MyEvent(Event):
    def __init__(self, string):
        self.string = string
        print("MyEvent Initialized")

ghosted - listeners.py

from events import EventListener
from .events import MyEvent


class MyListener(EventListener):
    listensFor = [
        MyEvent,
    ]

    def handle(self, event):
        print("Caught a event: {}".format(event))

ghosted - views.py

from django.views import generic
from .events import MyEvent

def event_caller():
    MyEvent.Dispatch("Just a sample event")

# Dispatching the event here works, because is not inside any class.
# By calling the event here, the event will display on console before System Check results
# To test it, uncomment the following line and start the server
# event_caller()


class IndexView(generic.ListView):
    template_name = 'index.html' # this template can be just a blank html, since it is here only for testing purposes
    context_object_name = 'context'

    def get_queryset(self):
        return None

    def get_context_data(self, *args, **kwargs):
        context = super(IndexView, self).get_context_data(*args, **kwargs)

        # Dispatching the event here doesn't work because is inside a class method.
        # Here the event will be dispatched but not listened to.
        # To test it, uncomment the following line and navigate to http://localhost:8000
        # event_caller()

        return context

Here at the views is where everything is acctually happening. By uncommenting event_caller() you'll be able to see that when the function is called from within a class method the listener's handle method will not be triggered.

This is the console output when calling event_caller() outside class methods:

And this is the console output when calling event_caller() within a class method, which in this case is get_context_data():

As you may see, in both the examples the __init__ method of MyEvent is triggered but only on the first one MyListener catches the event dispatch and executes the handle method.

In the real project I was able to bypass this by delegating MyEvent.Dispatch("somehting...") to a celery task function. So, calling a celery task within a class method and then calling MyEvent.Dispatch() from the task function worked fine. I can only think that when the Dispatch() is called with some reference to any class, something is preventing the listener to work correctly, and celery ends up with the part of breaking such reference to said class. But this is mere speculation on my side.

An important remark about the celery solution is that MyListener had to be imported within the task file even though it is not directly used there. If the handle method of the listener has any logic into it and it is not imported in the task file, the task succeeds but the logic contained in the handle method doesn't seem to be executed.

I did not configure celery for the sample project provided here

tasks.py

from celery import shared_task
from .events import MyEvent
from .listeners import MyListener # <~ important import, even though not used directly

@shared_task(queue='events_queue')
def event_dispatcher(message):
    MyEvent.Dispatch(message)

If there's anything else I can provide in order to help you with that, let me know.

[radding]

So when this happens, its usually because the Gevent event loop never goes to the greenlet that processes and events, so the event is never handled.

A few things to try here:

First, call monkey_patch all as the first line in manage.py. This is usually fine, but sometimes things don't play well with the gevent monkey patched version of things, so make sure you test thoroughly. Also use the Gevent Wsgi server. This will make django look to the event loop after a request is handle. You can see an example on this stack overflow page: https://stackoverflow.com/questions/10964571/how-to-combine-django-plus-gevent-the-basics

Next you can either try using DispacthAsync instead of Dispatch or use the ClearQueue method. DispacthAsync and Dispatch are essentially the same Function (and actually DispatchAsync calls Dispatch the Difference is that DispatchAsync calls gevent.sleep immediately, which forces gevent to look at the queue immediately.

The ClearQueue method, which ensures that all events are processed out of the queue. On this method there is the EnsureQueueIsClear helper decorator which basically calls ClearQueue after your function is called: https://github.com/radding/django-event-system/tree/master#clearing-the-queue.

Word of warning on ClearQueue: its been a minute since I thought about that specific code, but reading it now looks like there may be a bug where it constantly gets things out of the queue and just discards them, so it wont dispatch the event. Perhaps I should change that to Peek instead, but my tests are not failing so, I am not 100% sure that its broken. If you do try ClearQueue, let me know please :)

If none of these work, then I think that yes there is a bug that I would need to look deeper into. If you don't want to use the Gevent wsgi server or the monkey_patch all, then ClearQueue should work for you, but it may slow down your application response times since it has to clear the queue before returning the response.