Transaction support, maybe?

[mpasternak]

Hello,

not trying to come as rude/aggressive/criticizing... just found out your package and taking a look how do you do things,

What are your thoughts on transaction support?

Because, in this model, when you save things in post_save signal, if the transaction gets rolled back... the web hooks already been fired.

My idea was to mark saved objects as "dirty" in the on_commit handler... something similar to your store_events... then if the task queue picked those "marked-as-dirty" entries, it checked for the presence of the original object in the table (via a GenericField)... it had some delay in case the object did not exist yet... etc.

This is not an actual issue, I hope you and your users are happy with the approach you chosen. But this is pretty hard to synchronize a webhook and an in-database transaction. Just trying to see if you had some thoughts on it.

Thanks,

[danihodovic]

Hello Michal. I'll admit that I don't think the package has been battle-tested yet. I've deployed it on one client site and one project that doesn't see too much traffic (https://django.wtf).

What do you think about using the transaction.on_commit callback instead of a custom solution? https://stackoverflow.com/a/52703242/2966951

[mpasternak]

I love on_commit, I use it in my code all the time. The real question is somewhere else, I think, the real question is: "how much coupling do you need in your system actually, perhaps it could be solved another way"

As per what do I think... after creating database objects (after actual INSERT/UPDATE) I would use database triggers to fill a table of "dirty" objects (I do something like this in my (pretty advanced) fork of django-denorm, https://github.com/mpasternak/django-denorm-iplweb/). This way it is even better than on_commit because if the server fails right after committing the data to the database but before calling events in on_commit - well, you're left in inconsistent state. As per database triggers -- I know and use PostgreSQL, I guess MS SQL or Oracle would handle those no problem, I am not sure about MySQL. And, it looks like you would have to write perhaps not-so-portable SQL in the migrations, for every database.

But, if you inserted a row to a table, you then have a row in another table when the transaction was successful.

Then a worker should look for changes in the "dirty" table (in django-denorm-iplweb I use PostgreSQL LISTEN/NOTIFY mechanism, so I get immidately notified about changes, I am not performing a SELECT every 5 secs or so, I get immediate notification when the table changes). Then, the worker should execute webhooks, pretty much as you already do in your code.

What if webhook fails? Should it have any effect on the database? Are we trying to create a distributed transaction or are we just doing a simple notification mechanism? Who knows. That's the real question, how much coupling do you need.

PostgreSQL database triggers ensure data integrity at the database backend level. Task queue like RabbitMQ used via Celery ensures message delivery and can get the result.

There's a lot to such architectures and as many of those ideas are site-specific it is hard to come up with a general solution. I just realized, that my own solution actually pulls PostgreSQL into the whole image. So I guess it won't fit everybody.

Thanks,

[danihodovic]

In general I prefer to solve problems once they become common enough to be worth solving.

I also prefer writing as little code as possible to achieve what I want to do. In this case I prefer to rely on built-in Django transaction helpers rather than building a custom solution. I would have to test and maintain myself a custom implementation myself; I trust Django maintainers more than I trust my own code.

If you want to submit a PR for the on_commit helper it would be welcome :) .