dashboard
dashboard copied to clipboard
Webhook endpoint to log pipeline events
This is a proposal for a way to update metadata in the dashboard when something happens in the core environment. The use case that motivated this is @zstumgoren's idea of incorporating results crawling into the pipeline and needing some way to record when the crawl changes. This also might be useful to track when results are (re)published.
I'd classify this feature as nice to have. While nothing in the modifications are crazy or unconventional, it's still a significant chunk of work.
In the dashboard codebase:
- Add a model to track the status of different workflow actions.
class ElectionEventLog(models.Model):
EVENT_CHOICES = (
('load', "Load Results"),
('bake_raw', "Bake Raw"),
('bake', "Bake Clean"),
('scrape', "Scrape"),
)
state = models.CharField(max_length=2, db_index=True, choices=US_STATES)
event = models.CharField(max_length=15, choices=EVENT_CHOICES)
event_properties = JSONField()
timestamp = models.DateTimeField()
target_content_type = models.ForeignKey(ContentType)
target_id = models.PositiveIntegerField()
target = GenericForeignKey('target_content_type', 'target_id')
Note that this design uses Django's contenttype framework and a GenericForeignKey field to flexibly link to other model instances. In most of these cases this will be an Election.
Also use a JSONField implementation (we could roll our own or use an existing one) to allow the storing of different properties for each event type. This avoids field creep and lots of schema migrations.
- Add a resource,
ElectionEventLogResource
, to the Tastypie-based API that supports POST requests. Enable API key authentication for this endpoint. We probably want to use signals to make it easy to tie in other dashboard actions beside log creation.
In the openelex-core package:
- Update
openelex.api.base
to provide apost
method to post to the event log endpoint. If this event logging happens a lot, it might make sense to create apost_log()
utility function to simplify the building of requests to our endpoint. - Update various openelex subtasks to send signals when they complete. The publish task already does this.
- Set up signal handlers that post to the logging webhook. This is a little hairy because we don't want to hit the webhook for every user when they're testing. So, we'll need to toggle this with some kind of setting.