Greater extensibility for fourkeys

[landonpratt]

My organization is looking at implementing fourkeys across all our teams, and have run into a couple of competing requirements:

1. We need to support some "custom" event sources, which it doesn't make sense to contribute back to fourkeys. For instance, a majority of production deployments occur via a system that was developed in-house.
2. We want to be able to pick up fourkeys improvements, without requiring the burden of manually incorporating those changes into our own fork.

To further illustrate some of the difficulties:

• Adding a new event source requires changes to the event handler
• Adding a new source requires code changes to the install script (and the new terraform work as well I believe)
• Adding a new source requires code changes to the BigQuery queries shared by all sources

My team has been discussing some ideas of how we could make adding new sources easier, with a focus on configuration rather than code. We are now hoping to get some feedback on these ideas, or other suggestions about how we could satisfy our competing requirements.

Config file as input to setup script

The first part of the solution we have discussed would be to modify the install script to accept a file that would define the four keys configuration. The install script could then use this to run the appropriate gcloud commands (or create the appropriate terraform resources).

This could also potentially be used to configure the event handler, which leads to the next idea.

Generic Event Handler

We would like to eliminate the need to modify the event handler when adding a new source. This could be done by having the event handler read in configuration (possibly from an environment variable or file packaged in the container). This might look something like this:

routes:
  - newsource:
    topic: newsource      # name of the topic to forward messages to
    verification: simple_token      # verification function to use
    criteria: user_agent      # criteria for forwarding message. in this case, user_agent that matches topic name

Source Specific Queries

Instead of including the logic from all sources in each BQ query, only the logic for a single source would be included. These would then be UNIONED to produce the view/table used by the dashboard.

SELECT * FROM newsource_deployments
UNION
SELECT * FROM othersource_deployments

It is likely that the install script would need to generate the queries that would combine source specific tables.

This would line up with an approach that has already been discussed on the slack channel by @dinagraves and @twoodhouse

Those are the basic ideas so far. Please provide thoughts/feedback and let us know if this is a direction fourkeys would like to go. We are looking for opportunities to contribute as soon as we can get through the CLA process.