Add log subscribers and support for JSON logs

[wshihadeh]

Extend delayed jobs logging to be able to support json format using log subscribers. Example of the output

{"level":"INFO","timestamp":"2019-07-14 22:47:46 +0200","log_event":"starting","message":"2019-07-14T22:47:46+0200: [Worker(host:mb180.local pid:84605)] Starting job worker","dj_worker":"host:mb180.local pid:84605","dj_time":"2019-07-14T22:47:46+0200"}
{"level":"INFO","timestamp":"2019-07-14 22:47:46 +0200","log_event":"running","message":"2019-07-14T22:47:46+0200: [Worker(host:mb180.local pid:84605)] Job User#dummy_dj (id=7) RUNNING","name":"User#dummy_dj","id":7,"queue":null,"dj_worker":"host:mb180.local pid:84605","dj_time":"2019-07-14T22:47:46+0200"}
{"level":"INFO","timestamp":"2019-07-14 22:47:46 +0200","log_event":"completed","message":"2019-07-14T22:47:46+0200: [Worker(host:mb180.local pid:84605)] Job User#dummy_dj (id=7) COMPLETED after 0.0026","runtime":"0.0026","name":"User#dummy_dj","id":7,"queue":null,"dj_worker":"host:mb180.local pid:84605","dj_time":"2019-07-14T22:47:46+0200"}
{"level":"INFO","timestamp":"2019-07-14 22:47:46 +0200","log_event":"no_jobs_available","message":"2019-07-14T22:47:46+0200: [Worker(host:mb180.local pid:84605)] No more jobs available. Exiting","dj_worker":"host:mb180.local pid:84605","dj_time":"2019-07-14T22:47:46+0200"}
{"level":"INFO","timestamp":"2019-07-14 22:48:01 +0200","log_event":"running","message":"2019-07-14T22:48:01+0200: [Worker(host:mb180.local pid:84605)] Job User#dummy_dj (id=8) RUNNING","name":"User#dummy_dj","id":8,"queue":null,"dj_worker":"host:mb180.local pid:84605","dj_time":"2019-07-14T22:48:01+0200"}
{"level":"INFO","timestamp":"2019-07-14 22:48:01 +0200","log_event":"completed","message":"2019-07-14T22:48:01+0200: [Worker(host:mb180.local pid:84605)] Job User#dummy_dj (id=8) COMPLETED after 0.0033","runtime":"0.0033","name":"User#dummy_dj","id":8,"queue":null,"dj_worker":"host:mb180.local pid:84605","dj_time":"2019-07-14T22:48:01+0200"}
{"level":"INFO","timestamp":"2019-07-14 22:48:01 +0200","log_event":"no_jobs_available","message":"2019-07-14T22:48:01+0200: [Worker(host:mb180.local pid:84605)] No more jobs available. Exiting","dj_worker":"host:mb180.local pid:84605","dj_time":"2019-07-14T22:48:01+0200"}
{"level":"INFO","timestamp":"2019-07-14 22:49:28 +0200","log_event":"exiting","message":"2019-07-14T22:49:28+0200: [Worker(host:mb180.local pid:84605)] Exiting...","dj_worker":"host:mb180.local pid:84605","dj_time":"2019-07-14T22:49:28+0200"}

[wshihadeh]

@albus522 can you review this please ?

[derosm2]

Would like to see JSON formatted logs!

[wshihadeh]

@albus522 do you think that this feature is not relevant for delayed jobs ? should I close the PR ?

[caljess599]

OMG this would be heaven!

[s-andringa]

Would love to see this integrated. It is exactly what we need to feed Datadog with DJ logs that can be parsed w/o any extra configuration. In case JSON logging is considered too specific to integrate into DJ: having just the ActiveSupport::Notifications setup that this PR is built around would already be very useful. So people can easily attach their own JSON log subscriber, or whatever other type of subscriber. (Thanks @wshihadeh.)

[booleanbetrayal]

Would also love to see this merged. Would help with unifying log conventions with other Kubernetes logging configurations in our clusters.

[booleanbetrayal]

@wshihadeh - do you mind pulling master into this branch so people can direct-link this fork and use in Rails 6?

[wshihadeh]

@booleanbetrayal done , although it seems that this project is not active anymore.

cc: @albus522 is that the case ?

[wshihadeh]

@albus522 do you have any objections for this one ? should I close it ?