Introduce reporting methods

[mimir-d]

Issue by insomniacslk Thursday Feb 06, 2020 at 14:11 GMT Originally opened as https://github.com/facebookincubator/contest/issues/33

---

Currently every reporter is responsible for result calculation and delivery of the report. However some tasks are common to many reporters (e.g. sending results over e-mail), and it would be easier on the user if they were supported out of the box. I propose to add a Methods section in the reporters that will allow to specify multiple ways of delivering a report.

This should send the JSON report via the specified methods, and also allow (optional) filtering and templating so users can format the report as desired.

    "Reporting": {
        "RunReporters": [
            {
                "Name": "TargetSuccess",
                "Parameters": {
                    "SuccessExpression": ">80%"
                },
                "Methods": [
                    {
                        "name": "email",
                        "recipient": "[email protected]"
                    }
                }
            ]
        ],
        ...
    }

[mimir-d]

Comment by xaionaro Wednesday Mar 04, 2020 at 10:08 GMT

---

@insomniacslk , @marcoguerri , @pmazzini :

We need to get an agreement on how will Report-s look like to communicate between ReportGenerator and ReportDeliverer.

Let's consider this variant:

type Report struct {
    Title string
    HumanReadableSummary string
    ReporterName string
    IsSuccess bool
    ReportTime time.Time
    Custom CustomFields
}

type CustomFields map[string]CustomField
type CustomField interface{
    Tags() []string
    String() string
}

func (customFields CustomFields) CollectForTag(tag string) CustomFields {
    ...
}
func (customFields CustomFields) String() string {
    report := make([]string, 0, len(customFields))
    for key, customField := range customFields {
        report = append(report, fmt.Sprintf("%s: %s", key, customField))
    }
    return strings.Join(report, "\n")
}

So some deliverer may be able to use something like this:

_ = Report.Custom.CollectForTag(`security`).String()

Also:

for key, customField := range customFields {
    switch customField := customField.(type) {
    case someFieldTypeWeKnow:
        toBeSent.ConsiderMagicField(key, customField.SomeInternalData.SomeInternalField)
    }
}
_ = toBeSent.String()

Or:

switch report.ReporterName {
case "someReporterNameWeKnow", "anotherSimilarReporter":
    _ = customFields[`someField`].String()
}

It's not a strong opinion. Just a wild fantasy. Feel free to crush this :)

---

Also it would be nice to make this custom fields reusable. So it would be nice to think a little bit about file tree hierarchy. What do you think about just to make it next way? --

• plugins/reportgenerator
• plugins/reportdeliverer
• plugins/reportcustomfield

Again feel free to crush it (:

---

Generator and Deliverer interfaces:

type ReportGenerator interface {
    ParseConfig(map[string]interface{}) (interface{}, error)

    // it's also allowed to return `(nil, nil)` if no error, but also there's nothing to report about.
    GenerateReport(
        ctx       context.Context,
        config    interface{},
        runStatus *job.RunStatus /* const: don't modify internals */,
        ev        testevent.Fetcher,
    ) (*job.Report, error)
}

type ReportDeliverer interface {
    ParseConfig(map[string]interface{}) (interface{}, error)

    DeliverReports(
        ctx    context.Context,
        config interface{},
        report *job.JobReport /* const: don't modify internals */,
    ) error
}

---

Job description:

    "Reporting": {
        "ReportGenerators": [
            {
                "Name": "TargetSuccess",
                "Parameters": {
                    "SuccessExpression": ">80%"
                }
            },
            {
                "Name": "Noop"
            }
        ],
        "ReportDeliverers": [
            {
                "Name": "slack",
                "Parameters": {"URL": "https://my-workspace.slack.com/somemagic"}
            },
            {
                "Name": "gerrit",
                "Parameters": {"URL": "https://my-server/"}
            }
        ]
    }

---

Also it would be nice to have a contact with potential users to know what do they think about it.