Skip send-report notifications if all tests passed via --only-on-failures

[spider-man-tm]

Is your feature request related to a problem? Please describe. Currently, the send-report command always sends a Slack notification even when all tests have passed, which creates noise—reminders pile up even though nothing is actually failing.

Describe the solution you'd like Add a new option --only-on-failures (or an equivalent name) to send-report so that, when specified, Slack notifications are skipped only if all tests have passed. Here’s an example implementation:

def send_test_results_summary(
    self,
    days_back: int,
    test_runs_amount: int,
    disable_passed_test_metrics: bool = False,
    bucket_website_url: Optional[str] = None,
    include_description: bool = False,
    only_on_failures: bool = False
) -> bool:
    tests_api = TestsAPI(
        dbt_runner=self.internal_dbt_runner,
        days_back=days_back,
        invocations_per_test=test_runs_amount,
        disable_passed_test_metrics=disable_passed_test_metrics,
    )
    invocations_api = InvocationsAPI(
        dbt_runner=self.internal_dbt_runner,
    )
    invocation = invocations_api.get_test_invocation_from_filter(
        self.selector_filter.to_selector_filter_schema()
    )
    summary_test_results = tests_api.get_test_results_summary(
        filter=self.selector_filter.to_selector_filter_schema(),
        dbt_invocation=invocation,
    )
    all_passed = all(t.status == "passed" for t in summary_test_results)
    if self.slack_client:
        if only_on_failures and all_passed:
            send_succeeded = True
        else:
            send_succeeded = self.slack_client.send_message(
                channel_name=self.config.slack_channel_name,
                message=SlackReportSummaryMessageBuilder().get_slack_message(
                    test_results=summary_test_results,
                    bucket_website_url=bucket_website_url,
                    include_description=include_description,
                    filter=self.selector_filter.to_selector_filter_schema(),
                    days_back=days_back,
                    env=self.config.env,
                    project_name=self.config.project_name,
                ),
            )
    else:
        send_succeeded = False


    self.execution_properties[
        "sent_test_results_summary_successfully"
    ] = send_succeeded
    self.execution_properties[
        "test_results_all_passed"
    ] = all_passed
    self.success = send_succeeded

    if send_succeeded:
        logger.info("Sent test results summary to Slack")

    self.execution_properties["success"] = self.success
    return self.success

Describe alternatives you've considered A clear and concise description of any alternative solutions or features you've considered.

Additional context

• When the flag is not specified, the behavior remains unchanged (notifications are always sent).
• Do not break the existing updates to execution_properties or the return-value logic; when skipping, the method should still be treated as successful (return True).

Would you be willing to contribute this feature? Yes. I can implement the change and add corresponding tests. With your guidance, I will submit a PR.

[NoyaOffer]

Hey @spider-man-tm , thanks a lot for this! Curious to understand, what about using edr monitor? It will only send alerts on failed tests, and can be grouped into one alert with edr monitor --group-alerts-threshold INT.

[spider-man-tm]

@NoyaOffer Thank you! It certainly looks like this will achieve exactly what I wanted. Please feel free to close this issue.

[github-actions[bot]]

This issue is stale because it has been open for too long with no activity. If you would like the issue to remain open, please remove the stale label or leave a comment.