Elementary Tests generating False Errors instead of Warning

[AnnuJoshi]

Describe the bug We are using elementary test for our incrementals models. We observed that when we look back for last 2 weeks of data for the initials runs, elementary gives us an error, instead it should throw a warning to indicate that data is not present. In any case if the data is not present it should be a warning not an error.

To Reproduce Steps to reproduce the behavior:

1 . Create a test with below for any sample time series data
config: elementary: anomaly_sensitivity: 2 # Anomaly detection threshold timestamp_column: dt # name of the column that contains the timestamp or datetime information for each data point days_back: 30 # number of days of historical data to consider min_training_set_size: 14 # minimum number of data points required in the training set

tests:
  - elementary.volume_anomalies:
      seasonality: 'day_of_week' 
Now run the test incrementally each day. 

Expected behavior If the data is not present for last day it should give a warning and test should pass. Instead we are getting an error.

Environment (please complete the following information):

• edr Version: Version: 0.8.1,
• dbt package Version: 1.4.7,

Additional context Add any other context about the problem here.

[AnnuJoshi]

Error we get it like below:

Failure in test elementary_volume_anomalies_name_of_the_model Got 3 results, configured to fail if != 0 this is the error we get Sometimes its Got 1 result, configured to fail if != 0

[haritamar]

Hi @AnnuJoshi ! Thanks for opening this issue and sorry for the super delayed response, we are currently in the process of reviewing our open issues and improving our process there. Since the issue has been open for over 3 months I'm going to go ahead to close it - please feel free to re-open if still relevant.

Thanks! Itamar