sagemaker-python-sdk icon indicating copy to clipboard operation
sagemaker-python-sdk copied to clipboard

ModelExplainabilityMonitor cannot execute one-time monitoring job due to lack of `data_analysis_start_time`/`data_analysis_end_time` in `_create_monitoring_schedule_from_job_definition`

Open hightensan opened this issue 6 months ago • 0 comments

Describe the bug SageMaker Model Monitor provides one-time monitoring job schedule[Link]. However, ValueError are raised when specifying CronExpressionGenerator.now().

To reproduce

Using SageMaker-Monitoring-Feature-Attribution-Drift-for-Endpoint.ipynb:

  • Change schedule_expression to CronExpressionGenerator.now()
  • Specify data_analysis_start_time and data_analysis_end_time in ModelExplainabilityMonitor's create_monitoring_schedule
type(model_explainability_monitor) # sagemaker.model_monitor.clarify_model_monitoring.ModelExplainabilityMonitor

schedule_expression = sagemaker.model_monitor.CronExpressionGenerator.now()

model_explainability_monitor.create_monitoring_schedule(
    analysis_config=model_explainability_analysis_config,
    endpoint_input=sagemaker.model_monitor.EndpointInput(
        endpoint_name=endpoint_name,
        destination="/opt/ml/processing/input/endpoint",
        features_attribute=features_jmespath,
        inference_attribute=confidence_score_jmespath,
    ),
    output_s3_uri=monitor_output_s3_uri,
    schedule_cron_expression=schedule_expression,
    data_analysis_start_time="-PT2H",
    data_analysis_end_time="-PT1H"
)

Error Message:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[63], line 3
      1 schedule_expression = sagemaker.model_monitor.CronExpressionGenerator.now()
----> 3 model_explainability_monitor.create_monitoring_schedule(
      4     analysis_config=model_explainability_analysis_config,
      5     endpoint_input=sagemaker.model_monitor.EndpointInput(
      6         endpoint_name=endpoint_name,
      7         destination="/opt/ml/processing/input/endpoint",
      8         features_attribute=features_jmespath,
      9         inference_attribute=confidence_score_jmespath,
     10     ),
     11     output_s3_uri=monitor_output_s3_uri,
     12     schedule_cron_expression=schedule_expression,
     13     data_analysis_start_time="-PT2H",
     14     data_analysis_end_time="-PT1H"
     15 )

File ~/anaconda3/envs/python3/lib/python3.10/site-packages/sagemaker/model_monitor/clarify_model_monitoring.py:1104, in ModelExplainabilityMonitor.create_monitoring_schedule(self, endpoint_input, analysis_config, output_s3_uri, constraints, monitor_schedule_name, schedule_cron_expression, enable_cloudwatch_metrics, batch_transform_input, data_analysis_start_time, data_analysis_end_time)
   1102 # create schedule
   1103 try:
-> 1104     self._create_monitoring_schedule_from_job_definition(
   1105         monitor_schedule_name=monitor_schedule_name,
   1106         job_definition_name=new_job_definition_name,
   1107         schedule_cron_expression=schedule_cron_expression,
   1108     )
   1109     self.job_definition_name = new_job_definition_name
   1110     self.monitoring_schedule_name = monitor_schedule_name

File ~/anaconda3/envs/python3/lib/python3.10/site-packages/sagemaker/model_monitor/model_monitoring.py:1562, in ModelMonitor._create_monitoring_schedule_from_job_definition(self, monitor_schedule_name, job_definition_name, schedule_cron_expression, data_analysis_start_time, data_analysis_end_time)
   1559 message = "Creating Monitoring Schedule with name: {}".format(monitor_schedule_name)
   1560 logger.info(message)
-> 1562 self._check_monitoring_schedule_cron_validity(
   1563     schedule_cron_expression=schedule_cron_expression,
   1564     data_analysis_start_time=data_analysis_start_time,
   1565     data_analysis_end_time=data_analysis_end_time,
   1566 )
   1568 monitoring_schedule_config = {
   1569     "MonitoringJobDefinitionName": job_definition_name,
   1570     "MonitoringType": self.monitoring_type(),
   1571 }
   1572 if schedule_cron_expression is not None:

File ~/anaconda3/envs/python3/lib/python3.10/site-packages/sagemaker/model_monitor/model_monitoring.py:1536, in ModelMonitor._check_monitoring_schedule_cron_validity(self, schedule_cron_expression, data_analysis_start_time, data_analysis_end_time)
   1531 message = (
   1532     "Both data_analysis_start_time and data_analysis_end_time are required "
   1533     "for one time monitoring schedule "
   1534 )
   1535 _LOGGER.error(message)
-> 1536 raise ValueError(message)

ValueError: Both data_analysis_start_time and data_analysis_end_time are required for one time monitoring schedule 

Expected behavior Create one-time monitoring job schedule.

ModelBiasMonitor (Model Bias Monitoring Job) can create one-time monitoring job schedule, because data_analysis_start_time and data_analysis_end_time are correctly passed in ModelBiasMonitor's _create_monitoring_schedule_from_job_definition function. ModelBiasMonitor's _create_monitoring_schedule_from_job_definition

However, ModelExplainabilityMonitor's _create_monitoring_schedule_from_job_definition function does not pass data_analysis_start_time and data_analysis_end_time params, consequently ValueError was raised in _check_monitoring_schedule_cron_validity called in _create_monitoring_schedule_from_job_definition. data_analysis_start_time and data_analysis_end_time params are defined in ModelExplainabilityMonitor's create_monitoring_schedule args. ModelExplainabilityMonitor's _create_monitoring_schedule_from_job_definition ModelMonitor's _check_monitoring_schedule_cron_validity

In this commit, codes to support for one time monitoring schedule were merged, however ModelExplainabilityMonitor's one time monitoring schedule looks missed (data_analysis_start_time and data_analysis_end_time as ModelExplainabilityMonitor's create_monitoring_schedule args are added).

System information

  • SageMaker Python SDK version: v2.0.8 (latest)
  • Framework name (eg. PyTorch) or algorithm (eg. KMeans): N/A
  • Framework version: N/A
  • Python version: 3.10.13 | packaged by conda-forge
  • CPU or GPU: CPU
  • Custom Docker image (Y/N): N

Additional context N/A

hightensan avatar Feb 16 '24 15:02 hightensan