probability_threshold_attribute in ModelQualityCheckConfig cannot be a PipelineVariable

[leviBernadine]

Describe the bug

sagemaker-python-sdk/src/sagemaker/workflow/quality_check_step.py

Line 101 in 610d00f

probability_threshold_attribute (str or PipelineVariable): Threshold to

As an user of sagemaker pipeline, I want to use the QualityCheckStep for monitoring my model concept drift.
When I want to configure the ModelQualityCheckConfig I cannot set the probability_threshold_attribute attribute with a PipelineVariable (tested on either sagemaker.workflow.properties.Properties and sagemaker.workflow.parameters.Parameter)

To reproduce

from sagemaker.workflow.parameters import ParameterString
from sagemaker.workflow.quality_check_step import ModelQualityCheckConfig, QualityCheckStep

parameter_label_used_threshold = ParameterString(
    name="used_threshold",
    default_value="0.5",
)  
model_quality_check_config = ModelQualityCheckConfig(
    problem_type = "BinaryClassification",
    probability_attribute= ...,
    probability_threshold_attribute=parameter_label_used_threshold,
    ground_truth_attribute= ...,     
    baseline_dataset=...,
    dataset_format= ...,
    output_s3_uri=...
)

baseline_monitor_model_step_process = QualityCheckStep(
        name=...,
        skip_check=...,
        register_new_baseline=...,
        quality_check_config=model_quality_check_config,
        check_job_config=...,
        model_package_group_name=...,
        cache_config=..,
        supplied_baseline_statistics=...,
        supplied_baseline_constraints=...,
    )

Will raise error

File /opt/conda/lib/python3.10/site-packages/sagemaker/workflow/quality_check_step.py:426, in QualityCheckStep._generate_baseline_processor(self, baseline_dataset_input, baseline_output, post_processor_script_input, record_preprocessor_script_input)
    415     probability_attribute = (
    416         str(quality_check_cfg.probability_attribute)
    417         if quality_check_cfg.probability_attribute is not None
    418         else None
    419     )
    420     ground_truth_attribute = (
    421         str(quality_check_cfg.ground_truth_attribute)
    422         if quality_check_cfg.ground_truth_attribute is not None
    423         else None
    424     )
    425     probability_threshold_attr = (
--> 426         str(quality_check_cfg.probability_threshold_attribute)
    427         if quality_check_cfg.probability_threshold_attribute is not None
    428         else None
    429     )
    430     normalized_env = ModelMonitor._generate_env_map(
    431         env=self._model_monitor.env,
    432         dataset_format=quality_check_cfg.dataset_format,
   (...)
    442         probability_threshold_attribute=probability_threshold_attr,
    443     )
    445 return Processor(
    446     role=self._model_monitor.role,
    447     image_uri=self._model_monitor.image_uri,
   (...)
    459     network_config=self._model_monitor.network_config,
    460 )

File /opt/conda/lib/python3.10/site-packages/sagemaker/workflow/entities.py:86, in PipelineVariable.__str__(self)
     84 def __str__(self):
     85     """Override built-in String function for PipelineVariable"""
---> 86     raise TypeError(
     87         "Pipeline variables do not support __str__ operation. "
     88         "Please use `.to_string()` to convert it to string type in execution time"
     89         "or use `.expr` to translate it to Json for display purpose in Python SDK."
     90     )

TypeError: Pipeline variables do not support __str__ operation. Please use `.to_string()` to convert it to string type in execution timeor use `.expr` to translate it to Json for display purpose in Python SDK.

Expected behavior
To set probability_threshold_attr with a pipelineVariable.

System information
SageMaker Python SDK version: 2.194.0
Python version: 3.10.6

[qidewenwhen]

Adding a note:
The error was from here:

sagemaker-python-sdk/src/sagemaker/workflow/quality_check_step.py

Lines 425 to 429 in 610d00f

	probability_threshold_attr = (
	str(quality_check_cfg.probability_threshold_attribute)
	if quality_check_cfg.probability_threshold_attribute is not None
	else None
	)

This is a bug. We may need to add the following fix but it needs proper integ test to prove probability_threshold_attribute is parametrizable

     if not quality_check_cfg.probability_threshold_attribute: 
           probability_threshold_attr = None
     elif  is_pipeline_variable(quality_check_cfg.probability_threshold_attribute):
           if isinstance(quality_check_cfg.probability_threshold_attribute, Parameter) and not isinstance(quality_check_cfg.probability_threshold_attribute, ParameterString)
                throw ValueError("... probability_threshold_attribute cannot be Parameter other than ParameterString")

           logger.warn("probability_threshold_attribute's runtime value must be string type ...")
           probability_threshold_attr =  quality_check_cfg.probability_threshold_attribute
    else:
           probability_threshold_attr = str(quality_check_cfg.probability_threshold_attribute)

[qidewenwhen]

Tracking the fix in backlog

[leviBernadine]

Thank you very much
Do you have any updates on this issue please ?

[qidewenwhen]

Hi @leviBernadine, sorry for the delay. I just published a fix for the issue.
Reaching out to the team to review it.