Not able to have the value of the output_path in TensorFlow estimator as pipeline parameter

[mouhannadali]

Describe the bug
I am not able to have the value of the output_path in TensorFlow estimater as pipeline parameter

To reproduce
I created the following Sagemaker workflow parameters:

training_instance_type = ParameterString(name="TrainingInstanceType", default_value="ml.c5.2xlarge")
training_instance_count = ParameterInteger(name="TrainingInstanceCount", default_value=1)
use_spot_instances = ParameterBoolean(name="UseSpot_Instances", default_value=False)
train_output_s3 = ParameterString(name="TrainOutputS3Path", default_value="s3://bucketname/key/")

created the following TF estimator:

estimator = TensorFlow(
    sagemaker_session=sess,
    entry_point="unet.py",
    source_dir="scripts/",
    role=role,
    instance_count=training_instance_count,
    instance_type=training_instance_type,
    framework_version="2.1",
    py_version="py3",
    use_spot_instances=use_spot_instances,
    output_path=train_output_s3,
    container_log_level=logging.WARNING,
    hyperparameters=hyperparameters,
)

created the training step:

input_path = "s3://bucketname-2/key-2/"
inputs = TrainingInput(s3_data=input_path)

step_train = TrainingStep(
    name="pytorch-train",
    estimator=estimator,
    inputs=inputs,
)

and finally created the pipeline and tried to print the pipeline.definition()

pipeline = Pipeline(
    name=pipeline_name,
    parameters = [training_instance_type, training_instance_count,use_spot_instances, train_output_s3],
    steps=[step_train],
    sagemaker_session=sess,
)

pipeline.definition()

Expected behavior
The pipeline should be created without any problem

Screenshots or logs

Additional context
Now if I hardcode the output_path in the estimator then it is working correctly:

estimator = TensorFlow(
    sagemaker_session=sess,
    entry_point="unet.py",
    source_dir="scripts/",
    role=role,
    instance_count=training_instance_count,
    instance_type=training_instance_type,
    framework_version="2.1",
    py_version="py3",
    use_spot_instances=use_spot_instances,
    output_path="s3://bucketname/key/",
    container_log_level=logging.WARNING,
    hyperparameters=hyperparameters,
)

[qidewenwhen]

@mouhannadali thanks for using SageMaker Pipeline!

I'm not able to reproduce the issue based on the code snippets in this issue. Could you please do the following and see if the issue still persists?

• Update the sagemaker SDK to the latest version
  • As we have a breaking change in v2.92.0, you may see a new issue as described in Exception: instance_type should not be a pipeline variable (<class 'sagemaker.workflow.parameters.ParameterString'>) #3141. Please follow the solutions in the comment of issue3141 to get rid of it. To be more specific, please either 1) remove the parameterized training_instance_type and hard code it as plain string OR 2) give image_uri to the estimator constructor

Please let us know if you have any questions.

[mouhannadali]

Hi @qidewenwhen thanks for the replay. I did update the SDK package to sagemaker-2.92.2 and hard coded the training_instance_type. Now I am getting the following error:

[qidewenwhen]

Thanks @mouhannadali for the quick update!
Yea this is what I saw and we had a related issue reported: #3142

This pending PR should be able to fix it: #3146

[mouhannadali]

Thanks @qidewenwhen for the fast reply. one more question: is there a notebook/script example (best practice) where it shows how to pass all the parameters of an estimator (Tesorflow estimator) including hyperparameters as training pipeline parameters?

[qidewenwhen]

Thanks @qidewenwhen for the fast reply. one more question: is there a notebook/script example (best practice) where it shows how to pass all the parameters of an estimator (Tesorflow estimator) including hyperparameters as training pipeline parameters?

We have this sample notebook with Tensorflow estimator: https://github.com/aws/amazon-sagemaker-examples/blob/main/sagemaker-pipelines/tabular/tensorflow2-california-housing-sagemaker-pipelines-deploy-endpoint/tensorflow2-california-housing-sagemaker-pipelines-deploy-endpoint.ipynb

[mouhannadali]

It is working now after updating to the last SDK version. Thanks for the support