Running multiple Transform jobs with same Transformer uses same output_path for all transform jobs

[andremoeller]

Please fill out the form below.

System Information

• Framework (e.g. TensorFlow) / Algorithm (e.g. KMeans): TensorFlow
• Framework Version: 1.12.0
• Python Version: 3
• CPU or GPU: CPU
• Python SDK Version: sagemaker==1.16.3
• Are you using a custom image: No.

Describe the problem

If I run more than one Transform Job using a Transformer, the outputs are written to the same S3 output path, even though I'm running different transform jobs. So outputs can collide and be overwritten or I'll end up with outputs from multiple jobs in the same S3 output path

Minimal repro / logs

I'm running tensorflow_batch_transform_mnist.ipynb.

You can reproduce this by doing:
transformer.transform([your batch s3 input path], content_type='text/csv')
print(transformer.output_path)

transformer.transform([some other batch s3 input path], content_type='text/csv')
print(transformer.output_path)

And the transformer.output_path will be the same (but the jobs are different)

[nadiaya]

So current behavior is to pass to the platform s3 location provided by the user or create a job_id based folder inside default s3 bucket (similar to what SageMaker platform does for training output):
https://github.com/aws/sagemaker-python-sdk/blob/master/src/sagemaker/transformer.py#L111-L112

Moving this issue to the platform team to answer why the collision use case is not addressed at the platform level.

[andremoeller]

The issue is:

If I run one job with default generated name "sagemaker-tensorflow-1", that gets written to "s3:/...../sagemaker-tensorflow-1/".

If I run another job with the same transformer with default generated name "sagemaker-tensorflow-2", that gets written to "s3:/...../sagemaker-tensorflow-1", instead of "s3:/...../sagemaker-tensorflow-2", which means Python SDK tells the platform to write output from both jobs to the same output location. I don't think that's expected (or good for users), regardless of what the platform does -- wouldn't it make more sense to default to telling the platform to write to a prefix with the job name?

[zmjjmz]

I'd like to add that I think this is a little weird from an API perspective, since the CreateTransformJob that the Transformer.transform method presumably would correspond to takes a TransformOutput as an argument. Is there a specific reason that this has to be provided on the creation of the Transformer object?