Local deploy failure for MXNet estimator due to non-uniform output uri

[ehsanmok]

Describe the bug

This is related to the already resolved issue #1349 . After using the local mode for training via LocalSession(), deploying the estimator locally repeatedly throws this error:

Click to see the error

algo-1-zu1b2_1  | 2020-03-13 17:36:32,149 [WARN ] W-9001-model-stderr com.amazonaws.ml.mms.wlm.WorkerLifeCycle -     in DEFAULT_MODEL_FILENAMES.items()]))
algo-1-zu1b2_1  | 2020-03-13 17:36:32,149 [WARN ] W-9001-model-stderr com.amazonaws.ml.mms.wlm.WorkerLifeCycle - ValueError: Failed to load model with default model_fn: missing file model-symbol.json.Expected files: ['model-symbol.json', 'model-0000.params', 'model-shapes.json']
algo-1-zu1b2_1  | 2020-03-13 17:36:32,149 [WARN ] W-9001-model com.amazonaws.ml.mms.wlm.BatchAggregator - Load model failed: model, error: Worker died.

I investigated where the issue might be and manually created the MXNetModel that worked with non-local mode but fails in local mode with

---------------------------------------------------------------------------
ClientError                               Traceback (most recent call last)
<timed exec> in <module>()

~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/sagemaker/model.py in deploy(self, initial_instance_count, instance_type, accelerator_type, endpoint_name, update_endpoint, tags, kms_key, wait, data_capture_config)
    440             self.name = "{}{}".format(name_prefix, compiled_model_suffix)
    441 
--> 442         self._create_sagemaker_model(instance_type, accelerator_type, tags)
    443         production_variant = sagemaker.production_variant(
    444             self.name, instance_type, initial_instance_count, accelerator_type=accelerator_type

~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/sagemaker/model.py in _create_sagemaker_model(self, instance_type, accelerator_type, tags)
    175                 /api/latest/reference/services/sagemaker.html#SageMaker.Client.add_tags
    176         """
--> 177         container_def = self.prepare_container_def(instance_type, accelerator_type=accelerator_type)
    178         self.name = self.name or utils.name_from_image(container_def["Image"])
    179         enable_network_isolation = self.enable_network_isolation()

~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/sagemaker/mxnet/model.py in prepare_container_def(self, instance_type, accelerator_type)
    150 
    151         deploy_key_prefix = model_code_key_prefix(self.key_prefix, self.name, deploy_image)
--> 152         self._upload_code(deploy_key_prefix, self._is_mms_version())
    153         deploy_env = dict(self.env)
    154         deploy_env.update(self._framework_env_vars())

~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/sagemaker/model.py in _upload_code(self, key_prefix, repack)
    823                 repacked_model_uri=repacked_model_data,
    824                 sagemaker_session=self.sagemaker_session,
--> 825                 kms_key=self.model_kms_key,
    826             )
    827 

~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/sagemaker/utils.py in repack_model(inference_script, source_directory, dependencies, model_uri, repacked_model_uri, sagemaker_session, kms_key)
    481 
    482     with _tmpdir() as tmp:
--> 483         model_dir = _extract_model(model_uri, sagemaker_session, tmp)
    484 
    485         _create_or_update_code_dir(

~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/sagemaker/utils.py in _extract_model(model_uri, sagemaker_session, tmp)
    571     if model_uri.lower().startswith("s3://"):
    572         local_model_path = os.path.join(tmp, "tar_file")
--> 573         download_file_from_url(model_uri, local_model_path, sagemaker_session)
    574     else:
    575         local_model_path = model_uri.replace("file://", "")

~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/sagemaker/utils.py in download_file_from_url(url, dst, sagemaker_session)
    589     bucket, key = url.netloc, url.path.lstrip("/")
    590 
--> 591     download_file(bucket, key, dst, sagemaker_session)
    592 
    593 

~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/sagemaker/utils.py in download_file(bucket_name, path, target, sagemaker_session)
    607     s3 = boto_session.resource("s3")
    608     bucket = s3.Bucket(bucket_name)
--> 609     bucket.download_file(path, target)
    610 
    611 

~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/boto3/s3/inject.py in bucket_download_file(self, Key, Filename, ExtraArgs, Callback, Config)
    244     return self.meta.client.download_file(
    245         Bucket=self.name, Key=Key, Filename=Filename,
--> 246         ExtraArgs=ExtraArgs, Callback=Callback, Config=Config)
    247 
    248 

~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/boto3/s3/inject.py in download_file(self, Bucket, Key, Filename, ExtraArgs, Callback, Config)
    170         return transfer.download_file(
    171             bucket=Bucket, key=Key, filename=Filename,
--> 172             extra_args=ExtraArgs, callback=Callback)
    173 
    174 

~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/boto3/s3/transfer.py in download_file(self, bucket, key, filename, extra_args, callback)
    305             bucket, key, filename, extra_args, subscribers)
    306         try:
--> 307             future.result()
    308         # This is for backwards compatibility where when retries are
    309         # exceeded we need to throw the same error from boto3 instead of

~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/s3transfer/futures.py in result(self)
    104             # however if a KeyboardInterrupt is raised we want want to exit
    105             # out of this and propogate the exception.
--> 106             return self._coordinator.result()
    107         except KeyboardInterrupt as e:
    108             self.cancel()

~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/s3transfer/futures.py in result(self)
    263         # final result.
    264         if self._exception:
--> 265             raise self._exception
    266         return self._result
    267 

~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/s3transfer/tasks.py in _main(self, transfer_future, **kwargs)
    253             # Call the submit method to start submitting tasks to execute the
    254             # transfer.
--> 255             self._submit(transfer_future=transfer_future, **kwargs)
    256         except BaseException as e:
    257             # If there was an exception raised during the submission of task

~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/s3transfer/download.py in _submit(self, client, config, osutil, request_executor, io_executor, transfer_future, bandwidth_limiter)
    341                 Bucket=transfer_future.meta.call_args.bucket,
    342                 Key=transfer_future.meta.call_args.key,
--> 343                 **transfer_future.meta.call_args.extra_args
    344             )
    345             transfer_future.meta.provide_transfer_size(

~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/botocore/client.py in _api_call(self, *args, **kwargs)
    314                     "%s() only accepts keyword arguments." % py_operation_name)
    315             # The "self" in this scope is referring to the BaseClient.
--> 316             return self._make_api_call(operation_name, kwargs)
    317 
    318         _api_call.__name__ = str(py_operation_name)

~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/botocore/client.py in _make_api_call(self, operation_name, api_params)
    624             error_code = parsed_response.get("Error", {}).get("Code")
    625             error_class = self.exceptions.from_code(error_code)
--> 626             raise error_class(parsed_response, operation_name)
    627         else:
    628             return parsed_response

ClientError: An error occurred (404) when calling the HeadObject operation: Not Found

and looking more into it I found out that in local mode model.tar.gz location uri is stored differently from non-local.

• Local model uri: s3://{bucket}/{prefix}/outputmxnet-training-2020-03-12-23-19-23-971/model.tar.gz

• Non-local uri: s3://{bucket}/{prefix}/output/mxnet-training-2020-03-12-23-19-23-971/output/model.tar.gz.

System information
A description of your system. Please provide:

• SageMaker Python SDK version: 1.51.3
• Framework name (eg. PyTorch) or algorithm (eg. KMeans): MXNet
• Framework version: 1.6.0
• Python version: py3.6
• CPU or GPU: Both
• Custom Docker image (Y/N): N

[laurenyu]

what is your output_path set to in your estimator?

[ehsanmok]

@laurenyu it's the same output_path='s3://{}/{}/output'.format(bucket, prefix) for both modes.

[nadiaya]

    Local model uri: s3://{bucket}/{prefix}/outputmxnet-training-2020-03-12-23-19-23-971/model.tar.gz

    Non-local uri: s3://{bucket}/{prefix}/output/mxnet-training-2020-03-12-23-19-23-971/output/model.tar.gz.

is a known issue, but I don't think it would cause the error you are experiencing.

[nadiaya]

Could you share your code that creates the estimator and deploys the model?

[ehsanmok]

@nadiaya I'm using MXNetModel(model_data=os.path.join(output_path, 'model.tar.gz'), ...) for my custom inference and not being able to find the model for deploy was the issue where I had to fix manually using estimator._current_job_name to correctly get the path.

Here is the not-so-special inference code.

[nadiaya]

Could you give us the code of how you create the training job and then deploy the model?

[ehsanmok]

Take a look at the notebook. Because of this issue, I haven't included the local session mode in the notebook, you'd need to add it yourself. That's the best I can offer.

Also if this is a known issue please link in here for the record.

[nadiaya]

An alternative solution would be to get the model location from the training job which would work regardless of if the mode:

training_job_name = estimator.latest_training_job.name
desc = sagemaker_session.sagemaker_client.describe_training_job(TrainingJobName=training_job_name)
trained_model_location = desc['ModelArtifacts']['S3ModelArtifacts']

[laurenyu]

I've submitted a PR to address this: #1439

[laurenyu]

#1439 has been released as part of v1.56.1. does it resolve your issue?

[ehsanmok]

Thanks @laurenyu! I just upgraded and tested it and it partially solved the earlier issue. Here's the what's happening now:

I'm using output_path=train_output='s3://{}/{}/output'.format(bucket, prefix) in my estimator.

• local uri:

s3://{bucket}/{prefix}/output/mxnet-training-2020-04-28-23-01-06-447/model.tar.gz
s3://{bucket}/{prefix}/output/mxnet-training-2020-04-28-23-01-06-447/output.tar.gz

• non-local uri:

s3://{bucket}/{prefix}/output/mxnet-training-2020-04-28-23-42-45-505/output/model.tar.gz
s3://{bucket}/{prefix}/output/mxnet-training-2020-04-28-23-42-45-505/output/output.tar.gz

Notice the additional /output attached to the non-local uri so it needs some more work to be completely unified.

I also examined the content of output.tar.gz for both local and non-local cases and they do not match which they differ by a redundant data repo in the local mode.

• local:

output/
  data/
    a.csv

• non-local:

output/
   a.csv