Internal server error when using OpenNMT-tf

[anasir-ureed]

Reference: 0411850995

System Information

• Framework (e.g. TensorFlow) / Algorithm (e.g. KMeans): TensorFlow , OpenNMT-tf
• Framework Version: 1.14
• Python Version: py3
• CPU or GPU: GPU
• Python SDK Version: latest
• Are you using a custom image: script training

Describe the problem

I am using OpenNMT-tf for training. When the training reaches the export step, the exported model is not exported properly, and a strange error named "Internal server error" appears.

Minimal repro / logs

Unfortunately, there is nothing in the logs telling anything about the issue!

• Exact command to reproduce:

estimator = TensorFlow(entry_point=local_training_script_path,
                       dependencies=['model.py'],
                       train_instance_type='ml.p3.2xlarge',
                       train_instance_count=1,
                       checkpoint_s3_uri=checkpoint_path,
                       output_path=model_artifacts_location,
                       code_location=custom_code_upload_location,
                       role=sagemaker.get_execution_role(),
                       framework_version='1.14',
                       py_version='py3',
                       script_mode=True,
                       train_use_spot_instances=train_use_spot_instances,
                       train_max_run=train_max_run,
                       train_max_wait=train_max_wait,
                       train_volume_size=75)
estimator.fit(training_inputs_location, job_name=job_name_sagemaker, wait=True)

The shell script contains

pip install OpenNMT-tf
onmt-main ...

Please advise,

[mvsusp]

Hello @anasir-ureed

Please share with us a minimal code example, allowing us to reproduce the issue.

Thanks for using SageMaker

[anasir-ureed]

Hello @mvsusp

Thanks for your reply

This is the script I am using in the entry point. "train.sh"

#!/bin/bash
pip install OpenNMT-tf
onmt-main train_and_eval --model_type NMTMedium --config ${SM_CHANNEL_TRAINING}/config/exp1.yml --data_dir ${SM_CHANNEL_TRAINING} --run_dir /opt/ml/checkpoints/
cp -R /opt/ml/checkpoints/* ${SM_MODEL_DIR}/

In the config file,

model_dir: .

data:
  train_features_file: data/train.sp.en
  train_labels_file: data/train.sp.ar
  eval_features_file: data/val.sp.en
  eval_labels_file: data/val.sp.ar
  source_words_vocabulary: data/t.vocab
  target_words_vocabulary: data/t.vocab

params:
  optimizer: GradientDescentOptimizer
  learning_rate: 0.5
  param_init: 0.1
  clip_gradients: 5.0
  decay_type: exponential_decay
  decay_rate: 0.8
  decay_steps: 50000
  start_decay_steps: 50000
  min_learning_rate: 0.0001
  beam_width: 5
  maximum_iterations: 250
  replace_unknown_target: true

train:
  batch_size: 128
  bucket_width: 5
  save_checkpoints_steps: 300
  save_summary_steps: 50
  train_steps: 250000
  maximum_features_length: 300
  maximum_labels_length: 300
  keep_checkpoint_max: 10
  average_last_checkpoints: 8

eval:
  eval_delay: 7200
  batch_size: 8
  external_evaluators: BLEU
  exporters: last
  save_eval_predictions: true

infer:
  batch_size: 32
  n_best: 1

In the notebook please use

train_use_spot_instances = False # the issue happens in spot and on demand
train_max_run=3600 * 24 * 5
train_max_wait = 3600 * 24 * 5 if train_use_spot_instances else None

estimator = TensorFlow(entry_point="train.sh",
                       train_instance_type='ml.p3.2xlarge',
                       train_instance_count=1,
                       checkpoint_s3_uri=checkpoint_path, # Any location you wish
                       role=sagemaker.get_execution_role(),
                       framework_version='1.14',
                       py_version='py3',
                       script_mode=True,
                       train_use_spot_instances=train_use_spot_instances,
                       train_max_run=train_max_run,
                       train_max_wait=train_max_wait,
                       train_volume_size=75)
estimator.fit(training_inputs_location,  wait=True) 

In training_inputs_location, the directory structure is as following:

config/exp1.yml
data/train.sp.en
data/train.sp.ar
data/val.sp.en
data/val.sp.ar
data/t.vocab
data/t.vocab

Best wishes,

[mvsusp]

Thanks @anasir-ureed. Where do I find the data files?

[anasir-ureed]

Hello @mvsusp ,

Any data files I think can be used to reproduce the issue. It happened with me using several data sources.

I have tested also using toy-ende data that can be downloaded if you follow the following link. http://opennmt.net/OpenNMT-tf/quickstart.html

You will need to change the config file to match the data files names.

  train_features_file: data/src-train.txt
  train_labels_file: data/tgt-train.txt
  eval_features_file: data/src-val.txt
  eval_labels_file: data/tgt-val.txt
  source_words_vocabulary: data/src-vocab.txt
  target_words_vocabulary: data/tgt-vocab.txt

also please change train.sh script pip install OpenNMT-tf to :

pip install OpenNMT-tf==1.25.1

A new major version just was released.

Thanks,

[AbdallahNasir]

Hello @mvsusp ,

Any updates?

Best wishes,

[chuyang-deng]

Hi @AbdallahNasir, sorry for the delay. @mvsusp has left the team and we will continue looking into the issue.

[AbdallahNasir]

Best wishes to him.

Thank you @ChuyangDeng, appreciate your efforts.

[ChoiByungWook]

Reference: 0411850995

@AbdallahNasir would it be possible for you to open an AWS support ticket?

We would like to get some information to identify your training job that failed, so we can get the stack trace that is hidden behind the "internal server error".

Feel free to reference this Github issue in your AWS support ticket.

In the meanwhile, can you please provide the name for the training job that failed and region?

Thanks!

[AbdallahNasir]

Hello @ChoiByungWook ,

The training job name is error-exp---2019-01-10--08-54-45-225648 in us-east-1. Our account name alias is "ureed".

Thanks for your assistance.

[ChoiByungWook]

Hello @AbdallahNasir,

I have passed this information to the respective team.

Please make sure you file an AWS support ticket, as to ensure your issue doesn't get lost.

Thanks!

[AbdallahNasir]

Dear @ChoiByungWook ,

It looks like I cannot create support tickets, as we do not have purchased support yet.

Any chance we can do this from here?

Best wishes,

[ChoiByungWook]

Hello @AbdallahNasir,

I've reached out to the corresponding team and it looks like we are in need of your AWS account ID as well, however I would not post that on Github.

Can you please message me through the AWS forums? https://forums.aws.amazon.com/forum.jspa?forumID=285

After logging in, please click "Private Messages" -> "Compose Message" -> and send to DanielCAtAWS.

Feel free to also create a forum post in the AWS forums referencing this Github issue.

Apologies for the back and forth.

[AbdallahNasir]

Hello @ChoiByungWook ,

Thanks for your reply.

I have sent you the message on the form.

Thanks for your support.

Best wishes,

[ChoiByungWook]

Hello @AbdallahNasir,

I have forwarded the ID to the corresponding team. Awaiting response from them.

Thank you for your patience.

[AbdallahNasir]

Hi @ChoiByungWook,

Thank you so much dear :)

[ChoiByungWook]

Hello @AbdallahNasir,

The corresponding team has looked into the issue, and it seems to have been an issue on the platform side. A fix was recently deployed, can you please retry your training job?

Thank you!

[AbdallahNasir]

Hello @ChoiByungWook,

I have started another training job, called error-exp---2019-17-10--06-51-04-788983. It failed, producing the same error message.

Thank you,

[AbdallahNasir]

Hello @ChoiByungWook,

I hope you are having a great day.

Any updates dear?

Best wishes,

[martinRenou]

Closing for triage.

If you read this message and are also seeing this error message. Please open a new issue or file an AWS support ticket as suggested above.

Thank you.