ERROR - container_support.training - uncaught exception during training: None values not supported.

[cliffordgreen]

Please fill out the form below.

System Information

• Tensorflow
• framework_version='1.12.0'
• 36:
• CPU or GPU:
• Python SDK Version:
• Are you using a custom image:

Describe the problem

Getting an error during training as none type

Minimal repro / logs

2019-03-26 22:56:55 Starting - Starting the training job...
2019-03-26 22:56:57 Starting - Launching requested ML instances......
2019-03-26 22:58:25 Starting - Preparing the instances for training......
2019-03-26 22:59:17 Downloading - Downloading input data............
2019-03-26 23:01:11 Training - Training image download completed. Training in progress.
2019-03-26 23:01:12,116 INFO - root - running container entrypoint
2019-03-26 23:01:12,116 INFO - root - starting train task
2019-03-26 23:01:12,129 INFO - container_support.training - Training starting
Downloading s3://sagemaker-us-east-2-543400108592/sagemaker-tensorflow-2019-03-26-22-56-54-157/source/sourcedir.tar.gz to /tmp/script.tar.gz
2019-03-26 23:01:14,850 INFO - tf_container - ----------------------TF_CONFIG--------------------------
2019-03-26 23:01:14,851 INFO - tf_container - {"environment": "cloud", "cluster": {"master": ["algo-1:2222"]}, "task": {"index": 0, "type": "master"}}
2019-03-26 23:01:14,851 INFO - tf_container - ---------------------------------------------------------
2019-03-26 23:01:14,851 INFO - tf_container - creating RunConfig:
2019-03-26 23:01:14,851 INFO - tf_container - {'save_checkpoints_secs': 300}
2019-03-26 23:01:14,851 INFO - tensorflow - TF_CONFIG environment variable: {u'environment': u'cloud', u'cluster': {u'master': [u'algo-1:2222']}, u'task': {u'index': 0, u'type': u'master'}}
2019-03-26 23:01:14,851 INFO - tf_container - creating an estimator from the user-provided model_fn
2019-03-26 23:01:14,852 INFO - tensorflow - Using config: {'_save_checkpoints_secs': 300, '_keep_checkpoint_max': 5, '_task_type': u'master', '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f2f3be3bed0>, '_keep_checkpoint_every_n_hours': 10000, '_service': None, '_num_ps_replicas': 0, '_tf_random_seed': None, '_device_fn': None, '_num_worker_replicas': 1, '_task_id': 0, '_log_step_count_steps': 100, '_evaluation_master': '', '_eval_distribute': None, '_train_distribute': None, '_session_config': device_filters: "/job:ps"
device_filters: "/job:master"
allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_global_id_in_cluster': 0, '_is_chief': True, '_protocol': None, '_save_checkpoints_steps': None, '_experimental_distribute': None, '_save_summary_steps': 100, '_model_dir': u's3://sagemaker-us-east-2-543400108592/sagemaker-tensorflow-2019-03-26-22-56-54-157/checkpoints', '_master': ''}
2019-03-26 23:01:14,853 INFO - tensorflow - Not using Distribute Coordinator.
2019-03-26 23:01:14,853 INFO - tensorflow - Skip starting Tensorflow server as there is only one node in the cluster.
2019-03-26 23:01:16,505 WARNING - tensorflow - From /opt/ml/code/tfSwitch.py:145: load_csv_without_header (from tensorflow.contrib.learn.python.learn.datasets.base) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.data instead.
2019-03-26 23:01:22,251 WARNING - tensorflow - From /usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/inputs/queues/feeding_queue_runner.py:62: __init__ (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version.
Instructions for updating:
To construct input pipelines, use the `tf.data` module.
2019-03-26 23:01:22,253 WARNING - tensorflow - From /usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/inputs/queues/feeding_functions.py:500: add_queue_runner (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version.
Instructions for updating:
To construct input pipelines, use the `tf.data` module.
2019-03-26 23:01:22,262 INFO - tensorflow - Calling model_fn.
2019-03-26 23:01:22,409 WARNING - tensorflow - From /opt/ml/code/tfSwitch.py:76: get_global_step (from tensorflow.contrib.framework.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.train.get_global_step
2019-03-26 23:01:22,719 INFO - tensorflow - Done calling model_fn.
2019-03-26 23:01:22,721 INFO - tensorflow - Create CheckpointSaverHook.
2019-03-26 23:01:23,253 INFO - tensorflow - Graph was finalized.
2019-03-26 23:01:23,600 INFO - tensorflow - Running local_init_op.
2019-03-26 23:01:23,607 INFO - tensorflow - Done running local_init_op.
2019-03-26 23:01:23,636 WARNING - tensorflow - From /usr/local/lib/python2.7/dist-packages/tensorflow/python/training/monitored_session.py:804: start_queue_runners (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version.
Instructions for updating:
To construct input pipelines, use the `tf.data` module.
2019-03-26 23:01:24,121 INFO - tensorflow - Saving checkpoints for 0 into s3://sagemaker-us-east-2-543400108592/sagemaker-tensorflow-2019-03-26-22-56-54-157/checkpoints/model.ckpt.
2019-03-26 23:01:29,527 INFO - tensorflow - loss = 0.6829288, step = 1
2019-03-26 23:02:12,552 INFO - tensorflow - global_step/sec: 2.32418
2019-03-26 23:02:12,553 INFO - tensorflow - loss = 0.2901844, step = 101 (43.026 sec)
2019-03-26 23:02:55,116 INFO - tensorflow - global_step/sec: 2.34939
2019-03-26 23:02:55,117 INFO - tensorflow - loss = 0.30758405, step = 201 (42.564 sec)
2019-03-26 23:03:37,843 INFO - tensorflow - global_step/sec: 2.34045
2019-03-26 23:03:37,941 INFO - tensorflow - loss = 0.2938314, step = 301 (42.824 sec)
2019-03-26 23:04:21,769 INFO - tensorflow - global_step/sec: 2.27658
2019-03-26 23:04:21,770 INFO - tensorflow - loss = 0.24023758, step = 401 (43.829 sec)
2019-03-26 23:05:04,487 INFO - tensorflow - global_step/sec: 2.3409
2019-03-26 23:05:04,488 INFO - tensorflow - loss = 0.2212125, step = 501 (42.718 sec)
2019-03-26 23:05:47,008 INFO - tensorflow - global_step/sec: 2.35177
2019-03-26 23:05:47,106 INFO - tensorflow - loss = 0.24229552, step = 601 (42.618 sec)
2019-03-26 23:06:28,716 INFO - tensorflow - Saving checkpoints for 697 into s3://sagemaker-us-east-2-543400108592/sagemaker-tensorflow-2019-03-26-22-56-54-157/checkpoints/model.ckpt.
2019-03-26 23:06:34,000 INFO - tensorflow - Calling model_fn.
2019-03-26 23:06:34,429 INFO - tensorflow - Done calling model_fn.
2019-03-26 23:06:34,448 INFO - tensorflow - Starting evaluation at 2019-03-26-23:06:34
2019-03-26 23:06:34,581 INFO - tensorflow - Graph was finalized.
2019-03-26 23:06:34,635 INFO - tensorflow - Restoring parameters from s3://sagemaker-us-east-2-543400108592/sagemaker-tensorflow-2019-03-26-22-56-54-157/checkpoints/model.ckpt-697
2019-03-26 23:06:36,641 INFO - tensorflow - Running local_init_op.
2019-03-26 23:06:36,652 INFO - tensorflow - Done running local_init_op.
2019-03-26 23:06:38,512 INFO - tensorflow - Evaluation [10/100]
2019-03-26 23:06:40,105 INFO - tensorflow - Evaluation [20/100]
2019-03-26 23:06:41,711 INFO - tensorflow - Evaluation [30/100]
2019-03-26 23:06:43,351 INFO - tensorflow - Evaluation [40/100]

2019-03-26 23:06:59 Uploading - Uploading generated training model2019-03-26 23:06:45,008 INFO - tensorflow - Evaluation [50/100]
2019-03-26 23:06:46,619 INFO - tensorflow - Evaluation [60/100]
2019-03-26 23:06:48,213 INFO - tensorflow - Evaluation [70/100]
2019-03-26 23:06:49,816 INFO - tensorflow - Evaluation [80/100]
2019-03-26 23:06:51,431 INFO - tensorflow - Evaluation [90/100]
2019-03-26 23:06:53,058 INFO - tensorflow - Evaluation [100/100]
2019-03-26 23:06:53,125 INFO - tensorflow - Finished evaluation at 2019-03-26-23:06:53
2019-03-26 23:06:53,125 INFO - tensorflow - Saving dict for global step 697: accuracy = 0.9100781, global_step = 697, loss = 0.25744277
2019-03-26 23:06:53,606 INFO - tensorflow - Saving 'checkpoint_path' summary for global step 697: s3://sagemaker-us-east-2-543400108592/sagemaker-tensorflow-2019-03-26-22-56-54-157/checkpoints/model.ckpt-697
2019-03-26 23:06:53,856 INFO - tensorflow - Calling model_fn.
2019-03-26 23:06:54,005 ERROR - container_support.training - uncaught exception during training: None values not supported.
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/container_support/training.py", line 36, in start
    fw.train()
  File "/usr/local/lib/python2.7/dist-packages/tf_container/train_entry_point.py", line 173, in train
    train_wrapper.train()
  File "/usr/local/lib/python2.7/dist-packages/tf_container/trainer.py", line 73, in train
    tf.estimator.train_and_evaluate(estimator=estimator, train_spec=train_spec, eval_spec=eval_spec)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/training.py", line 471, in train_and_evaluate
    return executor.run()
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/training.py", line 637, in run
    getattr(self, task_to_run)()
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/training.py", line 674, in run_master
    self._start_distributed_training(saving_listeners=saving_listeners)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/training.py", line 788, in _start_distributed_training
    saving_listeners=saving_listeners)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/estimator.py", line 354, in train
    loss = self._train_model(input_fn, hooks, saving_listeners)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/estimator.py", line 1207, in _train_model
    return self._train_model_default(input_fn, hooks, saving_listeners)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/estimator.py", line 1241, in _train_model_default
    saving_listeners)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/estimator.py", line 1471, in _train_with_estimator_spec
    _, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss])
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/monitored_session.py", line 671, in run
    run_metadata=run_metadata)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/monitored_session.py", line 1156, in run
    run_metadata=run_metadata)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/monitored_session.py", line 1255, in run
    raise six.reraise(*original_exc_info)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/monitored_session.py", line 1240, in run
    return self._sess.run(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/monitored_session.py", line 1320, in run
    run_metadata=run_metadata))
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/basic_session_run_hooks.py", line 582, in after_run
    if self._save(run_context.session, global_step):
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/basic_session_run_hooks.py", line 607, in _save
    if l.after_save(session, step):
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/training.py", line 517, in after_save
    self._evaluate(global_step_value)  # updates self.eval_result
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/training.py", line 537, in _evaluate
    self._evaluator.evaluate_and_export())
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/training.py", line 924, in evaluate_and_export
    is_the_final_export)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/training.py", line 957, in _export_eval_result
    is_the_final_export=is_the_final_export))
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/exporter.py", line 472, in export
    is_the_final_export)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/exporter.py", line 126, in export
    strip_default_attrs=self._strip_default_attrs)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/estimator.py", line 663, in export_savedmodel
    mode=model_fn_lib.ModeKeys.PREDICT)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/estimator.py", line 789, in _export_saved_model_for_mode
    strip_default_attrs=strip_default_attrs)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/estimator.py", line 907, in _export_all_saved_models
    mode=model_fn_lib.ModeKeys.PREDICT)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/estimator.py", line 984, in _add_meta_graph_for_mode
    config=self.config)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/estimator.py", line 1195, in _call_model_fn
    model_fn_results = self._model_fn(features=features, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/tf_container/trainer.py", line 108, in _model_fn
    return self.customer_script.model_fn(features, labels, mode, params)
  File "/opt/ml/code/tfSwitch.py", line 73, in model_fn
    logits=output_layer, labels=tf.cast(labels, dtype=tf.int32)))
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/math_ops.py", line 675, in cast
    x = ops.convert_to_tensor(x, name="x")
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1050, in convert_to_tensor
    as_ref=False)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1146, in internal_convert_to_tensor
    ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/constant_op.py", line 229, in _constant_tensor_conversion_function
    return constant(v, dtype=dtype, name=name)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/constant_op.py", line 208, in constant
    value, dtype=dtype, shape=shape, verify_shape=verify_shape))
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/tensor_util.py", line 430, in make_tensor_proto
    raise ValueError("None values not supported.")
ValueError: None values not supported.

• Exact command to reproduce:

def model_fn(features, labels, mode, params):


    # 1. Configure the model via Keras functional api
    #model = Sequential()
    C1 = Conv1D(256,kernel_size=40, strides = 1 ,activation='relu')(tf.reshape(features[INPUT_TENSOR_NAME], [-1,100, 1]))     
    P1 = MaxPooling1D(pool_size=(3), strides=1, data_format=None)(C1)
    C2 = Conv1D(256, kernel_size=[2],activation='relu')(P1)      
    P2 = MaxPooling1D(pool_size=(3), strides=1, data_format=None)(C2)
    C3 = Conv1D(256, kernel_size=[2],activation='relu')(P2)        
    P3 = MaxPooling1D(pool_size=(1), strides=1, data_format=None)(C3)
    C4 = Conv1D(256, kernel_size=[2],activation='relu')(P3)  
    P4 = MaxPooling1D(pool_size=(3), strides=1, data_format=None)(C4)
    DO1 = Dropout(0.5)(P4)
    F = Flatten()(DO1)
    DE = Dense(1000)(F)
    output_layer = Dense(2)(DE)
    pred_classes = tf.argmax(output_layer, axis=1)
    pred_probas = tf.nn.softmax(output_layer)
    
    
    opt = AdamOptimizer(learning_rate=params['learning_rate'])

    loss_op = tf.reduce_mean(tf.nn.sparse_softmax_cross_entropy_with_logits(
        logits=output_layer, labels=tf.cast(labels, dtype=tf.int32)))
    train_op = tf.contrib.layers.optimize_loss(
        loss=loss_op,
        global_step=tf.contrib.framework.get_global_step(),
        learning_rate=params["learning_rate"],
        optimizer=opt)
    

 
    if mode == tf.estimator.ModeKeys.PREDICT:
        return tf.estimator.EstimatorSpec(
            mode=mode,
            predictions={"Gender": pred_classes},
            export_outputs={SIGNATURE_NAME: PredictOutput({"Gender": pred_classes})})



    # 4. Generate predictions as Tensorflow tensors.
    predictions_dict = {"Gender": pred_classes}

    # 5. Generate necessary evaluation metrics.
    # Calculate root mean squared error as additional eval metric
    eval_metric_ops = {
        "accuracy": 
        tf.metrics.accuracy(
        labels,
        pred_classes,
   
        )

    }

    # Provide an estimator spec for `ModeKeys.EVAL` and `ModeKeys.TRAIN` modes.
    return tf.estimator.EstimatorSpec(
        mode=mode,
        loss=loss_op,
        train_op=train_op,
        eval_metric_ops=eval_metric_ops)

[jesterhazy]

@cliffordgreen

Thanks for using SageMaker!

There is probably something wrong with your model function, but it's hard to tell with the formatting issues. can you post the code again, using a code block (triple backticks) so it isn't rendered as markdown?

[laurenyu]

@cliffordgreen can you also provide the code you're using to start the SageMaker Training Job and the input data (synthetic is fine if your data needs to be kept private - just something so that we can reproduce the issue). thanks for your patience!

[laurenyu]

closing due to inactivity