Skip to content

The TensorFlow container saves the exported models in Ascending order #296

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
chinazm opened this issue Jul 16, 2018 · 3 comments
Closed

Comments

@chinazm
Copy link

chinazm commented Jul 16, 2018

We found that there is a bug in sagemaker to export the model:

sagemaker first exported checkpoints as exported models in the output folder and after training there will be a final model zipped and exported. We found sagemaker exported and zipped the models on ascending order not the most recent model exported.

@andremoeller
Copy link
Contributor

Hi @chinazm ,

Thanks for the report! You're right that this isn't the expected behavior -- we'll investigate and comment on this ticket when we have updates.

Thanks again!

@andremoeller
Copy link
Contributor

Hi @chinazm ,

I tried this out and confirmed that we seem to be exporting the wrong checkpoint. I sent this PR to fix this: aws/sagemaker-tensorflow-training-toolkit#58

Thanks!

@andremoeller
Copy link
Contributor

@chinazm

That fix has been merged in. Thanks for the report!

apacker pushed a commit to apacker/sagemaker-python-sdk that referenced this issue Nov 15, 2018
Use SageMaker SDK RandomCutForest estimator
knakad added a commit to knakad/sagemaker-python-sdk that referenced this issue Dec 4, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants