Supporting custom frameworks in SageMaker??

[everpeace]

Thank you for shipping this great product!!!

After reading the documents, I understood SageMaker is fully extensible by docker container. And scikit_bring_your_own example clearly shows users how custom python codes can be applied to SageMaker.

However, when I wanted to support custom framework (like torch, chainer, etc.), how to do that effectively?? To achieve this, they possibly need to write notebook side and container side implementations.

On notebook side, I think mxnet support in sagemaker-python-sdk can be really helpful reference.

However, on container side, I couldn't find such reference. But, there would need a lot of common codes for example:

• download user assets(training data, user codes) from s3 to local filesystem
• loading custom user codes to train and save models
• spawning server via wsgi or similar libraries
• detecting cpu/gpu feature
• etc.

I imagined these codes might exist in MXNet, tensorflow support, too. Then, I found SageMakerContainerSupport package in sagemaker-mxnet-py2-cpu:1.0 container. This package defines very useful abstract classes to support custom frameworks for everybody who wants to do it.

Of course, I can write similar codes by myself, however, in OSS nature, I'm really happy if you could open this package to the public. Did you have such plan??

[jesterhazy]

@everpeace thanks for your interest in SageMaker!

Adding more frameworks and clearer support for user-defined frameworks is definitely on our roadmap for the near future. We are also considering open-sourcing the container-side code, and we will take your feedback into account as we do so.

[everpeace]

@jesterhazy Thanks for quick reply :-) Wow, that's a good news. I'm looking forward to it!

So, I'm closing this issue for now. If you would make this issue feedback-tracking issue, please re-open this.

[babupriyavrat]

we use DarkNet for video analysis and pyAudioanalysis for audio model training. I'm looking forward how we can integrate these framework into SageMaker.