Skip to content

Upgrade SKLearn to 1.2.1 #163

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Feb 24, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -184,7 +184,7 @@ If you want to run local integration tests, then use:

# Example
pytest test/integration --docker-base-name preprod-sklearn ``\``
--tag 1.0-1 ``\``
--tag 1.2-1 ``\``
--py-version 3 ``\``
--framework-version |FRAMEWORK_VERSION|

Expand Down Expand Up @@ -242,4 +242,4 @@ SageMaker Scikit-learn Container is licensed under the Apache 2.0 License. It is
.com, Inc. or its affiliates. All Rights Reserved. The license is available at:
http://aws.amazon.com/apache2.0/

.. |FRAMEWORK_VERSION| replace:: 1.0-1
.. |FRAMEWORK_VERSION| replace:: 1.2-1
Original file line number Diff line number Diff line change
Expand Up @@ -116,4 +116,4 @@ ENV PYTHONDONTWRITEBYTECODE=1 PYTHONUNBUFFERED=1 PYTHONIOENCODING=UTF-8 LANG=C.U
# Install Scikit-Learn
# Scikit-learn 0.20 was the last version to support Python 2.7 and Python 3.4.
# Scikit-learn now requires Python 3.6 or newer.
RUN python -m pip install --no-cache -I scikit-learn==1.0.2
RUN python -m pip install --no-cache -I scikit-learn==1.2.1
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
FROM preprod-sklearn:1.0-1
FROM preprod-sklearn:1.2-1

RUN pip freeze | grep -q 'scikit-learn==1.0.2'; \
RUN pip freeze | grep -q 'scikit-learn==1.2.1'; \
if [ $? -eq 0 ]; \
then echo 'scikit-learn version 1.0.2 requirement met'; \
else echo 'ERROR: Expected scikit-learn version is 1.0.2, check base images for scikit-learn version' && \
then echo 'scikit-learn version 1.2.1 requirement met'; \
else echo 'ERROR: Expected scikit-learn version is 1.2.1, check base images for scikit-learn version' && \
exit 1; fi

RUN pip install --upgrade --no-cache --no-deps sagemaker-scikit-learn-extension==2.5.0
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,15 +8,15 @@ The SageMaker Scikit-learn Extension Container is built in 3 steps. The first 2

The "base" Dockerfile encompass the installation of the framework and all of the dependencies needed.

Tagging scheme is based on <Scikit-learn_version>-<SageMaker_version>-cpu-py<python_version>. (e.g. 1.0-1-cpu-py3)
Tagging scheme is based on <Scikit-learn_version>-<SageMaker_version>-cpu-py<python_version>. (e.g. 1.2-1-cpu-py3)

All "final" Dockerfiles build images using base images that use the tagging scheme above.

```
docker build -t sklearn-base:1.0-1-cpu-py3 -f docker/1.0-1/base/Dockerfile.cpu .
docker build -t sklearn-base:1.2-1-cpu-py3 -f docker/1.2-1/base/Dockerfile.cpu .
```

Notice that this Dockerfile has the updated version of sklearn (1.0.2) installed.
Notice that this Dockerfile has the updated version of sklearn (1.2.1) installed.

### Step 2: Final Image

Expand All @@ -38,7 +38,7 @@ python setup.py bdist_wheel
Then build the final image, like in the sagemaker-sklearn-container

```
docker build -t preprod-sklearn:1.0-1-cpu-py3 -f docker/1.0-1/final/Dockerfile.cpu .
docker build -t preprod-sklearn:1.2-1-cpu-py3 -f docker/1.2-1/final/Dockerfile.cpu .
```

### Step 3: Build the extension image for SageMaker Scikit-learn Extension Container
Expand All @@ -47,10 +47,10 @@ The "extension" Dockerfiles encompass the installation of the SageMaker Autopilo

The "extension" Dockerfiles use final images for building.

Build the third additional Dockerfile needed for SageMaker Scikit-learn Extension Container. This Dockerfile specifies a hard dependency on a certain version of scikit-learn (i.e. v1.0.2).
Build the third additional Dockerfile needed for SageMaker Scikit-learn Extension Container. This Dockerfile specifies a hard dependency on a certain version of scikit-learn (i.e. v1.2.1).

Tagging scheme is based on extension-<Scikit-learn-Extension_version>-<SageMaker_version>-cpu-py<python_version>. (e.g. extension-2.5-1-cpu-py3). Make sure the "extension" image is tagged in accordance with the `extension` (i.e. `extension-2.5-1-cpu-py3`).

```
docker build -t preprod-sklearn-extension:2.5-1-cpu-py3 -f docker/1.0-1/extension/Dockerfile.cpu .
docker build -t preprod-sklearn-extension:2.5-1-cpu-py3 -f docker/1.2-1/extension/Dockerfile.cpu .
```
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
FROM sklearn-base:1.0-1
ENV SAGEMAKER_SKLEARN_VERSION 1.0-1
FROM sklearn-base:1.2-1
ENV SAGEMAKER_SKLEARN_VERSION 1.2-1

LABEL com.amazonaws.sagemaker.capabilities.accept-bind-to-port=true

Expand Down
2 changes: 1 addition & 1 deletion requirements.txt
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ retrying==1.3.3
sagemaker-containers==2.8.6.post2
sagemaker-inference==1.2.0
sagemaker-training==4.0.1
scikit-learn==1.0.2
scikit-learn==1.2.1
scipy==1.5.3
six==1.15.0
jinja2==3.0.3
Expand Down
2 changes: 1 addition & 1 deletion test/conftest.py
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ def pytest_addoption(parser):
parser.addoption('--install-container-support', '-C', action='store_true')
parser.addoption('--docker-base-name', default='sk-learn')
parser.addoption('--region', default='us-west-2')
parser.addoption('--framework-version', default='1.0.2')
parser.addoption('--framework-version', default='1.2.1')
parser.addoption('--py-version', choices=['2', '3'], default=str(sys.version_info.major))
parser.addoption('--processor', choices=['cpu'], default='cpu')
# If not specified, will default to {framework-version}-{processor}-py{py-version}
Expand Down
8 changes: 4 additions & 4 deletions test/integration/test_multiple_model_endpoint.py
Original file line number Diff line number Diff line change
Expand Up @@ -150,15 +150,15 @@ def test_load_and_unload_model():
res_json = json.loads(res)
assert res_json['status'] == 'Workers scaled'

code, res = make_invocation_request('0.0, 0.0, 0.0, 0.0, 0.0, 0.0', model_name)
code, res = make_invocation_request('0.0, 0.0, 0.0, 0.0, 0.0', model_name)
assert code == 200, res

code, res = make_unload_model_request(model_name)
assert code == 200, res
res_json = json.loads(res)
assert res_json['status'] == "Model \"{}\" unregistered".format(model_name), res

code, res = make_invocation_request('0.0, 0.0, 0.0, 0.0, 0.0, 0.0', model_name)
code, res = make_invocation_request('0.0, 0.0, 0.0, 0.0, 0.0', model_name)
assert code == 404, res
assert res['message'] == "Model not found: {}".format(model_name), res

Expand All @@ -184,10 +184,10 @@ def test_load_and_unload_two_models():
res_json = json.loads(res)
assert res_json['status'] == 'Workers scaled'

code, res = make_invocation_request('0.0, 0.0, 0.0, 0.0, 0.0, 0.0', model_name_0)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have you ran the container Integ tests?

Copy link
Contributor Author

@malav-shastri malav-shastri Feb 21, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes they ran as a part of codebuild, and that led me to this particular change. By mistake I retrained the models using the dummy data which had 5 features but these requests were having 6 which I realized later and thought its easier and faster to make a change here.

code, res = make_invocation_request('0.0, 0.0, 0.0, 0.0, 0.0', model_name_0)
assert code == 200, res

code, res = make_invocation_request('0.0, 0.0, 0.0, 0.0, 0.0, 0.0', model_name_1)
code, res = make_invocation_request('0.0, 0.0, 0.0, 0.0, 0.0', model_name_1)
assert code == 200, res

code, res = make_unload_model_request(model_name_0)
Expand Down
Binary file modified test/resources/models/pickled-model-1/sklearn-model
Binary file not shown.
Binary file modified test/resources/models/pickled-model-2/sklearn-model
Binary file not shown.
4 changes: 2 additions & 2 deletions tox.ini
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
[tox]
envlist = {py3}-sklearn{1.0},flake8
envlist = {py3}-sklearn{1.2},flake8

[flake8]
max-line-length = 120

[testenv]
deps =
sklearn1.0: scikit-learn==1.0.2
sklearn1.2: scikit-learn==1.2.1
-r{toxinidir}/requirements.txt
-r{toxinidir}/test-requirements.txt
conda_deps=
Expand Down