Narrowly specified dependencies and pointless docker_compose

[jfelectron]

Please fill out the form below.

System Information

• Framework (e.g. TensorFlow) / Algorithm (e.g. KMeans): NA
• Framework Version: NA
• Python Version: 3.7
• CPU or GPU: CPU
• Python SDK Version: 1.43.4.post1
• Are you using a custom image: No

Describe the problem

This is a very narrow specification that clashes with almost any other library that depends on requests, of which there are many. Requests is stable, I see no reason to pin it so narrowly.

"requests>=2.20.0, <2.21"

This only appears to be needed for local testing and should not be required. It requires much older version of pyYaml and the docker python sdk (<4), which again clashes with many other libraries:

"docker-compose>=1.23.0"

The real root of the problem is that sagemaker uses:

pkg_resources.require("sagemaker")[0].version

This incurs very stringent runtime dependency conflict chekcing that is atypical for python libraries. If such strict depenendencies are really required, they should be enforced at install time and not runtime.

File "/Users/foleyjon/Documents/Repos/panem/panem/model/conversion_classifier.py", line 5, in
import sagemaker
File "/Users/foleyjon/Documents/Repos/panem/venv/lib/python3.7/site-packages/sagemaker/init.py", line 18, in
from sagemaker import estimator, parameter, tuner # noqa: F401
File "/Users/foleyjon/Documents/Repos/panem/venv/lib/python3.7/site-packages/sagemaker/estimator.py", line 26, in
from sagemaker.analytics import TrainingJobAnalytics
File "/Users/foleyjon/Documents/Repos/panem/venv/lib/python3.7/site-packages/sagemaker/analytics.py", line 23, in
from sagemaker.session import Session
File "/Users/foleyjon/Documents/Repos/panem/venv/lib/python3.7/site-packages/sagemaker/session.py", line 35, in
from sagemaker.user_agent import prepend_user_agent
File "/Users/foleyjon/Documents/Repos/panem/venv/lib/python3.7/site-packages/sagemaker/user_agent.py", line 23, in
SDK_VERSION = pkg_resources.require("sagemaker")[0].version
File "/Users/foleyjon/Documents/Repos/panem/venv/lib/python3.7/site-packages/pkg_resources/init.py", line 900, in require
needed = self.resolve(parse_requirements(requirements))
File "/Users/foleyjon/Documents/Repos/panem/venv/lib/python3.7/site-packages/pkg_resources/init.py", line 791, in resolve
raise VersionConflict(dist, req).with_context(dependent_req)
g_resources.ContextualVersionConflict: (docker 4.0.2 (/Users/foleyjon/Documents/Repos/panem/venv/lib/python3.7/site-packages), Requirement.parse('docker[ssh]<4.0,>=3.7.0'), {'docker-compose'})

Minimal repro / logs

pip3 list

Package Version Location

---

appdirs 1.4.3
appnope 0.1.0
asn1crypto 1.1.0
atomicwrites 1.3.0
attrs 19.3.0
aws-cdk.assets 1.1.0
aws-cdk.aws-apigateway 1.1.0
aws-cdk.aws-applicationautoscaling 1.1.0
aws-cdk.aws-autoscaling 1.1.0
aws-cdk.aws-autoscaling-common 1.1.0
aws-cdk.aws-autoscaling-hooktargets 1.1.0
aws-cdk.aws-certificatemanager 1.1.0
aws-cdk.aws-cloudformation 1.1.0
aws-cdk.aws-cloudfront 1.1.0
aws-cdk.aws-cloudwatch 1.1.0
aws-cdk.aws-codebuild 1.1.0
aws-cdk.aws-codecommit 1.1.0
aws-cdk.aws-codedeploy 1.1.0
aws-cdk.aws-codepipeline 1.1.0
aws-cdk.aws-codepipeline-actions 1.1.0
aws-cdk.aws-ec2 1.1.0
aws-cdk.aws-ecr 1.1.0
aws-cdk.aws-ecr-assets 1.1.0
aws-cdk.aws-ecs 1.1.0
aws-cdk.aws-elasticloadbalancing 1.1.0
aws-cdk.aws-elasticloadbalancingv2 1.1.0
aws-cdk.aws-events 1.1.0
aws-cdk.aws-events-targets 1.1.0
aws-cdk.aws-iam 1.1.0
aws-cdk.aws-kms 1.1.0
aws-cdk.aws-lambda 1.1.0
aws-cdk.aws-logs 1.1.0
aws-cdk.aws-route53 1.1.0
aws-cdk.aws-route53-targets 1.1.0
aws-cdk.aws-s3 1.1.0
aws-cdk.aws-s3-assets 1.1.0
aws-cdk.aws-sagemaker 1.1.0
aws-cdk.aws-secretsmanager 1.1.0
aws-cdk.aws-servicediscovery 1.1.0
aws-cdk.aws-sns 1.1.0
aws-cdk.aws-sns-subscriptions 1.1.0
aws-cdk.aws-sqs 1.1.0
aws-cdk.aws-ssm 1.1.0
aws-cdk.aws-stepfunctions 1.1.0
aws-cdk.aws-stepfunctions-tasks 1.1.0
aws-cdk.core 1.1.0
aws-cdk.custom-resources 1.1.0
aws-cdk.cx-api 1.1.0
aws-cdk.region-info 1.1.0
awslogs 0.11.0
backcall 0.1.0
bcrypt 3.1.7
black 19.3b0
bleach 3.1.0
boto3 1.10.14
botocore 1.13.14
cached-property 1.5.1
cattrs 0.9.0
certifi 2019.9.11
cffi 1.13.2
chardet 3.0.4
Click 7.0
cloudpickle 1.2.2
cryptography 2.8
decorator 4.4.1
defusedxml 0.6.0
docker 4.0.2
docker-compose 1.24.1
docker-pycreds 0.4.0
dockerpty 0.4.1
docopt 0.6.2
docutils 0.15.2
entrypoints 0.3
fabric 2.5.0
flake8 3.7.8
flake8-blind-except 0.1.1
flake8-builtins 1.4.1
flake8-import-order 0.18.1
flake8-mypy 17.8.0
flake8-polyfill 1.0.2
idna 2.7
importlib-metadata 0.23
importlib-resources 1.0.2
invoke 1.3.0
ipykernel 5.1.3
ipython 7.9.0
ipython-genutils 0.2.0
ipywidgets 7.5.1
isort 4.3.18
jedi 0.15.1
Jinja2 2.10.1
jmespath 0.9.4
joblib 0.14.0
jsii 0.14.3
jsonschema 2.6.0
jupyter 1.0.0
jupyter-client 5.3.4
jupyter-console 6.0.0
jupyter-core 4.6.1
liquidpy 0.0.7
mantra 1.0.7.dev0 /Users/foleyjon/Documents/Repos/mantra
MarkupSafe 1.1.1
mccabe 0.6.1
mistune 0.8.4
more-itertools 7.2.0
mypy 0.720
mypy-extensions 0.4.1
mysql-connector-python 8.0.17
nbconvert 5.6.1
nbformat 4.4.0
nominis 1.0.0
notebook 6.0.1
numpy 1.17.3
packaging 19.2
pandas 0.24.2
pandocfilters 1.4.2
paramiko 2.6.0
parso 0.5.1
pep8-naming 0.8.2
pexpect 4.7.0
pickleshare 0.7.5
pip 19.0.3
pluggy 0.13.0
prometheus-client 0.7.1
prompt-toolkit 2.0.10
protobuf 3.10.0
protobuf3-to-dict 0.1.5
psycopg2-binary 2.8.2
ptyprocess 0.6.0
publication 0.0.3
py 1.8.0
pycodestyle 2.5.0
pycparser 2.19
pyflakes 2.1.1
Pygments 2.4.2
PyNaCl 1.3.0
pyparsing 2.4.2
pyrsistent 0.15.5
pytest 5.0.1
pytest-clarity 0.1.0a1
pytest-mock 1.10.4
python-dateutil 2.8.0
pytz 2019.3
PyYAML 3.13
pyzmq 18.1.0
qtconsole 4.5.5
requests 2.20.1
s3transfer 0.2.1
sagemaker 1.43.4.post1
scikit-learn 0.21.3
scipy 1.3.2
Send2Trash 1.5.0
setuptools 41.6.0
six 1.13.0
sklearn 0.0
termcolor 1.1.0
terminado 0.8.2
testpath 0.4.2
texttable 0.9.1
toml 0.10.0
tornado 6.0.3
tqdm 4.37.0
traitlets 4.3.3
typed-ast 1.4.0
typing-extensions 3.7.4
urllib3 1.24.3
wcwidth 0.1.7
webencodings 0.5.1
websocket-client 0.56.0
wheel 0.33.6
widgetsnbextension 3.5.1
xgboost 0.90
zipp 0.6.0

• Exact command to reproduce:

[jfelectron]

In fact these are related problems that sagemaker itself faced. The solution here is clearly to drop docker-compose, which should be wholly unnecessary.

c042a6c

[nadiaya]

Hi, thank you for the detail. docker-compose is needed for Local Mode, but I understand that it’s not needed for all Python SDK functionality.

We are aware of the issues caused by the current dependencies and we have plans on our roadmap to address them.

[nadiaya]

We moved docker-compose dependency under sagemaker[local] and it's no longer installed during regular installation.
I hope this has solved your issue.