diff --git a/doc/frameworks/rl/using_rl.rst b/doc/frameworks/rl/using_rl.rst index 8937f18cb3..d146b91fb0 100644 --- a/doc/frameworks/rl/using_rl.rst +++ b/doc/frameworks/rl/using_rl.rst @@ -19,20 +19,27 @@ Training RL models using ``RLEstimator`` is a two-step process: You should prepare your script in a separate source file than the notebook, terminal session, or source file you're using to submit the script to SageMaker via an ``RLEstimator``. This will be discussed in further detail below. -Suppose that you already have a training script called ``coach-train.py``. +Suppose that you already have a training script called ``coach-train.py`` and have an RL image in your ECR registry +called ``123123123123.dkr.ecr.us-west-2.amazonaws.com/your-rl-registry:your-cool-image-tag`` in ``us-west-2`` region. You can then create an ``RLEstimator`` with keyword arguments to point to this script and define how SageMaker runs it: .. code:: python from sagemaker.rl import RLEstimator, RLToolkit, RLFramework - rl_estimator = RLEstimator(entry_point='coach-train.py', - toolkit=RLToolkit.COACH, - toolkit_version='0.11.1', - framework=RLFramework.TENSORFLOW, - role='SageMakerRole', - instance_type='ml.p3.2xlarge', - instance_count=1) + # Train my estimator + rl_estimator = RLEstimator( + entry_point='coach-train.py', + image_uri='123123123123.dkr.ecr.us-west-2.amazonaws.com/your-rl-registry:your-cool-image-tag', + role='SageMakerRole', + instance_type='ml.c4.2xlarge', + instance_count=1 + ) + + +.. tip:: + Refer to `SageMaker RL Docker Containers <#sagemaker-rl-docker-containers>`_ for the more information on how to + build your custom RL image. After that, you simply tell the estimator to start a training job: @@ -40,6 +47,7 @@ After that, you simply tell the estimator to start a training job: rl_estimator.fit() + In the following sections, we'll discuss how to prepare a training script for execution on SageMaker and how to run that script on SageMaker using ``RLEstimator``. @@ -47,21 +55,20 @@ and how to run that script on SageMaker using ``RLEstimator``. Preparing the RL Training Script ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Your RL training script must be a Python 3.5 compatible source file from MXNet framework or Python 3.6 for TensorFlow. - The training script is very similar to a training script you might run outside of SageMaker, but you can access useful properties about the training environment through various environment variables, such as -* ``SM_MODEL_DIR``: A string representing the path to the directory to write model artifacts to. - These artifacts are uploaded to S3 for model hosting. -* ``SM_NUM_GPUS``: An integer representing the number of GPUs available to the host. -* ``SM_OUTPUT_DATA_DIR``: A string representing the filesystem path to write output artifacts to. Output artifacts may - include checkpoints, graphs, and other files to save, not including model artifacts. These artifacts are compressed - and uploaded to S3 to the same S3 prefix as the model artifacts. +* ``SM_MODEL_DIR``: A string representing the path to the directory to write model artifacts to. + These artifacts are uploaded to S3 for model hosting. +* ``SM_NUM_GPUS``: An integer representing the number of GPUs available to the host. +* ``SM_OUTPUT_DATA_DIR``: A string representing the filesystem path to write output artifacts to. Output artifacts may + include checkpoints, graphs, and other files to save, not including model artifacts. These artifacts are compressed + and uploaded to S3 to the same S3 prefix as the model artifacts. For the exhaustive list of available environment variables, see the `SageMaker Containers documentation `__. +Note that your RL training script must have the Python version compatible with your custom RL Docker image. RL Estimators ------------- @@ -81,28 +88,16 @@ these in the constructor, either positionally or as keyword arguments. endpoints use this role to access training data and model artifacts. After the endpoint is created, the inference code might use the IAM role, if accessing AWS resource. -- ``instance_count`` Number of Amazon EC2 instances to use for - training. +- ``instance_count`` Number of Amazon EC2 instances to use for training. - ``instance_type`` Type of EC2 instance to use for training, for example, 'ml.m4.xlarge'. -You must as well include either: - -- ``toolkit`` RL toolkit (Ray RLlib or Coach) you want to use for executing your model training code. - -- ``toolkit_version`` RL toolkit version you want to be use for executing your model training code. - -- ``framework`` Framework (MXNet or TensorFlow) you want to be used as - a toolkit backed for reinforcement learning training. - -or provide: +You must also provide: -- ``image_uri`` An alternative Docker image to use for training and - serving. If specified, the estimator will use this image for training and - hosting, instead of selecting the appropriate SageMaker official image based on - framework_version and py_version. Refer to: `SageMaker RL Docker Containers - <#sagemaker-rl-docker-containers>`_ for details on what the Official images support - and where to find the source code to build your custom image. +- ``image_uri`` An alternative Docker image to use for training and serving. + If specified, the estimator will use this image for training and + hosting. Refer to: `SageMaker RL Docker Containers <#sagemaker-rl-docker-containers>`_ + for the source code to build your custom RL image. Optional arguments @@ -140,10 +135,8 @@ Deploying RL Models After an RL Estimator has been fit, you can host the newly created model in SageMaker. After calling ``fit``, you can call ``deploy`` on an ``RLEstimator`` Estimator to create a SageMaker Endpoint. -The Endpoint runs one of the SageMaker-provided model server based on the ``framework`` parameter -specified in the ``RLEstimator`` constructor and hosts the model produced by your training script, -which was run when you called ``fit``. This was the model you saved to ``model_dir``. -In case if ``image_uri`` was specified it would use provided image for the deployment. +The Endpoint runs provided image specified with ``image_uri`` and hosts the model produced by your +training script, which was run when you called ``fit``. This is the model you saved to ``model_dir``. ``deploy`` returns a ``sagemaker.mxnet.MXNetPredictor`` for MXNet or ``sagemaker.tensorflow.TensorFlowPredictor`` for TensorFlow. @@ -153,19 +146,22 @@ In case if ``image_uri`` was specified it would use provided image for the deplo .. code:: python # Train my estimator - rl_estimator = RLEstimator(entry_point='coach-train.py', - toolkit=RLToolkit.COACH, - toolkit_version='0.11.0', - framework=RLFramework.MXNET, - role='SageMakerRole', - instance_type='ml.c4.2xlarge', - instance_count=1) + region = 'us-west-2' # the AWS region of your training job + rl_estimator = RLEstimator( + entry_point='coach-train.py', + image_uri=f'123123123123.dkr.ecr.{region}.amazonaws.com/your-rl-registry:your-cool-image-tag', + role='SageMakerRole', + instance_type='ml.c4.2xlarge', + instance_count=1 + ) rl_estimator.fit() # Deploy my estimator to a SageMaker Endpoint and get a MXNetPredictor - predictor = rl_estimator.deploy(instance_type='ml.m4.xlarge', - initial_instance_count=1) + predictor = rl_estimator.deploy( + instance_type='ml.m4.xlarge', + initial_instance_count=1 + ) response = predictor.predict(data) @@ -193,10 +189,8 @@ attach will block and display log messages from the training job, until the trai The ``attach`` method accepts the following arguments: -- ``training_job_name:`` The name of the training job to attach - to. -- ``sagemaker_session:`` The Session used - to interact with SageMaker +- ``training_job_name:`` The name of the training job to attach to. +- ``sagemaker_session:`` The Session used to interact with SageMaker RL Training Examples -------------------- @@ -212,4 +206,6 @@ These are also available in SageMaker Notebook Instance hosted Jupyter notebooks SageMaker RL Docker Containers ------------------------------ -For more about the Docker images themselves, visit `the SageMaker RL containers repository `_. +For more information about how build your own RL image and use script mode with your image, see +`Building your image section on SageMaker RL containers repository `_ +and `Bring your own model with Amazon SageMaker script mode `_. diff --git a/src/sagemaker/image_uris.py b/src/sagemaker/image_uris.py index 143ecc9bdb..b542e871dd 100644 --- a/src/sagemaker/image_uris.py +++ b/src/sagemaker/image_uris.py @@ -31,6 +31,7 @@ GRAVITON_ALLOWED_TARGET_INSTANCE_FAMILY, GRAVITON_ALLOWED_FRAMEWORKS, ) +from sagemaker.deprecations import deprecation_warn logger = logging.getLogger(__name__) @@ -45,6 +46,13 @@ DATA_WRANGLER_FRAMEWORK = "data-wrangler" STABILITYAI_FRAMEWORK = "stabilityai" SAGEMAKER_TRITONSERVER_FRAMEWORK = "sagemaker-tritonserver" +RL_FRAMEWORKS = [ + "coach-tensorflow", + "coach-mxnet", + "ray-tensorflow", + "ray-pytorch", + "vw", +] @override_pipeline_parameter_var @@ -188,6 +196,12 @@ def retrieve( ) _validate_arg(full_base_framework_version, list(version_config.keys()), "base framework") version_config = version_config.get(full_base_framework_version) + elif framework in RL_FRAMEWORKS: + deprecation_warn( + "SageMaker-hosted RL images no longer accept new pull requests and", + "April 2024", + " Please pass in `image_uri` to use RLEstimator", + ) py_version = _validate_py_version_and_set_if_needed(py_version, version_config, framework) version_config = version_config.get(py_version) or version_config diff --git a/src/sagemaker/rl/estimator.py b/src/sagemaker/rl/estimator.py index 3ed539fa2e..76ed019a80 100644 --- a/src/sagemaker/rl/estimator.py +++ b/src/sagemaker/rl/estimator.py @@ -25,6 +25,7 @@ from sagemaker.tensorflow.model import TensorFlowModel from sagemaker.vpc_utils import VPC_CONFIG_DEFAULT from sagemaker.workflow.entities import PipelineVariable +from sagemaker.deprecations import removed_function, deprecation_warn logger = logging.getLogger("sagemaker") @@ -70,9 +71,9 @@ class RLFramework(enum.Enum): class RLEstimator(Framework): """Handle end-to-end training and deployment of custom RLEstimator code.""" - COACH_LATEST_VERSION_TF = "0.11.1" - COACH_LATEST_VERSION_MXNET = "0.11.0" - RAY_LATEST_VERSION = "1.6.0" + COACH_LATEST_VERSION_TF = removed_function("COACH_LATEST_VERSION_TF") + COACH_LATEST_VERSION_MXNET = removed_function("COACH_LATEST_VERSION_MXNET") + RAY_LATEST_VERSION = removed_function("RAY_LATEST_VERSION") def __init__( self, @@ -84,7 +85,7 @@ def __init__( hyperparameters: Optional[Dict[str, Union[str, PipelineVariable]]] = None, image_uri: Optional[Union[str, PipelineVariable]] = None, metric_definitions: Optional[List[Dict[str, Union[str, PipelineVariable]]]] = None, - **kwargs + **kwargs, ): """Creates an RLEstimator for managed Reinforcement Learning (RL). @@ -112,11 +113,23 @@ def __init__( must point to a file located at the root of ``source_dir``. toolkit (sagemaker.rl.RLToolkit): RL toolkit you want to use for executing your model training code. - toolkit_version (str): RL toolkit version you want to be use for - executing your model training code. + + .. warning:: + This ``toolkit`` argument discontinued support for new RL users on April 2024. + To use RLEstimator, pass in ``image_uri``. + toolkit_version (str): RL toolkit version you want to be use for executing your + model training code. + + .. warning:: + This ``toolkit_version`` argument discontinued support for new RL users on + April 2024. To use RLEstimator, pass in ``image_uri``. framework (sagemaker.rl.RLFramework): Framework (MXNet or TensorFlow) you want to be used as a toolkit backed for reinforcement learning training. + + .. warning:: + This ``framework`` argument discontinued support for new RL users on April + 2024. To use RLEstimator, pass in ``image_uri``. source_dir (str or PipelineVariable): Path (absolute, relative or an S3 URI) to a directory with any other training source code dependencies aside from the entry point file (default: None). If ``source_dir`` is an S3 URI, it must @@ -127,20 +140,25 @@ def __init__( accessible as a dict[str, str] to the training code on SageMaker. For convenience, this accepts other types for keys and values. - image_uri (str or PipelineVariable): An ECR url. If specified, the estimator will use - this image for training and hosting, instead of selecting the - appropriate SageMaker official image based on framework_version - and py_version. Example: - 123.dkr.ecr.us-west-2.amazonaws.com/my-custom-image:1.0 + image_uri (str or PipelineVariable): An ECR url for an image the estimator would use + for training and hosting. + Example: 123.dkr.ecr.us-west-2.amazonaws.com/my-custom-image:1.0 metric_definitions (list[dict[str, str] or list[dict[str, PipelineVariable]]): A list of dictionaries that defines the metric(s) used to evaluate the - training jobs. Each dictionary contains two keys: 'Name' for the name of the metric, - and 'Regex' for the regular expression used to extract the + training jobs. Each dictionary contains two keys: 'Name' for the name of the + metric, and 'Regex' for the regular expression used to extract the metric from the logs. This should be defined only for jobs that don't use an Amazon algorithm. **kwargs: Additional kwargs passed to the :class:`~sagemaker.estimator.Framework` constructor. + .. seealso:: + For more information about how build your own RL image and use script mode with + your image, see `Building your image on sagemaker-rl-container + `_ + and `Bring your own model with Amazon SageMaker script mode + `_ + .. tip:: You can find additional parameters for initializing this class at @@ -149,6 +167,25 @@ def __init__( """ self._validate_images_args(toolkit, toolkit_version, framework, image_uri) + if toolkit: + deprecation_warn( + "The argument `toolkit`", + "April 2024", + " Pass in `image_uri` to use RLEstimator", + ) + if toolkit_version: + deprecation_warn( + "The argument `toolkit_version`", + "April 2024", + " Pass in `image_uri` to use RLEstimator", + ) + if framework: + deprecation_warn( + "The argument `framework`", + "April 2024", + " Pass in `image_uri` to use RLEstimator", + ) + if not image_uri: self._validate_toolkit_support(toolkit.value, toolkit_version, framework.value) self.toolkit = toolkit.value @@ -168,7 +205,7 @@ def __init__( hyperparameters, image_uri=image_uri, metric_definitions=metric_definitions, - **kwargs + **kwargs, ) def create_model( @@ -178,7 +215,7 @@ def create_model( entry_point=None, source_dir=None, dependencies=None, - **kwargs + **kwargs, ): """Create a SageMaker ``RLEstimatorModel`` object that can be deployed to an Endpoint. @@ -236,7 +273,7 @@ def create_model( base_args["name"] = self._get_or_create_name(kwargs.get("name")) if not entry_point and (source_dir or dependencies): - raise AttributeError("Please provide an `entry_point`.") + raise AttributeError("Provide an `entry_point`.") entry_point = entry_point or self._model_entry_point() source_dir = source_dir or self._model_source_dir() @@ -266,9 +303,7 @@ def create_model( return MXNetModel( framework_version=self.framework_version, py_version=PYTHON_VERSION, **extended_args ) - raise ValueError( - "An unknown RLFramework enum was passed in. framework: {}".format(self.framework) - ) + raise ValueError(f"An unknown RLFramework enum was passed in. framework: {self.framework}") def training_image_uri(self): """Return the Docker image to use for training. @@ -325,10 +360,9 @@ def _prepare_init_params_from_job_description(cls, job_details, model_channel_na toolkit, toolkit_version = cls._toolkit_and_version_from_tag(tag) if not cls._is_combination_supported(toolkit, toolkit_version, framework): + training_job_name = job_details["TrainingJobName"] raise ValueError( - "Training job: {} didn't use image for requested framework".format( - job_details["TrainingJobName"] - ) + f"Training job: {training_job_name} didn't use image for requested framework" ) init_params["toolkit"] = RLToolkit(toolkit) @@ -368,9 +402,7 @@ def _validate_framework_format(cls, framework): """Placeholder docstring.""" if framework and framework not in list(RLFramework): raise ValueError( - "Invalid type: {}, valid RL frameworks types are: {}".format( - framework, list(RLFramework) - ) + f"Invalid type: {framework}, valid RL frameworks types are: {list(RLFramework)}" ) @classmethod @@ -378,7 +410,7 @@ def _validate_toolkit_format(cls, toolkit): """Placeholder docstring.""" if toolkit and toolkit not in list(RLToolkit): raise ValueError( - "Invalid type: {}, valid RL toolkits types are: {}".format(toolkit, list(RLToolkit)) + f"Invalid type: {toolkit}, valid RL toolkits types are: {list(RLToolkit)}" ) @classmethod @@ -396,11 +428,8 @@ def _validate_images_args(cls, toolkit, toolkit_version, framework, image_uri): if not framework: not_found_args.append("framework") if not_found_args: - raise AttributeError( - "Please provide `{}` or `image_uri` parameter.".format( - "`, `".join(not_found_args) - ) - ) + not_found_args_joined = "`, `".join(not_found_args) + raise AttributeError(f"Provide `{not_found_args_joined}` or `image_uri` parameter.") else: found_args = [] if toolkit: @@ -431,9 +460,8 @@ def _validate_toolkit_support(cls, toolkit, toolkit_version, framework): """Placeholder docstring.""" if not cls._is_combination_supported(toolkit, toolkit_version, framework): raise AttributeError( - "Provided `{}-{}` and `{}` combination is not supported.".format( - toolkit, toolkit_version, framework - ) + f"Provided `{toolkit}-{toolkit_version}` and `{framework}` combination is" + " not supported." ) def _image_framework(self): @@ -463,7 +491,7 @@ def default_metric_definitions(cls, toolkit): float_regex = "[-+]?[0-9]*[.]?[0-9]+([eE][-+]?[0-9]+)?" # noqa: W605, E501 return [ - {"Name": "episode_reward_mean", "Regex": "episode_reward_mean: (%s)" % float_regex}, - {"Name": "episode_reward_max", "Regex": "episode_reward_max: (%s)" % float_regex}, + {"Name": "episode_reward_mean", "Regex": f"episode_reward_mean: ({float_regex})"}, + {"Name": "episode_reward_max", "Regex": f"episode_reward_max: ({float_regex})"}, ] - raise ValueError("An unknown RLToolkit enum was passed in. toolkit: {}".format(toolkit)) + raise ValueError(f"An unknown RLToolkit enum was passed in. toolkit: {toolkit}") diff --git a/tests/unit/sagemaker/image_uris/test_rl.py b/tests/unit/sagemaker/image_uris/test_rl.py index 88d4059ec3..85617438b1 100644 --- a/tests/unit/sagemaker/image_uris/test_rl.py +++ b/tests/unit/sagemaker/image_uris/test_rl.py @@ -13,6 +13,7 @@ from __future__ import absolute_import import pytest +from unittest.mock import patch from sagemaker import image_uris from tests.unit.sagemaker.image_uris import expected_uris @@ -45,17 +46,25 @@ def test_rl_image_uris(load_config_and_file_name): instance_type = INSTANCE_TYPES[processor] for py_version in py_versions: for region in ACCOUNTS.keys(): - uri = image_uris.retrieve( - framework, region, version=version, instance_type=instance_type - ) + with patch("logging.Logger.warning") as mocked_warning_log: + uri = image_uris.retrieve( + framework, region, version=version, instance_type=instance_type + ) - expected = expected_uris.framework_uri( - repo, - tag_prefix, - ACCOUNTS[region], - py_version=py_version, - processor=processor, - region=region, - ) + expected = expected_uris.framework_uri( + repo, + tag_prefix, + ACCOUNTS[region], + py_version=py_version, + processor=processor, + region=region, + ) - assert uri == expected + mocked_warning_log.assert_called_once_with( + "SageMaker-hosted RL images no longer accept new pull requests and will be " + "deprecated on April 2024." + " Please pass in `image_uri` to use RLEstimator in sagemaker>=2.\n" + "See: https://sagemaker.readthedocs.io/en/stable/v2.html for details." + ) + mocked_warning_log.reset_mock() + assert uri == expected diff --git a/tests/unit/test_rl.py b/tests/unit/test_rl.py index d9c4129cf6..1c9b261e22 100644 --- a/tests/unit/test_rl.py +++ b/tests/unit/test_rl.py @@ -602,7 +602,7 @@ def test_missing_required_parameters(sagemaker_session): instance_type=INSTANCE_TYPE, ) assert ( - "Please provide `toolkit`, `toolkit_version`, `framework`" + " or `image_uri` parameter." + "Provide `toolkit`, `toolkit_version`, `framework`" + " or `image_uri` parameter." in str(e.value) )