[Documentation Request] Sagemaker update requires new IAM permissions

[zmjjmz]

System Information

• Framework (e.g. TensorFlow) / Algorithm (e.g. KMeans): (n/a)
• Framework Version: (n/a)
• Python Version: 2.7.9
• CPU or GPU: (n/a)
• Python SDK Version: 1.5.3
• Are you using a custom image: (n/a)

Describe the problem

Our ops team has fairly locked down IAM permissions, and when initially using the Sagemaker SDK I had to trial-and-error which ones I needed. In the most recent version of the Sagemaker SDK PR 268 introduced a new requirement that any role used by the Sagemaker SDK needed iam:GetRole granted, which I did not have previously granted for this role. While it wasn't hard to get this role granted, it was a breaking change for us that I would have appreciated a warning for.

Is there documentation on what IAM permissions are used by the SDK? If so, I'd just like to request that new permissions end up in the changelog / with a warning so that it's not a surprise.

Minimal repro / logs

I don't have a good way to minimal repro this since of course you'd need our IAM setup, which is not doable :)

Here's the exact error I got though:

botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the GetRole operation: User: arn:aws:sts::<withheld>:assumed-role/sagemaker-user-role/
botocore-session-1531238236 is not authorized to perform: iam:GetRole on resource: role sagemaker-user-role

[wendtek]

I am also looking for a list of permissions needed to write a minimal policy. This would be very useful.

@zmjjmz any chance you can supply what permissions your team has narrowed it down to so far? It would help give me a jump start past the initial trial and error process. Thanks in advance.

[andremoeller]

Hi @zmjjmz ,

Thanks for the report! I sent a pull request to fix this behavior so that get_execution_role doesn't fail in this case: #305

We don't have documentation about the IAM permissions needed for the SDK, but you're right that we should add clear documentation on this.

@wendtek I believe (but haven't confirmed) that the Python SDK doesn't need additional permissions beyond those your SageMaker IAM role would need (with this exception), so adding the AmazonSageMakerFullAccess policy to your IAM role or IAM user would be a good start, plus potentially some additional S3 permissions.

Thanks again!

[harusametime]

@andremoeller Today I made a role with only AmazonSageMakerFullAccess policy in IAM and put the role onto a notebook instance. However, get_execution_role failed with a similar error message. This is probably because AmazonSageMakerFullAccess does not grant iam:GetRole.

[wendtek]

@andremoeller I'm looking for a minimal permission set, so using AmazonSageMakerFullAccess is not an option. We try to keep any user permissions pretty focused on what they're responsible for. I played around a bit yesterday and eventually landed with a much smaller list that a user needs to locally run a notebook with sagemaker, but we're also not using SageMaker's notebook instances or models, just training jobs.

{
  "Version": "2012-10-17",
  "Statement": [{
      "Effect": "Allow",
      "Action": [
        "sagemaker:AddTags",
        "sagemaker:CreateHyperParameterTuningJob",
        "sagemaker:CreateTrainingJob",
        "sagemaker:DeleteTags",
        "sagemaker:StopHyperParameterTuningJob",
        "sagemaker:StopTrainingJob"
      ],
      "Resource": [
        "arn:aws:sagemaker:<region>:<account-id>:hyper-parameter-tuning-job/<name>*",
        "arn:aws:sagemaker:<region>:<account-id>:training-job/<name>*"
      ]
    },
    {
        "Effect": "Allow",
        "Action": [
          "sagemaker:Describe*",
          "sagemaker:List*"

        ],
        "Resource": "*"
      },
    {
      "Effect": "Allow",
      "Action": [
        "logs:DescribeLogStreams",
        "logs:GetLogEvents"
      ],
      "Resource": "arn:aws:logs:*:*:log-group:/aws/sagemaker/*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "iam:PassRole"
      ],
      "Resource": "<sagemaker job role arn>",
      "Condition": {
        "StringEquals": {
          "iam:PassedToService": "sagemaker.amazonaws.com"
        }
      }
    },
    {
      "Effect": "Allow",
      "Action": [
        "kms:CreateGrant"
      ],
      "Resource": "*",
      "Condition": {
        "StringLike": {
          "kms:ViaService": [
            "ec2.*.amazonaws.com",
            "ec2.amazonaws.com",
            "s3.*.amazonaws.com",
            "s3.amazonaws.com",
            "sagemaker.*.amazonaws.com",
            "sagemaker.amazonaws.com"
          ]
        }
      }
    }
  ]
}

Not using KMS yet either, but we will be. The sagemaker service is actually not supported yet for the KMS condition keys. Running sagemaker 1.7.0 and I haven't run into needed the iam:GetRole perm yet.

[zmjjmz]

@wendtek it's a bit more complicated than a list of permissions, there's a lot of separate policies that we ended up enabling/creating.

AWS Managed:

• AmazonEC2ContainerRegistryReadOnly
• AmazonSageMakerFullAccess

Custom (separated into various logical policies):

• sts:AssumeRole for the developer accounts that have access
• Various S3 perms for the buckets we use for Sagemaker (R/W access largely)
• And recently (after this update) iam:GetRole, iam:GetRolePolicy, sts:GetCallerIdentity

[laurenyu]

Apologies again for the trouble this has caused. Updated the documentation in #575 (though perhaps we'll try and add iam:GetRole to AmazonSageMakerFullAccess down the line), so going to close this issue now. Feel free to reopen if necessary.

[abiodunjames]

I'm still having this error

[laurenyu]

@abiodunjames could you open a new issue in this repository? also, if you could include your code and the error/stacktrace, that would be helpful. thanks!