XGBoost estimator, security_group_ids parameter doesn't work.

[demetribu]

Describe the bug
I'm trying to submit a job from a local environment to Sagemaker in script mode, with custom VPC. I'm using "sagemaker.xgboost.estimator.XGBoost" and security_group_ids parameter. Job is submitted successfully, but without any changes regarding VPC, also I did notice non-errors in a log.

To reproduce

from sagemaker.session import Session
from sagemaker.inputs import TrainingInput
from sagemaker.xgboost.estimator import XGBoost
import boto3

boto_session = boto3.Session()
session = Session(boto_session=boto_session)

entry_point=""
source_dir="src.tar.gz"
role=""
instance_type=""
framework_version="1.0-1"
instance_count=None
security_group_ids=[""]

xgb_estimator = XGBoost(
    entry_point=entry_point,
    source_dir=source_dir,
    role=role,
    security_group_ids=security_group_ids,
    instance_count=instance_count,
    instance_type=instance_type,
    framework_version=framework_version
)

xgb_estimator.fit()

Expected behavior
There are non-errors, also as non-changes regarding VPC.
No custom VPC settings applied.

[ahsan-z-khan]

@dmitrybugakov Thank you for using Amazon SageMaker.

Which python version do you see the bug on?

[demetribu]

@ahsan-z-khan

python==3.8.6
boto3==1.16.59
sagemaker==2.24.0

[ChoiByungWook]

Hey @dmitrybugakov,

Apologies on the late response.

In order to use custom security groups, you will need to also provide corresponding subnets.

The Python SDK determines this during fit in this line:

sagemaker-python-sdk/src/sagemaker/job.py

Line 81 in e08c04e

vpc_config = estimator.get_vpc_config()

Which ends up calling:

sagemaker-python-sdk/src/sagemaker/estimator.py

Line 1245 in e08c04e

return vpc_utils.to_dict(self.subnets, self.security_group_ids)

and runs into this conditional:

sagemaker-python-sdk/src/sagemaker/vpc_utils.py

Line 26 in e08c04e

def to_dict(subnets, security_group_ids):

This is not a nice user experience, as it should fail if the user provides one of the two required inputs.

I have made note of this issue on our end.

[demetribu]

@ChoiByungWook thank you!
In case, I will have free time, and the issue will still exist, I'm going to fix that.

[pbendevis]

I am seeing that the subnet and security groups do not appear in the console (even when both are set using the xgboost estimator client). Seems like there not being forwarded. If I set them manually in the console I don't get errors so I don't think it is due silent failure