Unable to pass eval_metrics to KMeans estimator

[rddefauw]

Please fill out the form below.

System Information

• Framework (e.g. TensorFlow) / Algorithm (e.g. KMeans): KMeans (Sagemaker built-in algorithm)
• Framework Version: n/a
• Python Version: Python 3.6.5 :: Anaconda, Inc.
• CPU or GPU: CPU
• Python SDK Version: sagemaker==1.28.3
• Are you using a custom image: no

Describe the problem

I am trying to pass in the eval_metrics parameter to the KMeans estimator:

kmeans = KMeans(role=role,
            train_instance_count=1,
            train_instance_type='ml.c4.xlarge',
            output_path='s3://____',              
            k=3,
           eval_metrics=["msd", "ssd"])

That's the example value for eval_metrics used in the unit test for KMeans. However, when I run the training job I get this error:

[06/27/2019 22:30:58 ERROR 139696703387456] Customer Error: Hyperparameter must be valid json, but found eval_metrics: (caused by ValueError)

Caused by: No JSON object could be decoded

I tried several formats including 'eval_metrics': '[\"msd\",\"ssd\"]'.

However I am able to pass in the parameters if I use boto3:

    import boto3
client = boto3.client('sagemaker')
response = client.create_training_job(
    TrainingJobName='rdevalmetrics',
    HyperParameters={
        'feature_dim': '34',
        'k': '3',
        'eval_metrics': '[\"msd\",\"ssd\"]'
    },
    AlgorithmSpecification={
        'TrainingImage': '174872318107.dkr.ecr.us-west-2.amazonaws.com/kmeans:1',
        'TrainingInputMode': 'File'
    },
    RoleArn='arn:aws:iam::____:role/service-role/AmazonSageMaker-ExecutionRole-20180717T085401',
    InputDataConfig=[
        {
            'ChannelName': 'train',
            'DataSource': {
                'S3DataSource': {
                    'S3DataType': 'ManifestFile',
                    'S3Uri': 's3://sagemaker-us-west-2-____/sagemaker-record-sets/KMeans-2019-06-27-22-48-57-424/.amazon.manifest',
                    'S3DataDistributionType': 'FullyReplicated'
                    
                }
            }
            
        }
    ],
    OutputDataConfig={
        
        'S3OutputPath': 's3://___/kmeanstest'
    },
    StoppingCondition={
        'MaxRuntimeInSeconds': 600
    },
    ResourceConfig={
        'InstanceType': 'ml.m4.xlarge',
        'InstanceCount': 1,
        'VolumeSizeInGB': 50,
    }
)

That job completed successfully.

Minimal repro / logs

See above code.

[icywang86rui]

@rddefauw When did you see the error? After creating the estimator or during training? I run the same code it seems to be working fine:

~/work/test$ cat test_kmeans.py
from sagemaker.amazon.kmeans import KMeans, KMeansPredictor

kmeans = KMeans(role='role',
                train_instance_count=1,
                train_instance_type='ml.c4.xlarge',
                output_path='s3://____',
                k=3,
           	eval_metrics=["msd", "ssd"])

print(kmeans.hyperparameters())
~/work/test$ python test_kmeans.py
{'force_dense': 'True', 'k': '3', 'eval_metrics': "['msd', 'ssd']"}

[rddefauw]

I saw it during training.

…

Sent from my iPhone On Jul 1, 2019, at 2:32 PM, icywang86rui <[email protected]<mailto:[email protected]>> wrote: @rddefauw<https://github.com/rddefauw> When did you see the error? After creating the estimator or during training? I run the same code it seems to be working fine: ~/work/test$ cat test_kmeans.py from sagemaker.amazon.kmeans import KMeans, KMeansPredictor kmeans = KMeans(role='role', train_instance_count=1, train_instance_type='ml.c4.xlarge', output_path='s3://____', k=3, eval_metrics=["msd", "ssd"]) print(kmeans.hyperparameters()) ~/work/test$ python test_kmeans.py {'force_dense': 'True', 'k': '3', 'eval_metrics': "['msd', 'ssd']"} — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#889?email_source=notifications&email_token=AIF3BNCIDZIUEVMUCYQ5ZLTP5JSRRA5CNFSM4H4A3OIKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODY7IOGI#issuecomment-507414297>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AIF3BNAMMPONDIOPTHOXOEDP5JSRRANCNFSM4H4A3OIA>.

[chuyang-deng]

Hi @rddefauw, can you make an object of hyperparameter and assign it to eval_metrics? From the information you provided, I think we need to make some changes in the Python SDK code. But at the mean time, assigning a hyperparameter object to eval_metrics might be a workaround.

Thanks.

[rddefauw]

I'm not entirely sure what you mean. An object of what type?

I tried passing in a dictionary as with the low level SDK:

kmeans = KMeans(role=role,
                train_instance_count=1,
                train_instance_type='ml.c4.xlarge',
                output_path='s3://rdnocdata/counties/',              
                k=num_clusters,
                eval_metrics = {
        'eval_metrics': '[\"msd\",\"ssd\"]'
    })

But I got the same output from kmeans.fit:

[07/08/2019 21:54:14 ERROR 139978699167552] Customer Error: Hyperparameter must be valid json, but found eval_metrics: (caused by ValueError)

Caused by: No JSON object could be decoded

[chuyang-deng]

Sorry I should've been more specific: an object of hyperparameter. https://github.com/aws/sagemaker-python-sdk/blob/master/src/sagemaker/amazon/kmeans.py#L43

[rddefauw]

I guess I'm not clear on what you're suggesting. The HP object is a descriptor so I tried just using the set method:

kmeans.eval_metrics = ["msd","ssd"]

But that resulted in the same error.

[ChoiByungWook]

Hello @rddefauw,

I just ran an example with the same eval_metrics hyperparameter value and was able to reproduce the error. Going to investigate now.

[07/10/2019 01:00:56 ERROR 139636592990016] Customer Error: Hyperparameter must be valid json, but found eval_metrics: (caused by ValueError)

Caused by: No JSON object could be decoded`

[ChoiByungWook]

@rddefauw,

It looks like a bug on our end. We end up sending the list as a string as defined here: https://github.com/aws/sagemaker-python-sdk/blob/master/src/sagemaker/amazon/hyperparameter.py#L70. On the algorithm side it expects the incoming list input to be JSON formatted, however we only serialize as a string, so some of the encoding gets lost.

Here is an easy repro showcasing this, which can be run using python -s.

CURRENT

incorrect_list_json = {'eval_metrics': str(['msd', 'ssd'])}
eval_metrics = hp['eval_metrics']
json.loads(raw_value) # ERROR

CORRECT

import json
correct_list_json = {'eval_metrics': json.dumps(['msd', 'ssd'])}
eval_metrics = hp['eval_metrics']
json.loads(raw_value) # CORRECT

The json.dumps() works with int, float, boolean and list. Which should handle all of the current existing hyperparameters correctly.

I'll submit a PR.

Thank you for bringing this to our attention.

[ChoiByungWook]

PR: #922

[laurenyu]

released in 1.34.0: https://github.com/aws/sagemaker-python-sdk/blob/master/CHANGELOG.md#v1340-2019-07-18