ScriptProcessor in Local Mode

[aiwarrior-23]

Hi,

it would be awesome if we can use ScriptProcessor in Local Mode. Currently it throws error.

ERROR: 'LocalSagemakerClient' object has no attribute 'create_processing_job'

[knakad]

Thanks for reaching out!
This is on our radar and we currently have a backlog item tracking it.
Although I don't have any dates to share, our backlog is constantly being re-prioritized, and feedback such as yours is instrumental in that process =)

(Reference: MLFW-2850)

[ram-nadella]

Just ran into this issue, would be great to have local mode for Processing, similar to training and inference serving have it.

Got this exception when I tried to use 'local' for instance_type:

ClientError: An error occurred (ValidationException) when calling the CreateProcessingJob operation: 1 validation error detected: Value 'local' at 'processingResources.clusterConfig.instanceType' failed to satisfy constraint: Member must satisfy enum value set: [ml.r5.12xlarge, ml.m5.4xlarge, ml.p2.xlarge, ml.m4.16xlarge, ml.r5.24xlarge, ml.t3.xlarge, ml.r5.16xlarge, ml.m5.large, ml.p3.16xlarge, ml.p2.16xlarge, ml.c4.2xlarge, ml.c5.2xlarge, ml.c4.4xlarge, ml.c5.4xlarge, ml.c4.8xlarge, ml.c5.9xlarge, ml.c5.xlarge, ml.c4.xlarge, ml.t3.2xlarge, ml.t3.medium, ml.c5.18xlarge, ml.r5.2xlarge, ml.p3.2xlarge, ml.m5.xlarge, ml.m4.10xlarge, ml.r5.4xlarge, ml.m5.12xlarge, ml.m4.xlarge, ml.t3.large, ml.m5.24xlarge, ml.m4.2xlarge, ml.m5.2xlarge, ml.p2.8xlarge, ml.r5.8xlarge, ml.r5.xlarge, ml.r5.large, ml.p3.8xlarge, ml.m4.4xlarge]

Is there something the community can contribute? If you could share some pointers to how training/serving do it, can review to see if this is something I can help with.

[yuanzhua]

Thank you for reaching out. Please use SageMaker Studio for now, and we will prioritize this accordingly.

[laurenyu]

Is there something the community can contribute? If you could share some pointers to how training/serving do it, can review to see if this is something I can help with.

all of the code related to local mode is in src/sagemaker/local.

using training as an example, the LocalSageMakerClient implements create_training_job: https://github.com/aws/sagemaker-python-sdk/blob/master/src/sagemaker/local/local_session.py#L66-L103.

That then ties to a _LocalTrainingJob entity: https://github.com/aws/sagemaker-python-sdk/blob/master/src/sagemaker/local/entities.py#L42-L110. This ultimately calls the train method of _SageMakerContainer: https://github.com/aws/sagemaker-python-sdk/blob/master/src/sagemaker/local/image.py#L101-L173

[rebeku]

It would be really great to have this feature available. My team is trying to evaluate whether we want to keep using SageMaker or if it introduces more problems than it solves. Being able to run our ScriptProcessors in local mode would make us more likely to keep using the product.

[perdasilva]

@laurenyu the convention for the containers seems to be to call docker-entrypoint with either "train" or "serve". Is there a convention for processing jobs as well? Or do just run their own entrypoint?

[laurenyu]

@perdasilva it seems like Processing Jobs simply run the Docker container without any default command like "train" or "serve":

Amazon SageMaker Processing runs your processing container image in a similar way as the following command, where AppSpecification.ImageUri is the Amazon ECR image URI that you specify in a CreateProcessingJob operation.

docker run [AppSpecification.ImageUri]

This command runs the ENTRYPOINT command configured in your Docker image.

https://docs.aws.amazon.com/sagemaker/latest/dg/build-your-own-processing-container.html

[MrTomerLevi]

as a workaround you can: !pip install sagemaker -U inside a SageMaker notebook

[gerold-csendes-epam]

hi, any updates on this one? It is strange because I could actually run the job locally?

By the way, is there any documentation on the local processing job? I am missing a good description

[gerold-csendes-epam]

okay I think it has been taken care of here #1961

[martinRenou]

Closing as fixed by #1961