feature: Add python 3.9 and spark 3.2 support for spark processor

[xiaoxshe]

Issue #, if available:

Issue: We are not able to use newer version of Spark image because validation failure on the SageMaker Python SDK side.

Description of changes:

I deep dive into the code. This file should support our current Spark set up which doesn't right now. It should support python 3.9 and Spark 3.2 according to processing spark image.

Testing done:

Merge Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.

General

• [X ] I have read the CONTRIBUTING doc
• I certify that the changes I am introducing will be backward compatible, and I have discussed concerns about this, if any, with the Python SDK team
• [X ] I used the commit message format described in CONTRIBUTING
• [X ] I have passed the region in to all S3 and STS clients that I've initialized as part of this change.
• [ X] I have updated any necessary documentation, including READMEs and API docs (if appropriate)

Tests

• [ X] I have added tests that prove my fix is effective or that my feature works (if appropriate)
• [ X] I have added unit and/or integration tests as appropriate to ensure backward compatibility of the changes
• [ X] I have checked that my tests are not configured for a specific region or account (if appropriate)
• [ X] I have used unique_name_from_base to create resource names in integ tests (if appropriate)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[sagemaker-bot]

AWS CodeBuild CI Report

• CodeBuild project: sagemaker-python-sdk-notebook-tests
• Commit ID: c51dcf7
• Result: SUCCEEDED
• Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

[sagemaker-bot]

AWS CodeBuild CI Report

• CodeBuild project: sagemaker-python-sdk-local-mode-tests
• Commit ID: c51dcf7
• Result: SUCCEEDED
• Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

[sagemaker-bot]

AWS CodeBuild CI Report

• CodeBuild project: sagemaker-python-sdk-slow-tests
• Commit ID: c51dcf7
• Result: SUCCEEDED
• Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

[sagemaker-bot]

AWS CodeBuild CI Report

• CodeBuild project: sagemaker-python-sdk-unit-tests
• Commit ID: c51dcf7
• Result: SUCCEEDED
• Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

[codecov-commenter]

Codecov Report

Merging #3643 (c51dcf7) into master (8d2c16b) will decrease coverage by 0.80%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #3643      +/-   ##
==========================================
- Coverage   89.59%   88.80%   -0.80%     
==========================================
  Files         968      227     -741     
  Lines       90084    22278   -67806     
==========================================
- Hits        80707    19783   -60924     
+ Misses       9377     2495    -6882     

Impacted Files	Coverage Δ
...hon3.9/site-packages/sagemaker/local/exceptions.py
....10/site-packages/sagemaker/workflow/parameters.py
...maker/cli/compatibility/v2/modifiers/image_uris.py
...ython3.8/site-packages/sagemaker/mxnet/defaults.py
.../site-packages/sagemaker/model_monitor/__init__.py
.../site-packages/sagemaker/experiments/_api_types.py
...thon3.10/site-packages/sagemaker/instance_group.py
...ite-packages/sagemaker/experiments/_run_context.py
...on3.9/site-packages/sagemaker/amazon/record_pb2.py
...on3.8/site-packages/sagemaker/debugger/__init__.py
... and 1185 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[sagemaker-bot]

AWS CodeBuild CI Report

• CodeBuild project: sagemaker-python-sdk-pr
• Commit ID: c51dcf7
• Result: SUCCEEDED
• Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

[cfregly]

@knikure any update? will this be merged soon? thanks!

[knikure]

@knikure any update? will this be merged soon? thanks!

Can we get review from someone in the team on this as per directions earlier? I am fine to merge it after that.

[sagemaker-bot]

AWS CodeBuild CI Report

• CodeBuild project: sagemaker-python-sdk-unit-tests
• Commit ID: c51dcf7
• Result: SUCCEEDED
• Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

[sagemaker-bot]

AWS CodeBuild CI Report

• CodeBuild project: sagemaker-python-sdk-local-mode-tests
• Commit ID: c51dcf7
• Result: SUCCEEDED
• Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

[sagemaker-bot]

AWS CodeBuild CI Report

• CodeBuild project: sagemaker-python-sdk-notebook-tests
• Commit ID: c51dcf7
• Result: SUCCEEDED
• Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

[sagemaker-bot]

AWS CodeBuild CI Report

• CodeBuild project: sagemaker-python-sdk-slow-tests
• Commit ID: c51dcf7
• Result: FAILED
• Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

[sagemaker-bot]

AWS CodeBuild CI Report

• CodeBuild project: sagemaker-python-sdk-pr
• Commit ID: c51dcf7
• Result: FAILED
• Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository