refactor(typing): enable boto3 implicit type annotations

[ericbn]

Issue number: #4607

Summary

Changes

Add boto3-stubs and botocore-stubs to enable implicit type annotations with mypy_boto3_* packages. See https://youtype.github.io/boto3_stubs_docs/mypy_boto3_s3/usage/#implicit-type-annotations

This enables removing very few unneeded explicit type annotations and helps detect some new errors automatically.

No need to do

boto3_config = boto3_config or Config()

before passing the boto3_config to the boto3_session client or resource methods, as None is an accepted value.

Refactor the aws_lambda_powertools.utilities.parameters.base.BaseProvider class: move the instantiation of clients in _build_boto3_client and _build_boto3_resource_client to the subclasses and the user_agent method calls to the constructor. Keeping correct typing information with the original implementation was going to be too complex and hopefully the refactor even made the code tidier.

Keep imports consistent: use boto3.session.Session, mypy_boto3_*.client, mypy_boto3_*.service_resource and mypy_boto3_*.type_defs as documented at https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html and https://youtype.github.io/boto3_stubs_docs/mypy_boto3_s3/usage/#explicit-type-annotations

User experience

The initial motivation of this PR is what we agreed as step 1 of #4607:

1. Review the use of mypy_boto3_* types, and try to keep their imports to a minimum, to only when those types cannot be discovered automatically by mypy.

In the process of reviewing the respective code, I've noticed that implicit type annotations for mypy_boto3_* packages was not enabled, due to missing dev dependencies as mentioned above. Adding this allows both reducing the need to explicitly annotate these types -- which we were not doing much anyway -- and letting types be discovered automatically -- which we were missing, given some errors were discovered. All new errors reported by mypy were fixed, unless for one. I'm not sure if this is intentional in the code:

aws_lambda_powertools/utilities/idempotency/persistence/dynamodb.py: note: In member "_put_record" of class "DynamoDBPersistenceLayer":
aws_lambda_powertools/utilities/idempotency/persistence/dynamodb.py:251:74: error: TypedDict "_ClientErrorResponseTypeDef" has no key "Item"  [typeddict-item]
                    old_data_record = self._item_to_data_record(exc.response["Item"]) if "Item" in exc.response else None
                                                                             ^~~~~~
Found 1 error in 1 file (checked 616 source files)
make: *** [mypy] Error 1

Checklist

If your change doesn't seem to apply, please leave them unchecked.

• Meet tenets criteria
• I have performed a self-review of this change
• Changes have been tested
• Changes are documented
• PR title follows conventional commit semantics

Is this a breaking change?

RFC issue number:

Checklist:

• Migration process documented
• Implement warnings (if it can live side by side)

Acknowledgment

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Disclaimer: We value your time and bandwidth. As such, any pull requests created on non-triaged issues might not be successful.

[leandrodamascena]

Hello @ericbn! I'm planning to review this PR right after we finish #4606!

[leandrodamascena]

I will start the review tomorrow morning. 🚀

[leandrodamascena]

Hi @ericbn! I've started looking into this PR. Everything looks great, but I see that you refactored an important part of the code where we register the UserAgent in the session. This code is critical and I'll need a few days to do some exploratory testing to see if we won't have any side effects.

As always, you have great attention to detail and your code is elegant and clean, but I need to be 100% sure that refactoring this part of the code won't have any issues that our tests haven't caught.

[leandrodamascena]

mypy is complaining about something, but it seems easy to fix. Can you take a look, please @ericbn?

[ericbn]

Cool! @leandrodamascena, I've updated the code to ignore that mypy error, since this was a case for a DynamoDB request with ReturnValuesOnConditionCheckFailure, for which the response type was not covered by mypy_boto3_dynamodb.

[leandrodamascena]

Hello @ericbn! The only thing missing is to review the Idempotency change in the put_record method, but the other changes are fantastic! Thank you for improving the quality of the project in development environments, making the project more compatible with python styles and removing unnecessary code.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

[sthulb]

Overall, I think this PR is a win.

[leandrodamascena]

Another amazing PR @ericbn! I'm more than happy to say APPROVED!!!