Merge branch 'develop' into feat/reduce-batch-boilerplate

* develop:
  fix(batch): handle early validation errors for pydantic models (poison pill) #2091 (#2099)
  update changelog with latest changes
  docs(homepage): remove banner for end-of-support v1 (#2098)
  chore(deps-dev): bump aws-cdk-lib from 2.72.1 to 2.73.0 (#2097)
  chore(deps-dev): bump filelock from 3.10.7 to 3.11.0 (#2094)
  chore(deps-dev): bump coverage from 7.2.2 to 7.2.3 (#2092)
  chore(deps-dev): bump aws-cdk from 2.72.1 to 2.73.0 (#2093)
  chore(deps-dev): bump mypy-boto3-cloudformation from 1.26.60 to 1.26.108 (#2095)

Signed-off-by: heitorlessa <[email protected]>

Author: heitorlessa

CHANGELOG.md

@@ -6,6 +6,7 @@
 
 ## Documentation
 
+* **homepage:** remove banner for end-of-support v1 ([#2098](https://github.com/awslabs/aws-lambda-powertools-python/issues/2098))
 * **idempotency:** fixes to testing your code section ([#2073](https://github.com/awslabs/aws-lambda-powertools-python/issues/2073))
 * **idempotency:** new sequence diagrams, fix idempotency record vs DynamoDB TTL confusion ([#2074](https://github.com/awslabs/aws-lambda-powertools-python/issues/2074))
 * **parser:** fix highlighted line ([#2064](https://github.com/awslabs/aws-lambda-powertools-python/issues/2064))
@@ -18,13 +19,18 @@
 
 * **deps:** bump aws-xray-sdk from 2.11.0 to 2.12.0 ([#2080](https://github.com/awslabs/aws-lambda-powertools-python/issues/2080))
 * **deps:** bump peaceiris/actions-gh-pages from 3.9.2 to 3.9.3 ([#2069](https://github.com/awslabs/aws-lambda-powertools-python/issues/2069))
+* **deps-dev:** bump aws-cdk-lib from 2.72.1 to 2.73.0 ([#2097](https://github.com/awslabs/aws-lambda-powertools-python/issues/2097))
+* **deps-dev:** bump aws-cdk from 2.72.1 to 2.73.0 ([#2093](https://github.com/awslabs/aws-lambda-powertools-python/issues/2093))
+* **deps-dev:** bump mypy-boto3-cloudformation from 1.26.60 to 1.26.108 ([#2095](https://github.com/awslabs/aws-lambda-powertools-python/issues/2095))
 * **deps-dev:** bump types-python-dateutil from 2.8.19.11 to 2.8.19.12 ([#2085](https://github.com/awslabs/aws-lambda-powertools-python/issues/2085))
+* **deps-dev:** bump cfn-lint from 0.76.1 to 0.76.2 ([#2084](https://github.com/awslabs/aws-lambda-powertools-python/issues/2084))
 * **deps-dev:** bump aws-cdk from 2.72.0 to 2.72.1 ([#2081](https://github.com/awslabs/aws-lambda-powertools-python/issues/2081))
+* **deps-dev:** bump coverage from 7.2.2 to 7.2.3 ([#2092](https://github.com/awslabs/aws-lambda-powertools-python/issues/2092))
 * **deps-dev:** bump mkdocs-material from 9.1.4 to 9.1.5 ([#2077](https://github.com/awslabs/aws-lambda-powertools-python/issues/2077))
 * **deps-dev:** bump aws-cdk-lib from 2.72.0 to 2.72.1 ([#2076](https://github.com/awslabs/aws-lambda-powertools-python/issues/2076))
 * **deps-dev:** bump mypy-boto3-s3 from 1.26.99 to 1.26.104 ([#2075](https://github.com/awslabs/aws-lambda-powertools-python/issues/2075))
 * **deps-dev:** bump aws-cdk from 2.71.0 to 2.72.0 ([#2071](https://github.com/awslabs/aws-lambda-powertools-python/issues/2071))
-* **deps-dev:** bump cfn-lint from 0.76.1 to 0.76.2 ([#2084](https://github.com/awslabs/aws-lambda-powertools-python/issues/2084))
+* **deps-dev:** bump filelock from 3.10.7 to 3.11.0 ([#2094](https://github.com/awslabs/aws-lambda-powertools-python/issues/2094))
 * **deps-dev:** bump aws-cdk-lib from 2.71.0 to 2.72.0 ([#2070](https://github.com/awslabs/aws-lambda-powertools-python/issues/2070))
 * **deps-dev:** bump black from 23.1.0 to 23.3.0 ([#2066](https://github.com/awslabs/aws-lambda-powertools-python/issues/2066))
 * **deps-dev:** bump aws-cdk from 2.70.0 to 2.71.0 ([#2067](https://github.com/awslabs/aws-lambda-powertools-python/issues/2067))

aws_lambda_powertools/utilities/batch/base.py

@@ -26,6 +26,7 @@
     KinesisStreamRecord,
 )
 from aws_lambda_powertools.utilities.data_classes.sqs_event import SQSRecord
+from aws_lambda_powertools.utilities.parser import ValidationError
 from aws_lambda_powertools.utilities.typing import LambdaContext
 
 logger = logging.getLogger(__name__)
@@ -305,21 +306,36 @@ def _get_messages_to_report(self) -> List[Dict[str, str]]:
     def _collect_sqs_failures(self):
         failures = []
         for msg in self.fail_messages:
-            msg_id = msg.messageId if self.model else msg.message_id
+            # If a message failed due to model validation (e.g., poison pill)
+            # we convert to an event source data class...but self.model is still true
+            # therefore, we do an additional check on whether the failed message is still a model
+            # see https://github.com/awslabs/aws-lambda-powertools-python/issues/2091
+            if self.model and getattr(msg, "parse_obj", None):
+                msg_id = msg.messageId
+            else:
+                msg_id = msg.message_id
             failures.append({"itemIdentifier": msg_id})
         return failures
 
     def _collect_kinesis_failures(self):
         failures = []
         for msg in self.fail_messages:
-            msg_id = msg.kinesis.sequenceNumber if self.model else msg.kinesis.sequence_number
+            # # see https://github.com/awslabs/aws-lambda-powertools-python/issues/2091
+            if self.model and getattr(msg, "parse_obj", None):
+                msg_id = msg.kinesis.sequenceNumber
+            else:
+                msg_id = msg.kinesis.sequence_number
             failures.append({"itemIdentifier": msg_id})
         return failures
 
     def _collect_dynamodb_failures(self):
         failures = []
         for msg in self.fail_messages:
-            msg_id = msg.dynamodb.SequenceNumber if self.model else msg.dynamodb.sequence_number
+            # see https://github.com/awslabs/aws-lambda-powertools-python/issues/2091
+            if self.model and getattr(msg, "parse_obj", None):
+                msg_id = msg.dynamodb.SequenceNumber
+            else:
+                msg_id = msg.dynamodb.sequence_number
             failures.append({"itemIdentifier": msg_id})
         return failures
 
@@ -336,6 +352,17 @@ def _to_batch_type(self, record: dict, event_type: EventType, model: Optional["B
             return model.parse_obj(record)
         return self._DATA_CLASS_MAPPING[event_type](record)
 
+    def _register_model_validation_error_record(self, record: dict):
+        """Convert and register failure due to poison pills where model failed validation early"""
+        # Parser will fail validation if record is a poison pill (malformed input)
+        # this means we can't collect the message id if we try transforming again
+        # so we convert into to the equivalent batch type model (e.g., SQS, Kinesis, DynamoDB Stream)
+        # and downstream we can correctly collect the correct message id identifier and make the failed record available
+        # see https://github.com/awslabs/aws-lambda-powertools-python/issues/2091
+        logger.debug("Record cannot be converted to customer's model; converting without model")
+        failed_record: "EventSourceDataClassTypes" = self._to_batch_type(record=record, event_type=self.event_type)
+        return self.failure_handler(record=failed_record, exception=sys.exc_info())
+
 
 class BatchProcessor(BasePartialBatchProcessor):  # Keep old name for compatibility
     """Process native partial responses from SQS, Kinesis Data Streams, and DynamoDB.
@@ -460,14 +487,17 @@ def _process_record(self, record: dict) -> Union[SuccessResponse, FailureRespons
         record: dict
             A batch record to be processed.
         """
-        data = self._to_batch_type(record=record, event_type=self.event_type, model=self.model)
+        data: Optional["BatchTypeModels"] = None
         try:
+            data = self._to_batch_type(record=record, event_type=self.event_type, model=self.model)
             if self._handler_accepts_lambda_context:
                 result = self.handler(record=data, lambda_context=self.lambda_context)
             else:
                 result = self.handler(record=data)
 
             return self.success_handler(record=record, result=result)
+        except ValidationError:
+            return self._register_model_validation_error_record(record)
         except Exception:
             return self.failure_handler(record=data, exception=sys.exc_info())
 
@@ -595,13 +625,16 @@ async def _async_process_record(self, record: dict) -> Union[SuccessResponse, Fa
         record: dict
             A batch record to be processed.
         """
-        data = self._to_batch_type(record=record, event_type=self.event_type, model=self.model)
+        data: Optional["BatchTypeModels"] = None
         try:
+            data = self._to_batch_type(record=record, event_type=self.event_type, model=self.model)
             if self._handler_accepts_lambda_context:
                 result = await self.handler(record=data, lambda_context=self.lambda_context)
             else:
                 result = await self.handler(record=data)
 
             return self.success_handler(record=record, result=result)
+        except ValidationError:
+            return self._register_model_validation_error_record(record)
         except Exception:
             return self.failure_handler(record=data, exception=sys.exc_info())

aws_lambda_powertools/utilities/parser/models/dynamodb.py

@@ -9,8 +9,8 @@
 class DynamoDBStreamChangedRecordModel(BaseModel):
     ApproximateCreationDateTime: Optional[date]
     Keys: Dict[str, Dict[str, Any]]
-    NewImage: Optional[Union[Dict[str, Any], Type[BaseModel]]]
-    OldImage: Optional[Union[Dict[str, Any], Type[BaseModel]]]
+    NewImage: Optional[Union[Dict[str, Any], Type[BaseModel], BaseModel]]
+    OldImage: Optional[Union[Dict[str, Any], Type[BaseModel], BaseModel]]
     SequenceNumber: str
     SizeBytes: int
     StreamViewType: Literal["NEW_AND_OLD_IMAGES", "KEYS_ONLY", "NEW_IMAGE", "OLD_IMAGE"]

aws_lambda_powertools/utilities/parser/models/kinesis.py

@@ -15,7 +15,7 @@ class KinesisDataStreamRecordPayload(BaseModel):
     kinesisSchemaVersion: str
     partitionKey: str
     sequenceNumber: str
-    data: Union[bytes, Type[BaseModel]]  # base64 encoded str is parsed into bytes
+    data: Union[bytes, Type[BaseModel], BaseModel]  # base64 encoded str is parsed into bytes
     approximateArrivalTimestamp: float
 
     @validator("data", pre=True, allow_reuse=True)

aws_lambda_powertools/utilities/parser/models/sqs.py

@@ -52,7 +52,7 @@ class SqsMsgAttributeModel(BaseModel):
 class SqsRecordModel(BaseModel):
     messageId: str
     receiptHandle: str
-    body: Union[str, Type[BaseModel]]
+    body: Union[str, Type[BaseModel], BaseModel]
     attributes: SqsAttributesModel
     messageAttributes: Dict[str, SqsMsgAttributeModel]
     md5OfBody: str

aws_lambda_powertools/utilities/parser/types.py

@@ -3,16 +3,18 @@
 import sys
 from typing import Any, Dict, Type, TypeVar, Union
 
-from pydantic import BaseModel
+from pydantic import BaseModel, Json
 
 # We only need typing_extensions for python versions <3.8
 if sys.version_info >= (3, 8):
-    from typing import Literal  # noqa: F401
+    from typing import Literal
 else:
-    from typing_extensions import Literal  # noqa: F401
+    from typing_extensions import Literal
 
 Model = TypeVar("Model", bound=BaseModel)
 EnvelopeModel = TypeVar("EnvelopeModel")
 EventParserReturnType = TypeVar("EventParserReturnType")
 AnyInheritedModel = Union[Type[BaseModel], BaseModel]
 RawDictOrModel = Union[Dict[str, Any], AnyInheritedModel]
+
+__all__ = ["Json", "Literal"]

docs/overrides/main.html

@@ -1,10 +1,5 @@
 {% extends "base.html" %}
 
-{% block announce %}
-👋 Powertools for Python v1 will no longer receive updates or releases after 31/03/2023!
-We encourage you to read our <a href="{{ base_url }}/upgrade/#end-of-support-v1">upgrade guide on how to migrate to v2</a>.
-{% endblock %}
-
 {% block outdated %}
   You're not viewing the latest version.
   <a href="{{ '../' ~ base_url }}">

docs/upgrade.md

@@ -7,7 +7,7 @@ description: Guide to update between major Powertools versions
 
 ## End of support v1
 
-On March 31st, AWS Lambda Powertools for Python v1 will reach end of support. After that, Powertools v1 will no longer receive updates or releases. If you are still using v1, we encourage you to read our upgrade guide and update to the latest version.
+!!! warning "On March 31st, 2023, AWS Lambda Powertools for Python v1 reached end of support and will no longer receive updates or releases. If you are still using v1, we strongly recommend you to read our upgrade guide and update to the latest version."
 
 Given our commitment to all of our customers using AWS Lambda Powertools for Python, we will keep [Pypi](https://pypi.org/project/aws-lambda-powertools/) v1 releases and documentation 1.x versions to prevent any disruption.
 

package-lock.json

@@ -2,6 +2,6 @@
   "name": "aws-lambda-powertools-python-e2e",
   "version": "1.0.0",
   "devDependencies": {
-	"aws-cdk": "^2.72.1"
+	"aws-cdk": "^2.73.0"
   }
 }