parse function raises InvalidModelTypeError instead of ValidationError

[jhare-rectangle]

Expected Behaviour

When any validation in a Pydantic model fails during a call to Powertools parse() function, I expect parse() to raise ValidationError. This is what happened prior to updating to version 3 of Powertools and is reflected in this example from Powertools documentation: https://docs.powertools.aws.dev/lambda/python/latest/utilities/parser/#parse-function

Current Behaviour

When the passed in object fails model validation in parse(), it is raising InvalidModelTypeError.

Code snippet

# Again, taken from the docs
from aws_lambda_powertools.utilities.parser import parse, BaseModel, ValidationError
from typing import List, Optional

class OrderItem(BaseModel):
    id: int
    quantity: int
    description: str

class Order(BaseModel):
    id: int
    description: str
    items: List[OrderItem] # nesting models are supported
    optional_field: Optional[str] = None # this field may or may not be available when parsing


payload = {
    "id": 10876546789,
    "description": "My order",
    "items": [
        {
            # this will cause a validation error
            "id": [1015938732],
            "quantity": 1,
            "description": "item xpto"
        }
    ]
}

def my_function():
    try:
        parsed_payload: Order = parse(event=payload, model=Order)
        # payload dict is now parsed into our model
        return parsed_payload.items
    except ValidationError:
        return {
            "status_code": 400,
            "message": "Invalid order"
        }

Possible Solution

No response

Steps to Reproduce

I am just running this locally at the moment. I don't know if the Lambda Layer might somehow behavior differently since.

Powertools for AWS Lambda (Python) version

3.0.2.dev1

AWS Lambda function runtime

3.12

Packaging format used

PyPi

Debugging logs

/home/john/.pyenv/versions/sftp-processor-3.12/lib/python3.12/site-packages/aws_lambda_powertools/package_logger.py:20: UserWarning: POWERTOOLS_DEBUG environment variable is enabled. Setting logging level to DEBUG.
  if powertools_debug_is_set():
2024-09-26 16:07:09,201 aws_lambda_powertools.utilities.parser.parser [DEBUG] Parsing and validating event model; no envelope used
Traceback (most recent call last):
  File "/home/john/.pyenv/versions/sftp-processor-3.12/lib/python3.12/site-packages/aws_lambda_powertools/utilities/parser/parser.py", line 195, in parse
    return adapter.validate_python(event)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/john/.pyenv/versions/sftp-processor-3.12/lib/python3.12/site-packages/pydantic/type_adapter.py", line 135, in wrapped
    return func(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/john/.pyenv/versions/sftp-processor-3.12/lib/python3.12/site-packages/pydantic/type_adapter.py", line 366, in validate_python
    return self.validator.validate_python(object, strict=strict, from_attributes=from_attributes, context=context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
pydantic_core._pydantic_core.ValidationError: 1 validation error for Order
items.0.id
  Input should be a valid integer [type=int_type, input_value=[1015938732], input_type=list]
    For further information visit https://errors.pydantic.dev/2.9/v/int_type

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/john/sandbox/sftp-processor/sftp-request-processor/sftp_request_model.py", line 260, in <module>
    parsed_payload: Order = parse(event=payload, model=Order)
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/john/.pyenv/versions/sftp-processor-3.12/lib/python3.12/site-packages/aws_lambda_powertools/utilities/parser/parser.py", line 203, in parse
    raise InvalidModelTypeError(
aws_lambda_powertools.utilities.parser.exceptions.InvalidModelTypeError: Error: 1 validation error for Order
items.0.id
  Input should be a valid integer [type=int_type, input_value=[1015938732], input_type=list]
    For further information visit https://errors.pydantic.dev/2.9/v/int_type. Please ensure the Input model inherits from BaseModel,
and your payload adheres to the specified Input model structure.
Model=<class '__main__.Order'>

[boring-cyborg]

Thanks for opening your first issue here! We'll come back to you as soon as we can.
In the meantime, check out the #python channel on our Powertools for AWS Lambda Discord: Invite link

[leandrodamascena]

Hey @jhare-rectangle! Thanks for reporting this bug! Actually I think you're right. We should not catch ValidationError and re-raise it as InvalidModelTypeError because it's a data validation error. We made this change in V3, but we may have missed that customers are using ValidationError to catch exceptions when data is wrong, what make completely sense to me.

I am just running this locally at the moment. I don't know if the Lambda Layer might somehow behavior differently since.

No, there is no diference, you'll experiment the same problem when using Layer.

Give me until tomorrow to talk to the other maintainers, but I think we'll need to revert this change and continue raising ValidationError instead of InvalidModelTypeError.

Again, thank you very much for reporting this to allow us to fix this in the early days of v3.

[leandrodamascena]

Hey @jhare-rectangle! We expect to merge the PR #5259 today and revert this regression.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

This issue is now closed. Please be mindful that future comments are hard for our team to see.

If you need more assistance, please either tag a team member or open a new issue that references this one.

If you wish to keep having a conversation with other community members under this issue feel free to do so.

[github-actions]

This is now released under 3.1.0 version!