Bug: API Gateway Console Test button fails validation with @event_parser because requestContext -> identity -> sourceIp value is not a valid IPv4 or IPv6 network

[mikelane]

Expected Behaviour

We should be able to generate a test form the API Gateway console that allows the ApiGatewayProxyEventModel to be validated and parsed by Pydantic.

Current Behaviour

When generating a test from the API Gateway console, the value for the sourceIp in requestContext -> identity is "test-invoke-source-ip". Since this value is defined to be IPvAnyNetwork here, the validation fails in the event_parser decorator here. And so test events from the API Gateway console fail with a 502 error because of the resulting pydantic ValidationError.

Code snippet

Create a lambda handler with an event parser like this...

from aws_lambda_powertools.utilities.parser import event_parser
from aws_lambda_powertools.utilities.parser.models import APIGatewayProxyEventModel
from aws_lambda_powertools.utilities.typing import LambdaContext

@logger.inject_lambda_context(log_event=settings.debug)
@event_parser(model=APIGatewayProxyEventModel)
def handler(event: APIGatewayProxyEvent, context: LambdaContext) -> dict[str, Any]:
    logger.info('event: %s', event.json())
    return {'statusCode': 200, 'body': json.dumps({'wow': 42})}

Use the following message to the lambda:

{
    "level": "INFO",
    "location": "decorate:352",
    "message": {
        "resource": "/v1/redacted/12345",
        "path": "/v1/redacted/12345",
        "httpMethod": "GET",
        "headers": {
            "Accept": "application/json"
        },
        "multiValueHeaders": {
            "Accept": [
                "application/json"
            ]
        },
        "queryStringParameters": null,
        "multiValueQueryStringParameters": null,
        "pathParameters": {
            "redacted_id": "12345"
        },
        "stageVariables": null,
        "requestContext": {
            "resourceId": "redacted",
            "resourcePath": "/v1/redacted/{redacted_id}",
            "httpMethod": "GET",
            "extendedRequestId": "Redacted=",
            "requestTime": "30/Sep/2022:19:19:03 +0000",
            "path": "/v1/redacted/{redacted_id}",
            "accountId": "012345678912",
            "protocol": "HTTP/1.1",
            "stage": "test-invoke-stage",
            "domainPrefix": "testPrefix",
            "requestTimeEpoch": 1664565543376,
            "requestId": "9c4368e8-8510-42f1-875b-ecb63f3f5c5a",
            "identity": {
                "cognitoIdentityPoolId": null,
                "cognitoIdentityId": null,
                "apiKey": "test-invoke-api-key",
                "principalOrgId": null,
                "cognitoAuthenticationType": null,
                "userArn": "arn:aws:sts::012345678912:assumed-role/redacted/joe.bagadonuts",
                "apiKeyId": "test-invoke-api-key-id",
                "userAgent": "aws-internal/3 aws-sdk-java/1.12.302 Linux/5.4.207-126.363.amzn2int.x86_64 OpenJDK_64-Bit_Server_VM/25.342-b07 java/1.8.0_342 vendor/Oracle_Corporation cfg/retry-mode/standard",
                "accountId": "012345678912",
                "caller": "REDACTED:joe.bagadonuts",
                "sourceIp": "test-invoke-source-ip",
                "accessKey": "REDACTED",
                "cognitoAuthenticationProvider": null,
                "user": "REDACTED:joe.bagadonuts"
            },
            "domainName": "testPrefix.testDomainName",
            "apiId": "1234ASFD"
        },
        "body": null,
        "isBase64Encoded": false
    },
    "timestamp": "2022-09-30 19:19:03,390+0000",
    "service": "redacted-redacto",
    "cold_start": false,
    "function_name": "redacted",
    "function_memory_size": "128",
    "function_arn": "arn:aws:lambda:us-east-1:012345678912:function:redacted",
    "function_request_id": "bcf14ad4-c46f-471c-8b9f-60e56932c9e6",
    "xray_trace_id": "9-12345678-1234asdf2345qwer"
}

Note the error that the sourceIp is not an IPv4 or IPv6 value.

Possible Solution

Update the sourceIp here from this:

    sourceIp: IPvAnyNetwork

to this:

    sourceIp: Union[IPvAnyNetwork, Literal['test-invoke-source-ip']]

You may want to add a test (maybe in this file?) that tests for the 'test-invoke-source-ip' value explicitly.

Note this may also be an issue with the APIGatewayProxyEventV2Model because of the sourceIp definition here.

Steps to Reproduce

Create the lambda function provided above. Deploy the lambda function provided above behind an api gateway. Use the api gateway console to test the function passing in the redacted_id of 12345, and Accept:application/json for the headers.

AWS Lambda Powertools for Python version

latest

AWS Lambda function runtime

3.9

Packaging format used

PyPi

Debugging logs

No response

[boring-cyborg]

Thanks for opening your first issue here! We'll come back to you as soon as we can.
In the meantime, check out the #python channel on our AWS Lambda Powertools Discord: Invite link

[heitorlessa]

Thank you @mikelane for flagging this issue in the API Gateway service.

Would you be able to create a support case to suggest API Gateway team to use an actual IP Address in the test event?

We'll cut an internal ticket to them next week.

I'm removing the bug label as Parser is behaving as expected. That said, we should fix this at the source, otherwise this will happen again next time this changes to another literal we have no control over.

If helpful, we keep track of real events for each event source we support here: https://github.com/awslabs/aws-lambda-powertools-python/blob/develop/tests/events/

Have a great weekend!

[heitorlessa]

Quick update: We pinged the API Gateway team asking to fix the event in the Console; we'll revert once we hear back from them.

[mikelane]

I heard back from the API Gateway support team. They said this:

"test-invoke-source-ip" is the expected result when using the API-GW test feature and unfortunately this isn't something which can be changed.
My recommendation would be to add a line of code in your Lambda as the event comes in, and then check for "test-invoke-source-ip" and then change it to a valid source IP before the validation is done.

Someone needs to budge here 😅

[heitorlessa]

Leave with us @mikelane. We reached out to the engineering team and will revert with an actionable plan from either side to get this resolved.

This can be changed on API Gateway side. Question here is whether they have the bandwidth to do it.

Worst case we disable all ip validation if that can't be honoured by the product team.

Early to tell but issues like these make me reconsider our thoughts of merging Parser/Pydantic and Event Source Data Classes - I like the value Pydantic brings, but it puts us as the weakest link in contract situations like this

[mikelane]

The fix I've suggested will work (it does work, this is what I've done to work around the issue) and would, I think, be a better solution than disabling all ip validation.

[heitorlessa]

I hear you. I truly do. My concern here is that we're dealing with data validation - it works with an actual event. Because there is no official event source schema, there is no guarantee that tomorrow test-invoke-source-ip will turn into test.

I'll wait to hear back from the API Gateway engineering team.

[heitorlessa]

Update. I have good news to share @mikelane. API Gateway Engineering team will fix the source by the end of the month - no more test-invoke-source-ip value.

We'll update here when it's done and confirmed.

[mikelane]

Wow that's awesome! Thank you.

[heitorlessa]

Update: no update from the team on completion yet, since re:Invent is taking most of everyone's bandwidth. I'll keep you posted once that changes.

[mikelane]

Thanks for the update!

[heitorlessa]

Update: no update from the team on completion yet, since re:Invent is taking most of everyone's bandwidth. I'll keep you posted once that changes.

Same as above ^. Sent a message to the team asking whether we can have an ETA. Thank you for your patience.

[boivinmjc]

Update: It was revealed that this fix is more impactful than initially estimated. While I still do believe that changing APIGW is the ideal route, I don't have a clear ETA at this point. I would consider altering the validation model as suggested by @mikelane as a temporary solution until APIGW testInvoke can be fixed. I'll update this thread with an ETA when possible.

[heitorlessa]

@mikelane given the elapsed time, are you still looking to use this feature in API Gateway (test button)? If you are, then happy to have a PR for this temporary fix until the permanent one is done in API Gateway, as @boivinmjc from the API Gateway team suggested.

If you aren't planning to use it anymore, then I'll close it and put a reminder every quarter to check with the API Gateway team for closure.

Thank you

[mikelane]

I am still looking for this feature in API-G, yes please.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

This issue is now closed. Please be mindful that future comments are hard for our team to see.

If you need more assistance, please either tag a team member or open a new issue that references this one.

If you wish to keep having a conversation with other community members under this issue feel free to do so.

[github-actions]

This is now released under 2.9.1 version!