Bug: OpenAPI schema invalid when using Pydantic v2 (latest)

[kbakk]

Expected Behaviour

Using ApiGatewayResolver.get_openapi_json_schema should produce a valid OpenAPI schema

Current Behaviour

When using Pydantic version 2.6.1 (latest at the time of writing) the OpenAPI spec produced is invalid. This is not the case when using Pydantic version 1.10.14.

Errors seen in schema:

...
  "info": {
    "title": "TODO's API",
    "summary": "API to manage TODOs",
    "version": "1.0.0"
  },
...

Structural error at info
should NOT have additional properties
additionalProperty: summary

Note: This error is present when using both v1 and v2 of Pydantic.

...
      "Todo": {
        "properties": {
          "type": {
            "const": "ingest",
            "title": "Type"
          },
...

Structural error at components.schemas.Todo.properties.type
should NOT have additional properties
additionalProperty: const

...
          "id": {
            "anyOf": [
              {
                "type": "integer"
              },
              {
                "type": "null"
              }
            ],
...

Structural error at components.schemas.TodoAttributes.properties.id.anyOf.1.type
should be equal to one of the allowed values
allowedValues: array, boolean, integer, number, object, string

Code snippet

Python code: handler.py

from typing import Literal

from aws_lambda_powertools.event_handler import APIGatewayRestResolver
from aws_lambda_powertools.event_handler.openapi.models import Contact, Server
from aws_lambda_powertools.event_handler.openapi.types import OpenAPIResponse
from pydantic import BaseModel, Field

app = APIGatewayRestResolver(enable_validation=True)
app.enable_swagger(path="/swagger", title="TODO")


class TodoAttributes(BaseModel):
    userId: int
    id_: int | None = Field(alias="id", default=None)
    title: str
    completed: bool


class Todo(BaseModel):
    type: Literal["ingest"]
    attributes: TodoAttributes


class TodoEnvelope(BaseModel):
    data: Todo


@app.get("/todos/<todo_id>", tags=["todo"])
def get_todo_by_id(todo_id: int) -> TodoEnvelope: ...


@app.post(
    "/todos",
    tags=["todo"],
    responses={
        204: OpenAPIResponse(
            description="Successful creation",
            content={"": {"schema": {}}},
        )
    },
)
def create_todo(todo: TodoEnvelope): ...


if __name__ == "__main__":
    print(
        app.get_openapi_json_schema(
            title="TODO's API",
            summary="API to manage TODOs",
        ),
    )

Possible Solution

No response

Steps to Reproduce

Run the provided script: python handler.py > swagger.json

Then paste the JSON data in https://editor.swagger.io/ to see the errors (there are also tools like spectral which can be used to validate the output).

Powertools for AWS Lambda (Python) version

latest

AWS Lambda function runtime

3.12

Packaging format used

PyPi

Debugging logs

No response

[kbakk]

According to Pydantic docs: JSON Schema, Pydantic is compaible with OpenAPI 3.1.0. Running get_openapi_json_schema(openapi_version="3.1.0") will then produce a valid schema.

I will leave this issue open, since I believe this could be addressed in an automatic fashion: if Pydantic <2.0 use 3.0.0, else use 3.1.0. Also the summary should only be added if using OpenAPI version >=3.1.0.

[sthulb]

Hi @kbakk, thanks for filing the issue – we'll look into it to see if this is something we should add!

[heitorlessa]

cc @rubenfonseca who's oncall this week and might have missed.

[rubenfonseca]

Looking at this now

[rubenfonseca]

@kabkk Thank you for the very detailed explanation! I need to run some extra checks on this, but the major problem I have right now is that it's very hard to support multiple versions of OpenAPI spec with the same API. And on the other hand I'm not 100% comfortable in bumping the default to 3.1.0. Do you have any feedback on what would be the best for you?

[kbakk]

it's very hard to support multiple versions of OpenAPI spec with the same API.

That was not my intention. The point is that the version of Pydantic you have installed (1.x or 2.x) will determine which API spec will be produced.

In case Pydantic 1.x is installed, the default OpenAPI spec produced with get_openapi_json_schema() will be valid (since it's defaulting to 3.0.0). But if Pydantic 2.x is installed, the user should (now) specify version 3.1.0: get_openapi_json_schema(openapi_version="3.1.0").

I was thinking that get_openapi_json_schema could determine to use version 3.1.0 in case Pydantic 2.x is installed, so that get_openapi_json_schema() produces a valid output in both cases. I would consider this a bugfix rather than a breaking change (bug being that an invalid OpenAPI spec is produced).

[heitorlessa]

thanks a lot for the feedback @kbakk :) We'll sync internally for a way out on this -- Ruben briefly mentioned it'd impact the upcoming Agent for Amazon Bedrock Event Handler feature at a first glance.

Luckily all of this will go away on v3 too when we drop Pydantic v1. We'll come back to you as soon as we have a meaningful update to share - appreciate your patience!!

[rubenfonseca]

Hi all, apologies for the late update. Here are my findings:

• @kbakk is obviously correct that Pydantic v1 and v2 generates two different types of JSON Schemas (v1 compatible with OpenAPI v3.0.x, and v2 compatible with OpenAPI v3.1.x)
• There's no internal Pydantic flag to change that behavior. Pydantic v1 will always be OpenAPI 3.0, Pydantic v2 will always be OpenAPI 3.1.
• We actually have control over the majority of the generated OpenAPI schemas (so we could fix the summary part above). However, the problem will always be when we ask Pydantic to create a JSON schema from a Pydantic model (which will generate two different things depending on v1 / v2)

In my opinion we need to:

• Document that different versions of pydantic generate different types of OpenAPI schemas
• Follow @kbakk's advice and automatically set the correct version depending on the Pydantic version used (right now it always says 3.0.0, which is wrong)
• Add a big fat warning in docs (and a warning in code) when using BedrockAgentResolver + Pydantic v2: the generated schema will not be supported by Bedrock's API at this moment

What do you think of this?

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

This issue is now closed. Please be mindful that future comments are hard for our team to see.

If you need more assistance, please either tag a team member or open a new issue that references this one.

If you wish to keep having a conversation with other community members under this issue feel free to do so.

[github-actions]

This is now released under 2.35.0 version!