RFC: Auto-generate OpenAPI docs from routes

[rubenfonseca]

Is this related to an existing feature request or issue?

#1236

Which Powertools for AWS Lambda (Python) utility does this relate to?

Event Handler - REST API

Summary

This RFC proposes the implementation of an automatic OpenAPI documentation generation feature for the AWS Lambda Powertools for Python's Event Handler - REST API utility.

The goal is to save developers time and effort by automating the generation of OpenAPI documentation based on the defined routes. The proposal suggests integrating this functionality into the existing utility, allowing developers to easily incorporate it into their applications.

Additionally, an optional Swagger UI endpoint would be exposed to provide a user-friendly interface for accessing the API documentation.

The scope of this proposal focuses solely on the auto-generation of OpenAPI docs from routes, excluding middleware support and authorization aspects. Some potential challenges discussed include handling dependencies, performance considerations, and deciding the level of exposure of the underlying apispec library.

Use case

The proposal aims to address the need for automatic generation of OpenAPI documentation based on the routes defined in the AWS Lambda Powertools for Python's Event Handler - REST API utility.

By automating the generation of OpenAPI docs, developers can save time and effort in maintaining the documentation manually. It ensures that the docs stay up to date with the application's routes, making the process efficient and hassle-free.

This automation optimizes developers' productivity and enhances the overall quality of the documentation, serving as a reliable reference for internal teams and external stakeholders.

With this integration of automation, developers can focus more on innovation and coding, knowing that their OpenAPI docs are effortlessly maintained and reflect the current state of the application.

Task list

• Merge PR feat(event_handler): generate OpenAPI specifications and validate input/output #3109
• Create documentation and examples
• Create a mechanism to expose a Swagger endpoint
• Create a CLI / utility to export an OpenAPI spec from an existing event handler API
• Create a more advanced CLI / utility to merge different APIs into a single Swagger specification

Proposal (updated 11/07/2023)

The design proposes integrating a functionality within the AWS Lambda Powertools for Python's Event Handler - REST API utility that auto-generates OpenAPI documentation based on the defined routes. The implementation should be intuitive and accessible for developers familiar with the project, allowing them to easily incorporate this feature into their applications.

We took inspiration from FastAPI and their excelent first class support for generationg OpenAPI specs. The implementation will need to introspect named parameters to add it to the documentation.

Additionally, we want to optionally expose a Swagger UI endpoint with the API documentation.

import json

from aws_lambda_powertools.event_handler import ApiGatewayResolver
from aws_lambda_powertools.event_handler.openapi import (
    Example,
    MediaType,
    OpenAPIInfo,
    Operation,
    Parameter,
    ParameterLocation,
    Response,
    Responses,
)

app = ApiGatewayResolver(generate_openapi=True, openapi_info=OpenAPIInfo(
    title="This is my API",
    version="0.0.1b",
))


@app.get(rule="/viewer",
         operation=Operation(
             operationId="geViewerDetailsv1",
             summary="Viewer user information",
             description="Returns information about the current signed-in user",
             tags=["users"],
             parameters=[
                 Parameter(name="include_details",
                           location=ParameterLocation.QUERY,
                           required=False,
                           schema={
                               "type": "boolean",
                           }),
             ],
             responses=Responses(codes={
                 200: Response(description="Success",
                               content={
                                   "application/json": MediaType(
                                       schema={"$ref": "#/components/schemas/User"},
                                       examples={
                                           "application/json": Example(
                                               value=json.dumps({"id": 1, "name": "John Smith"}),
                                           ),
                                       },
                                   ),
                               }),
             }),
         ))
def viewer(event: dict):
    print(event)


print(app._spec.to_yaml())

Results in

paths:
  /viewer:
    get:
      tags:
      - users
      summary: Viewer user information
      description: Returns information about the current signed-in user
      operationId: geViewerDetailsv1
      parameters:
      - name: include_details
        in: query
        required: false
        schema:
          type: boolean
      responses:
        '200':
          description: Success
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/User'
              examples:
                application/json:
                  value: '{"id": 1, "name": "John Smith"}'
    summary: Viewer user information
info:
  title: This is my API
  version: 0.0.1b
openapi: 3.0.2

TODO:

• Explore way of defining global components (on the example, the User schema isn't defined in the spec)
• Make sure that the schema mechanism can be easily extended in the future (e.g: Pydantic models)
• Explore auto generation of parameters based on path parameters
• Create an example of serving the Swagger API

Enriching the OpenAPI data

OpenAPI is a rich spec that allows a lot of detail to be added. Currently, there's only a very limited amount of information we can derive automatically:

• HTTP method
• path
• parameters
• eventually CORS configuration

In order to allow the user to extend the specification with additional metadata, we can consider different options:

Expose the apispec instance on the resolver's decorator (abandoned)

app = ApiGatewayResolver(generate_openapi=True)

app.spec.title = "Swagger Petstore"
app.spec.version = "1.0.0"
app.spec.path("/hello", operations={...})

@app.get("/hello")
def hello():
    print("world")

Create a DTO for the OpenAPI properties (accepted)

app = ApiGatewayResolver(generate_openapi=True, openapi_properties={
  "title": "Swagger Petstore",
  "version": "1.0.0",
  "openapi_version": "3.0.2"
})

@app.get("/hello", spec={
    "operation": {
        "summary": "This is a hello world endpoint",
        ...
    }
})

In this case we could of course improve the DX by creating utility classes instead of making the user have to define arbitrary dictionaries.

Extend through documentation (abandoned for now)

This would follow the apispec-frameworks model, where the documentation for the handler would get parsed and data would be extracted from it. Example from Flask:

class GistApi(MethodView):
        '''Gist API.
        ---
        x-extension: metadata
        '''
        def get(self):
           '''Gist view
           ---
           responses:
               200:
                   schema:
                       $ref: '#/definitions/Gist'
           '''
           pass

        def post(self):
           pass

Serving a Swagger UI

We could add a new parameter to the resolver enabling a new GET /api-docs route that would serve a Swagger UI.

app = ApiGatewayResolver(generate_openapi=True, enable_swagger=True, swagger_path="/api-docs")

This would still require configuring the mapping in the API Gateway (or equivalent).

To implement this, check how chalice-spec implements it, by loading the swagger ui from their CDN, and passing the API spec into the page as a JSON file.

Out of scope

The scope of this proposal does not include adding middleware support, such as incorporating authorization into the Swagger UI. While authorization is an important aspect of API documentation, it falls outside the scope of this specific RFC. The focus is solely on the auto-generation of OpenAPI docs from routes.

Additionally, we won't support any type of model validation for this RFC. This means we won't be able to attach Pydantic models to the API and have Powertools inspect and generate the appropriate OpenAPI sections for it.

Potential challenges

• Include apispec as a dependency or an optional dependency: the current version of apispec weighs around 29.6Kb. Should we just import the dependency, or mark it as optinal?
• Performance problems: should we opt-out from generating the API spec during a production workload? How to identify that we are running in production mode?
• How much of apispec to expose: should we just "escape-hatch" the apispec and let the customer use its API to enrich the API spec, or should we create a thin wrapper around it, allowing us to switch implementations in the future?

Acknowledgment

• This feature request meets Powertools for AWS Lambda (Python) Tenets
• Should this be considered in other Powertools for AWS Lambda languages? i.e. Java, TypeScript, and .NET

[rubenfonseca]

Added a section about Swagger UI

[pharindoko]

Hey @rubenfonseca

Why model schemas are out of scope of this rfc ?
Is the reason to reduce complexity in the first step ?

[ran-isenberg]

That's a great idea and would be so powerful.
I do feel that once that's ready, we should also add a non event handler related generator. + add pydantic model support maybe it will be easier to implement?

[ran-isenberg]

You can use an environment variable to generate/or not the api spec. Perhaps even supply a cli command to generate it and during runtime the decorator/open api code will not affect production code.

[darnley]

That's cool! SwaggerUI facilitates the debugging. We use something like that in FastAPI with Mangum and REALLY facilitates development when Lambda is behind a API Gateway through proxy.

Maybe we can use POWERTOOLS_DEV or debug argument (like debug mode) to expose the SwaggerUI when developer did not set enable_swagger argument in ApiGatewayResolver?

[pharindoko]

@darnley we use the same approach to be able to debug and work locally.

[pharindoko]

You can use an environment variable to generate/or not the api spec. Perhaps even supply a cli command to generate it and during runtime the decorator/open api code will not affect production code.

I would prefer to autogenerate it at runtime on the fly and keep the same spec for the whole lifetime of the lambda. At least as a first option.

• Makes it easy to locally start and debug it
• You don't have to pre-generate the spec manually.
• A flag to enable spec generation makes sense

[leandrodamascena]

Hi @rubenfonseca! Impressive RFC, dude! Clear purpose and valuable addition to Powertools.

Following my considerations.

1 - Extending the specification with additional metadata
In my opinion, the second option (Create a thin-wrapper around apispec) makes more sense because we can improve DX a lot by creating utilities/helpers and restricting some arbitrary operations. The first option can lead customers to errors and an unclear experience on how to use this feature.

2 - Serving a Swagger UI

To implement this, check how chalice-spec implements it, by loading the swagger ui from their CDN, and passing the API spec into the page as a JSON file.

I really like that because we don't burden the customer with additional needs to handle static assets.

This would still require configuring the mapping in the API Gateway (or equivalent).

I'm thinking here that we don't have a workaround for this, if the customer doesn't have a route defined (or ANY) in the APIGW there's nothing to do. We must be very careful in communicating this in the documentation.

3 - Dependencies

Include apispec as a dependency or an optional dependency: the current version of apispec weighs around 29.6Kb. Should we just import the dependency, or mark it as optional?

This is actually 29.6kb if you just use the apispec and export to json, but there is an option to export to_yaml() and it requires the yaml library which is ~2MB. The OpenAPI specification specifies YAML as a possible format, and in my opinion we must consider this.
That said, I think we should bring the apispec as optional and install yaml library when installing apispec.

4 - Development environment

Maybe we can use POWERTOOLS_DEV or debug argument (like debug mode) to expose the SwaggerUI when developer did not set enable_swagger argument in ApiGatewayResolver?

I agree with @darnley, this can make life easier when running in development environments like SAM.

I'm eager to see this implementation and learn from your code, Ruben!
Thank you so much.

[sthulb]

That's a great idea and would be so powerful.
I do feel that once that's ready, we should also add a non event handler related generator. + add pydantic model support maybe it will be easier to implement?

I agree with this comment. I'd like to see the generation away from the handler – I think a few people will eventually forget to remove it and generate the OpenAPI spec in their actual functions or perhaps more frustrating to customers, forget to add it and wonder why it's not generating.

[rubenfonseca]

Hi everyone, I've updated the RFC with an example of the final DX that we are working on. Can you please take a look at it? There is still a couple of things to resolve, but I'll keep you posted with the updates.

[ran-isenberg]

Nice, looking very good!
Do you think we can get pydantic response/input schemas added?
I see that you have used 'value=json.dumps({"id": 1, "name": "John Smith"}),' in there, so perhaps that's a way?

[rubenfonseca]

@ran-isenberg direct integration with pydantic models would make this RFC / PR too big, but I'm happy to look at it as soon as this one is merged. I just want to make sure that we will be able to plug in any schema generator, and not be tied to Pydantic only.

[ran-isenberg]

@ran-isenberg direct integration with pydantic models would make this RFC / PR too big, but I'm happy to look at it as soon as this one is merged. I just want to make sure that we will be able to plug in any schema generator, and not be tied to Pydantic only.

makes sense.

[Tankanow]

Hi @rubenfonseca, thanks for the RFC. We have been thinking about how to better document all of our APIs. I have a lot of thoughts, but they're not quite fully baked:

1. Aggregation: I find that OpenAPI specs are most valuable when aggregated. For example, we deploy dozens of stacks of lambdas, where each stack may have lambdas that provide external (programmatic and web) APIs or internal (http or direct lambda invoke) APIs. The ideal solution is one where downstream I can easily aggregate all of those API docs into a tool, e.g. backstage.io for internal docs and our SaaS documentation platform for external docs. I'm a bit worried if the only access is via live endpoint. What about adding an offline generation mode?

2. Performance: Speaking of offline generation. Our platform is entirely lambda (so I love lambda), but it is not the most performant for HTTP APIs. I do worry about performance implications.

3. Integration: with other tooling within AWS, for example the OpenAPI and API Gateway. How would this integrate with teams who are already using AWS::Serverless::Api and OpenAPI Specs.

4. Synchronization and Code Maintenance: I love when specs are in the repo; however, full specs in the code can cause clutter. I like how the currently accepted proposal is for simple data structures that can be declared in a different module if necessary. It would also be nice if there were affordances for ensuring synchronization between the specs and the code. That's where something like Pydantic would be useful. That way a developer could use the same model for the spec and the code itself (thus keeping them in sync). In the future, something crazy like generating hypothesis tests that confirm the response from the lambda is in sync with the spec. (I know that this would not work for every implementation, I'm just dreaming here).

I don't expect you to address any of these in this RFC. Just wanted to post some ideas.

Thanks for an amazing library. ❤️

[dreamorosi]

To help me decide I’d like to understand better one of the tradeoffs.

Option 3 seems to offer the best of both worlds, but I’m not sure I’m seeing the full implication of the cons.

In which cases would I want to customize the handler for this specific route, besides adding middlewares to it?

[ran-isenberg]

must we include /swagger to the API?

[darnley]

Update: we are actively reviewing the code now that will add OpenAPI spec generation and validation. We figured we couldn't do one without the other anyway :)

Create a mechanism to expose a Swagger endpoint

I would like to gather feedback on how to do this. We came up with a couple of recommendations, trying to balance the experience and the security of the features.

Explicitly and manually

@app.get("/swagger", middlewares=[...]):
  return app.generate_openapi_swagger()

Pros:

• you have to write it, so you will remember you have to make it secure
• allows you to add custom logic inside the handler
• you can mount the swagger specification inside any path

Cons:

• you have to write it, it's not magic

Implicitly and magical

app = APIGatewayRestResolver(enable_validation=True, enable_swagger=True)

Pros:

• it's magic!

Cons:

• no easy way to specify middlewares
• no easy way to customize the path where swagger will be mounted

Something in between

app.enable_swagger() # defaults to /swagger, with no middleware

# alternatively
app.enable_swagger(path="/custom-swagger", middlewares=[...])

Pros:

• it's easy to write, the IDE will help
• you can still customize the mounting point, add middlewares for security, etc.
• super easy to add a app.enable_redoc(...) too

Cons:

• no easy way to customize the handler

What option do you prefer? Is there any other approach that you would prefer instead?

I like the second and third option. They look cleaner and I see some similarity with implementations in other frameworks/languages.

[Tankanow]

Hi @rubenfonseca,

Thanks for the proposal.

I'm worried about the use case where the swagger endpoint needs a different authorization model from the actual API endpoints. For example, let's say we want swagger for an API protected by a Lambda Authorizer that validates user-based Bearer JWTs. Do we really want to require a end-user Bearer token for the Swagger endpoint? That might be onerous for document automation.

Now, this ☝🏽 may be a corner case. So, ultimately, my vote is to make the common case easy and the hard cases possible.

Option 1 is a bit too much on the user and I'm having trouble thinking of customization the user would need outside of middleware. Also, to my point above, I think any real customization the user will need to do will live outside the app. Option 2 is too much magic and - more importantly - restricts you if you want/need to add more functionality in the future.

I vote for a combination of Option 3 with making the underlying generation code available as public methods in a utility module in case a user needs to expose the swagger via a separate lambda outside app.

[tonnico]

It would be very helpful when we could export the json/yaml somehow. I would prefer solution 1.

Btw. is json the preferred format or yaml? Is it configurable?

[ran-isenberg]

+1 for json/yaml export. I dont care for the /swagger at all.
I intend to embed the output in a github pages website.

[Tankanow]

As a datapoint, I too will be exporting this for consumption by a documentation tool.

[rubenfonseca]

@dreamorosi, @Tankanow, @wurstnase even if we go with Option 3, you can still write Option 1 yourself manually and do whatever you want with the handler, including different authorization, etc. We will have both an app.get_openapi_schema (returning a Pydantic model) and an app.get_openapi_json_schema (returning a JSON serialized representation of the model). So you are free to plug it in however you like, or even export it for consumption elsewhere. With Option 3, we just want to optimize for the most common use case, but you still have the possibility of doing it manually. We haven't considered YAML yet, but because we're generating a Pydantic model, it shouldn't be too difficult to serialize it as YAML.

@ran-isenberg no, that's just a default ("/swagger"), but you could customize it, of course.

@darnley thank you!

Thank you everyone for your great feedback! We will try to move forward with Option 3 while still giving you the flexibility to build your own logic and/or export the raw OpenAPI model for consumption/processing elsewhere.

[tonnico]

Currently there are only .get-examples. How will a .post endpoint will looks like? Can we add the body schema to the OpenApi model?

[leandrodamascena]

Currently there are only .get-examples. How will a .post endpoint will looks like? Can we add the body schema to the OpenApi model?

Hi @wurstnase! When you create a POST endpoint we are already adding requestBody to the schema generated by OpenAPI.

app.py

from dataclasses import dataclass, field
import split_route
import requests
from requests import Response
from pydantic import BaseModel, Field

from aws_lambda_powertools import Logger
from aws_lambda_powertools.event_handler import APIGatewayRestResolver
from aws_lambda_powertools.logging import correlation_paths
from aws_lambda_powertools.utilities.typing import LambdaContext

logger = Logger()
app = APIGatewayRestResolver(enable_validation=True)
#app.include_router(split_route.router)  

@dataclass
class Todo:
    id: int
    completed: bool
    title: str = field(default_factory="PT")

@app.post("/hello")
def post_todo(todo: Todo):
    logger.info("Processing todo", todo_title=todo.title)
    return {"todos": todo}


# You can continue to use other utilities just as before
@logger.inject_lambda_context(correlation_id_path=correlation_paths.API_GATEWAY_REST)
def lambda_handler(event: dict, context: LambdaContext) -> dict:
    print(app.get_openapi_json_schema(title="My API", version="2.5.0", description="Powertools OpenAPI"))
    return app.resolve(event, context)

OpenAPI Schema

{
  "openapi": "3.1.0",
  "info": {
    "title": "My API",
    "description": "Powertools OpenAPI",
    "version": "2.5.0"
  },
  "servers": [
    {
      "url": "/"
    }
  ],
  "paths": {
    "/hello": {
      "post": {
        "summary": "POST /hello",
        "operationId": "get_todos_hello_post",
        "requestBody": {
          "content": {
            "application/json": {
              "schema": {
                "$ref": "#/components/schemas/Todo"
              }
            }
          },
          "required": true
        },
        "responses": {
          "200": {
            "description": "Successful Response",
            "content": {
              "application/json": {
                "schema": {}
              }
            }
          },
          "422": {
            "description": "Validation Error",
            "content": {
              "application/json": {
                "schema": {
                  "$ref": "#/components/schemas/HTTPValidationError"
                }
              }
            }
          }
        }
      }
    }
  },
  "components": {
    "schemas": {
      "HTTPValidationError": {
        "properties": {
          "detail": {
            "items": {
              "$ref": "#/components/schemas/ValidationError"
            },
            "type": "array",
            "title": "Detail"
          }
        },
        "type": "object",
        "title": "HTTPValidationError"
      },
      "Todo": {
        "properties": {
          "id": {
            "type": "integer",
            "title": "Id"
          },
          "completed": {
            "type": "boolean",
            "title": "Completed"
          },
          "title": {
            "type": "string",
            "title": "Title"
          }
        },
        "type": "object",
        "required": [
          "id",
          "completed"
        ],
        "title": "Todo"
      },
      "ValidationError": {
        "properties": {
          "loc": {
            "items": {
              "anyOf": [
                {
                  "type": "string"
                },
                {
                  "type": "integer"
                }
              ]
            },
            "type": "array",
            "title": "Location"
          },
          "msg": {
            "type": "string",
            "title": "Message"
          },
          "type": {
            "type": "string",
            "title": "Error Type"
          }
        },
        "type": "object",
        "required": [
          "loc",
          "msg",
          "type"
        ],
        "title": "ValidationError"
      }
    }
  }
}

[tonnico]

Thanks for the example.

@app.post("/hello")
def get_todos(leo: Todo):
    ...

The body parsing as argument is a new feature?

[leandrodamascena]

The body parsing as argument is a new feature?

Yes, we have added this support and will release it when we release OpenAPI! For existing users there is no change, but if someone wants to use the body as an argument they must enable the enable_validation parameter, like this:

from aws_lambda_powertools.event_handler import APIGatewayRestResolver
app = APIGatewayRestResolver(enable_validation=True)

and then

@app.post("/hello")
def get_todos(todo: Todo):
    print(todo.id)
    return {"todo": todo}

[tlinhart]

Hi, how about documenting the headers? I'm creating a webhook handler which needs a Signature header to validate the event. My initial idea was not to use APIGatewayRestResolver but plain event_parser with a custom Pydantic model which would validate both the headers and the payload. I was also thinking about handling another route which would return the JSON schema returned by the model's model_json_schema method. However, seeing this initiative, it might be a better idea to use APIGatewayRestResolver instead to get all the things automatically. I can see support for POST body is already on the way, what about the headers, is it considered?

[rubenfonseca]

Hi, how about documenting the headers? I'm creating a webhook handler which needs a Signature header to validate the event. My initial idea was not to use APIGatewayRestResolver but plain event_parser with a custom Pydantic model which would validate both the headers and the payload. I was also thinking about handling another route which would return the JSON schema returned by the model's model_json_schema method. However, seeing this initiative, it might be a better idea to use APIGatewayRestResolver instead to get all the things automatically. I can see support for POST body is already on the way, what about the headers, is it considered?

@tlinhart thank you for this valuable feedback! This is something that we have in our head. However, the PR is getting already too big to handle, so we will initially merge without support for Headers. BUT I don't think it will require a lot of effort to add it ! Thank you again for raising this up! I'll make sure I open a new issue for this this one

[tlinhart]

Awesome, thanks!

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

This issue is now closed. Please be mindful that future comments are hard for our team to see.

If you need more assistance, please either tag a team member or open a new issue that references this one.

If you wish to keep having a conversation with other community members under this issue feel free to do so.

[leandrodamascena]

Hey everyone! We are excited to let you know that we have completed the first step to add support for OpenAPI Spec + SwaggerUI. After intense months of work, we have finally added support and plan to release this feature in the next version! We are writing the documentation to explain how to use this new feature.

This first version brings several functionalities, such as:

1 - Integration with Pydantic
2 - Dataclass support
3 - Support for serialization of different types of data in addition to JSON
4 - Data type validation
5 - Inference to know the types of input and output data and create models/bodyRequest
6 - Integration with all available resolvers: Rest, HTTP, ALB, LambdaFunction, VPC Lattice, and in the near future Bedrock Agent.
7 - SwaggerUI with self-hosted UI or built-in in the package.
8 - And several other things.

We know that we have things to improve, we have to develop a CLI to make your lives easier. Still, this first version is a big step in the project and fulfills our main objective: improving the development experience in Lambda environments.

Thank you for all your support, collaboration, and contributions. 🚀 🥇

[github-actions]

This is now released under 2.26.1 version!

[heitorlessa]

issue to document this new feature as it requires a new subsection only for OpenAPI: #3324