RFC: Collection of tiny utilities

[heitorlessa]

Background

At GA, Powertools will offer Tracer, Metrics, Logger, and Middleware factory as the core utilities.

Optionally, I'm pondering on the idea of providing a set of tiny handy utilities that can be used either as standalone functions, or as part of a custom middleware - For example, JSON serialization, detect retries, fetch secrets, etc.

The benefit of providing utilities as functions is two-fold a) ease of maintenance, b) pick and choose and create a single custom middleware instead of nesting a myriad of decorators.

Propose solution

Use case: Custom middleware using a handful of utilities

from aws_lambda_powertools.middleware_factory import lambda_handler_decorator
from aws_lambda_powertools.utilities import validate_event, cors, decode_json, detect_retry

@lambda_handler_decorator
def custom_middleware_name(handler, event, context):
    # Before
    detect_retry(event)
    event = decode_json(event)
    validate_event(event)

    response = handler(event, context)

    # After
    response = cors(response)
    return response

@custom_middleware_name
def lambda_handler(event, context):
    ...

Use case: Using utilities standalone

from aws_lambda_powertools.utilities import validate_event, cors, decode_json, detect_retry

@custom_middleware_name
def lambda_handler(event, context):
    detect_retry(event)
    event = decode_json(event)
    validate_event(event)

    return cors(result)

Request for comments

As part of this RFC, I'd like to know what utilities are the most useful to have upfront - Leave a comment, and vote using 👍 on each comment instead of a new comment.

For ideas, here are some utilities as decorators created by other awesome authors: Lambda Decorators by Grid smarter cities, and Lambda Decorators by Daniel Schep.

Tenets

• AWS Lambda only – We optimise for AWS Lambda function environments only. Utilities might work with web frameworks and non-Lambda environments, though they are not officially supported.
• Eases the adoption of best practices – The main priority of the utilities is to facilitate best practices adoption, as defined in the AWS Well-Architected Serverless Lens; all other functionality is optional.
• Keep it lean – Additional dependencies are carefully considered for security and ease of maintenance, and prevent negatively impacting startup time.
• We strive for backwards compatibility – New features and changes should keep backwards compatibility. If a breaking change cannot be avoided, the deprecation and migration process should be clearly defined.
• We work backwards from the community – We aim to strike a balance of what would work best for 80% of customers. Emerging practices are considered and discussed via Requests for Comment (RFCs)
• Idiomatic – Utilities follow programming language idioms and language-specific best practices.

* Core utilities are Tracer, Logger and Metrics. Optional utilities may vary across languages.

[nmoutschen]

One question I have here is the balance regarding the number of utility functions. This should provide utilities that are useful for most use cases out of the box (e.g. CORS, validate authentication for APIGW, etc.) and not add additional dependencies to this project. Having strong guidelines here would prevent bloat/scope creep.

Some are hard to implement without external dependencies/prerequisites. E.g. detecting retries in a case where a Lambda function has multiple execution environments means fetching it from an external data source.

Here are some I'd find useful:

• Around common Lambda use-cases (processing events from API Gateway proxy integration, creating API Gateway responses, SNS, SQS, Kinesis, etc.).
• Validating event against a JSON schema (if it doesn't add too much dependencies).

[heitorlessa]

Absolutely - I wasn't explicit as I didn't publish the tenets yet but we'd favour those that don't need additional dependencies, and second the most useful cases.

We use fastjsonschema so a schema validator isn't a problem. Detecting retries would mean a retried Lambda invocation where the request_id from context would be the same - There are other more sophisticated ways to do, that's for sure.

[keithrozario]

Would definitely love to see a native download to memory functionality for S3. Current method of doing it is too clunky.

Something like:

data = aws_lambda_powertools.json_from_s3('bucket', '/path/to/file.json')
some_config = data['some']['config']

[bahrmichael]

I had troubles with the Decimal type from DynamoDB. I'm now using import simplejson as json everywhere as a workaround. I'm curious if that workaround can be replaced with something better.

[bahrmichael]

To auto enable CORS with SPAs I wrote my own class to convert dict to APIGW compatible payloads.

import json
from log import logger


def deep_render(body, to_json_method):
    if isinstance(body, str) or isinstance(body, int) or isinstance(body, dict):
        return body
    elif isinstance(body, list):
        result = []
        for el in body:
            result.append(deep_render(el, to_json_method))
        return result
    else:
        return to_json_method(body)


class ApiResponse:

    status_code = 200
    body = None
    headers = {
        "Access-Control-Allow-Origin": "*",
        "Access-Control-Allow-Credentials": "true"
    }

    def __init__(self, status_code=None, body=None, headers=None):
        if status_code is not None:
            self.status_code = status_code
        if body is not None:
            self.body = body
        if headers is not None:
            for key, value in headers.items():
                self.headers[key] = value

    def render(self, to_json_method=None):
        response = {'statusCode': self.status_code, 'headers': self.headers}
        if self.body is not None:
            self.body = deep_render(self.body, to_json_method)
            response['body'] = json.dumps(self.body)
        logger.debug('Response: %s', json.dumps(response))
        return response

Note that this code is more than a year old and was written when I started out with Lambda+Python. I hope there's something better out there that I just was too lazy to search for yet. Having some help to convert payloads to CORS compatible responses plus fully configurable status codes (I might want to return a payload and a certain http error code) would be nice.

[keithrozario]

+1 for the DynamoDB decimal helper. I paste a helper function in my code I copied from AWS, but better if that is in a proper library somewhere.

[heitorlessa]

Hey everyone - I've just published the Tenets to help us prioritize suggestions from this RFC.

Adding them below to facilitate discussions

Tenets

• AWS Lambda only – We optimise for AWS Lambda function environments only. Utilities might work with web frameworks and non-Lambda environments, though they are not officially supported.
• Eases the adoption of best practices – The main priority of the utilities is to facilitate best practices adoption, as defined in the AWS Well-Architected Serverless Lens; all other functionality is optional.
• Keep it lean – Additional dependencies are carefully considered for security and ease of maintenance, and prevent negatively impacting startup time.
• We strive for backwards compatibility – New features and changes should keep backwards compatibility. If a breaking change cannot be avoided, the deprecation and migration process should be clearly defined.
• We work backwards from the community – We aim to strike a balance of what would work best for 80% of customers. Emerging practices are considered and discussed via Requests for Comment (RFCs)
• Idiomatic – Utilities follow programming language idioms and language-specific best practices.

* Core utilities are Tracer, Logger and Metrics. Optional utilities may vary across languages.

[keithrozario]

I fully agree to this, great job @heitorlessa. Especially on the AWS Lambda only, and keeping it lean.

1. But I always thought this was Python Specific -- but the tenets seem to suggest a more broader language base. Are we planning for something in other languages.

2. Should there be a tenet for us to continue support for Python 3.6 (and above) or at least support officially supported runtimes. (stop supporting Python3.6 only when support for it expires on Lambda)

3. I think backwards compatibility is a real important one -- as everybody loves that aspect of AWS. That being said, should a breaking change be necessary, because this is a python package, it probably isn't that big a deal, since consumers can upgrade their packages whenever they're ready, and continue using the older one till then. I'm on the fence over whether this should be a tenet -- or if it is, we should be more specific, something along the lines of we will only break backwards compatibility if there are security considerations...etc.

[heitorlessa]

Hey @keithrozario - Your suggestion is spot on, we do plan to have other languages after Python is GA :)

On 2., that's a very good point - Let me think this through as to whether this should be a tenet or state this more openly in the project's description.

On 3., I'm planning to create GitHub Issue and PR templates to make it explicit the reasoning for a breaking change, and migration process to this regard. Breaking changes are inevitable, but as you said, I want to make sure this is a tenet so all contributions, new features, or changes by AWS or outside have this in mind.

API themselves are now stable, so I'll focus on creating a docs website using portray, and create Issue templates to make this easier to maintain as we think of other languages to work on - Feature Design, Feature Request, RFC, Breaking Change checklist, etc.

It goes without saying that I appreciate the support :)