Decorator factory Feat: Create your own middleware (#17)

* feat(utils): add decorator factory

* improv: use partial to reduce complexity

* improv: add error handling

* chore: type hint

* docs: include pypi downloads badge

* feat: opt in to trace each middleware that runs

* improv: add initial util tests

* improv: test explicit and implicit trace_execution

* improv: test decorator with params

* chore: linting

* docs: include utilities

* improv: correct tests, dec_factory only for func

* improv: make util name more explicit

* improv: doc trace_execution, fix casting

* docs: add limitations, improve syntax

* docs: use new docs syntax

* fix: remove middleware decorator from libs

* feat: build docs in CI

* chore: linting

* fix: CI python-version type

* chore: remove docs CI

* chore: kick CI

* chore: include build badge master branch

* chore: refactor naming

* fix: rearrange tracing tests

* improv(tracer): toggle default auto patching

* feat(tracer): retrieve registered class instance

* fix(Makefile):  make cov target more explicit

* improv(Register): support multiple classes reg.

* improv(Register): inject class methods correctly

* docs: add how to reutilize Tracer

* improv(tracer): test auto patch method

* improv: address nicolas feedback

* improv: update example to reflect middleware feat

* fix: metric dimension in root blob

* chore: version bump

Co-authored-by: heitorlessa <[email protected]>

Author: heitorlessa

Diff for: python/HISTORY.md

@@ -1,5 +1,12 @@
 # HISTORY 
 
+## April 20th, 2020
+
+**0.7.0**
+
+* Introduces Middleware Factory to build your own middleware
+* Fixes Metrics dimensions not being included correctly in EMF
+
 ## April 9th, 2020
 
 **0.6.3**

Diff for: python/Makefile

@@ -17,7 +17,7 @@ lint: format
 test:
 	poetry run pytest -vvv
 
-test-html:
+coverage-html:
 	poetry run pytest --cov-report html
 
 pr: lint test 

Diff for: python/README.md

@@ -1,6 +1,6 @@
 # Lambda Powertools
 
-![PackageStatus](https://img.shields.io/static/v1?label=status&message=beta&color=blueviolet?style=flat-square) ![PythonSupport](https://img.shields.io/static/v1?label=python&message=3.6%20|%203.7|%203.8&color=blue?style=flat-square&logo=python) ![PyPI version](https://badge.fury.io/py/aws-lambda-powertools.svg) ![PyPi monthly downloads](https://img.shields.io/pypi/dm/aws-lambda-powertools)
+![PackageStatus](https://img.shields.io/static/v1?label=status&message=beta&color=blueviolet?style=flat-square) ![PythonSupport](https://img.shields.io/static/v1?label=python&message=3.6%20|%203.7|%203.8&color=blue?style=flat-square&logo=python) ![PyPI version](https://badge.fury.io/py/aws-lambda-powertools.svg) ![PyPi monthly downloads](https://img.shields.io/pypi/dm/aws-lambda-powertools) ![Build](https://github.com/awslabs/aws-lambda-powertools/workflows/Powertools%20Python/badge.svg?branch=master)
 
 A suite of utilities for AWS Lambda Functions that makes tracing with AWS X-Ray, structured logging and creating custom metrics asynchronously easier - Currently available for Python only and compatible with Python >=3.6.
 
@@ -32,12 +32,20 @@ A suite of utilities for AWS Lambda Functions that makes tracing with AWS X-Ray,
 * Validate against common metric definitions mistakes (metric unit, values, max dimensions, max metrics, etc)
 * No stack, custom resource, data collection needed — Metrics are created async by CloudWatch EMF
 
+**Bring your own middleware**
+
+* Utility to easily create your own middleware
+* Run logic before, after, and handle exceptions
+* Receive lambda handler, event, context
+* Optionally create sub-segment for each custom middleware
+
 **Environment variables** used across suite of utilities
 
 Environment variable | Description | Default | Utility
 ------------------------------------------------- | --------------------------------------------------------------------------------- | --------------------------------------------------------------------------------- | -------------------------------------------------
 POWERTOOLS_SERVICE_NAME | Sets service name used for tracing namespace, metrics dimensions and structured logging | "service_undefined" | all
 POWERTOOLS_TRACE_DISABLED | Disables tracing | "false" | tracing
+POWERTOOLS_TRACE_MIDDLEWARES | Creates sub-segment for each middleware created by lambda_handler_decorator | "false" | middleware_factory
 POWERTOOLS_LOGGER_LOG_EVENT | Logs incoming event | "false" | logging
 POWERTOOLS_LOGGER_SAMPLE_RATE | Debug log sampling  | 0 | logging
 POWERTOOLS_METRICS_NAMESPACE | Metrics namespace  | None | metrics
@@ -85,6 +93,23 @@ def handler(event, context)
   ...
 ```
 
+**Fetching a pre-configured tracer anywhere**
+
+```python
+# handler.py
+from aws_lambda_powertools.tracing import Tracer
+tracer = Tracer(service="payment")
+
+@tracer.capture_lambda_handler
+def handler(event, context)
+  charge_id = event.get('charge_id')
+  payment = collect_payment(charge_id)
+  ...
+
+# another_file.py
+from aws_lambda_powertools.tracing import Tracer
+tracer = Tracer(auto_patch=False) # new instance using existing configuration with auto patching overriden
+```
 
 ### Logging
 
@@ -154,7 +179,7 @@ def handler(event, context)
 }
 ```
 
-#### Custom Metrics async
+### Custom Metrics async
 
 > **NOTE** `log_metric` will be removed once it's GA.
 
@@ -204,6 +229,97 @@ with single_metric(name="ColdStart", unit=MetricUnit.Count, value=1) as metric:
     metric.add_dimension(name="function_context", value="$LATEST")
 ```
 
+
+### Utilities
+
+#### Bring your own middleware
+
+This feature allows you to create your own middleware as a decorator with ease by following a simple signature. 
+
+* Accept 3 mandatory args - `handler, event, context` 
+* Always return the handler with event/context or response if executed
+  - Supports nested middleware/decorators use case
+
+**Middleware with no params**
+
+```python
+from aws_lambda_powertools.middleware_factory import lambda_handler_decorator
+
+@lambda_handler_decorator
+def middleware_name(handler, event, context):
+    return handler(event, context)
+
+@lambda_handler_decorator
+def middleware_before_after(handler, event, context):
+    logic_before_handler_execution()
+    response = handler(event, context)
+    logic_after_handler_execution()
+    return response
+
+
+# middleware_name will wrap Lambda handler 
+# and simply return the handler as we're not pre/post-processing anything
+# then middleware_before_after will wrap middleware_name
+# run some code before/after calling the handler returned by middleware_name
+# This way, lambda_handler is only actually called once (top-down)
+@middleware_before_after # This will run last
+@middleware_name # This will run first
+def lambda_handler(event, context):
+    return True
+```
+
+**Middleware with params**
+
+```python
+@lambda_handler_decorator
+def obfuscate_sensitive_data(handler, event, context, fields=None):
+    # Obfuscate email before calling Lambda handler
+    if fields:
+        for field in fields:
+            field = event.get(field, "")
+            event[field] = obfuscate_pii(field)
+
+    return handler(event, context)
+
+@obfuscate_sensitive_data(fields=["email"])
+def lambda_handler(event, context):
+    return True
+```
+
+**Optionally trace middleware execution**
+
+This makes use of an existing Tracer instance that you may have initialized anywhere in your code, otherwise it'll initialize one using default options and provider (X-Ray).
+
+```python
+from aws_lambda_powertools.middleware_factory import lambda_handler_decorator
+
+@lambda_handler_decorator(trace_execution=True)
+def middleware_name(handler, event, context):
+    return handler(event, context)
+
+@middleware_name
+def lambda_handler(event, context):
+    return True
+```
+
+Optionally, you can enrich the final trace with additional annotations and metadata by retrieving a copy of the Tracer used.
+
+```python
+from aws_lambda_powertools.middleware_factory import lambda_handler_decorator
+from aws_lambda_powertools.tracing import Tracer
+
+@lambda_handler_decorator(trace_execution=True)
+def middleware_name(handler, event, context):
+    tracer = Tracer() # Takes a copy of an existing tracer instance
+    tracer.add_anotation...
+    tracer.metadata...
+    return handler(event, context)
+
+@middleware_name
+def lambda_handler(event, context):
+    return True
+```
+
 ## Beta
 
 > **[Progress towards GA](https://github.com/awslabs/aws-lambda-powertools/projects/1)**

Diff for: python/aws_lambda_powertools/logging/logger.py

@@ -93,32 +93,34 @@ def logger_inject_lambda_context(lambda_handler: Callable[[Dict, Any], Any] = No
     Environment variables
     ---------------------
     POWERTOOLS_LOGGER_LOG_EVENT : str
-        instruct logger to log Lambda Event (e.g. "true", "True", "TRUE")
+        instruct logger to log Lambda Event (e.g. `"true", "True", "TRUE"`)
 
     Example
     -------
-    Captures Lambda contextual runtime info (e.g memory, arn, req_id)
-        >>> from aws_lambda_powertools.logging import logger_setup, logger_inject_lambda_context
-        >>> import logging
-        >>>
-        >>> logger = logging.getLogger(__name__)
-        >>> logging.setLevel(logging.INFO)
-        >>> logger_setup()
-        >>>
-        >>> @logger_inject_lambda_context
-        >>> def handler(event, context):
+    **Captures Lambda contextual runtime info (e.g memory, arn, req_id)**
+
+        from aws_lambda_powertools.logging import logger_setup, logger_inject_lambda_context
+        import logging
+
+        logger = logging.getLogger(__name__)
+        logging.setLevel(logging.INFO)
+        logger_setup()
+
+        @logger_inject_lambda_context
+        def handler(event, context):
                 logger.info("Hello")
 
-    Captures Lambda contextual runtime info and logs incoming request
-        >>> from aws_lambda_powertools.logging import logger_setup, logger_inject_lambda_context
-        >>> import logging
-        >>>
-        >>> logger = logging.getLogger(__name__)
-        >>> logging.setLevel(logging.INFO)
-        >>> logger_setup()
-        >>>
-        >>> @logger_inject_lambda_context(log_event=True)
-        >>> def handler(event, context):
+    **Captures Lambda contextual runtime info and logs incoming request**
+
+        from aws_lambda_powertools.logging import logger_setup, logger_inject_lambda_context
+        import logging
+
+        logger = logging.getLogger(__name__)
+        logging.setLevel(logging.INFO)
+        logger_setup()
+
+        @logger_inject_lambda_context(log_event=True)
+        def handler(event, context):
                 logger.info("Hello")
 
     Returns
@@ -128,9 +130,7 @@ def logger_inject_lambda_context(lambda_handler: Callable[[Dict, Any], Any] = No
     """
 
     # If handler is None we've been called with parameters
-    # We then return a partial function with args filled
-    # Next time we're called we'll call our Lambda
-    # This allows us to avoid writing wrapper_wrapper type of fn
+    # Return a partial function with args filled
     if lambda_handler is None:
         logger.debug("Decorator called with parameters")
         return functools.partial(logger_inject_lambda_context, log_event=log_event)
@@ -178,14 +178,16 @@ def log_metric(
 ):
     """Logs a custom metric in a statsD-esque format to stdout.
 
+    **This will be removed when GA - Use `aws_lambda_powertools.metrics.metrics.Metrics` instead**
+
     Creating Custom Metrics synchronously impact on performance/execution time.
     Instead, log_metric prints a metric to CloudWatch Logs.
     That allows us to pick them up asynchronously via another Lambda function and create them as a metric.
 
     NOTE: It takes up to 9 dimensions by default, and Metric units are conveniently available via MetricUnit Enum.
     If service is not passed as arg or via env var, "service_undefined" will be used as dimension instead.
 
-    Output in CloudWatch Logs: MONITORING|<metric_value>|<metric_unit>|<metric_name>|<namespace>|<dimensions>
+    **Output in CloudWatch Logs**: `MONITORING|<metric_value>|<metric_unit>|<metric_name>|<namespace>|<dimensions>`
 
     Serverless Application Repository App that creates custom metric from this log output:
     https://serverlessrepo.aws.amazon.com/applications/arn:aws:serverlessrepo:us-east-1:374852340823:applications~async-custom-metrics
@@ -195,23 +197,39 @@ def log_metric(
     POWERTOOLS_SERVICE_NAME: str
         service name
 
+    Parameters
+    ----------
+    name : str
+        metric name, by default None
+    namespace : str
+        metric namespace (e.g. application name), by default None
+    unit : MetricUnit, by default MetricUnit.Count
+        metric unit enum value (e.g. MetricUnit.Seconds), by default None\n
+        API Info: https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_MetricDatum.html
+    value : float, optional
+        metric value, by default 0
+    service : str, optional
+        service name used as dimension, by default "service_undefined"
+    dimensions: dict, optional
+        keyword arguments as additional dimensions (e.g. `customer=customerId`)
+
     Example
     -------
-    Log metric to count number of successful payments; define service via env var
+    **Log metric to count number of successful payments; define service via env var**
 
         $ export POWERTOOLS_SERVICE_NAME="payment"
-        >>> from aws_lambda_powertools.logging import MetricUnit, log_metric
-        >>> log_metric(
+        from aws_lambda_powertools.logging import MetricUnit, log_metric
+        log_metric(
             name="SuccessfulPayments",
             unit=MetricUnit.Count,
             value=1,
             namespace="DemoApp"
         )
 
-    Log metric to count number of successful payments per campaign & customer
+    **Log metric to count number of successful payments per campaign & customer**
 
-        >>> from aws_lambda_powertools.logging import MetricUnit, log_metric
-        >>> log_metric(
+        from aws_lambda_powertools.logging import MetricUnit, log_metric
+        log_metric(
             name="SuccessfulPayments",
             service="payment",
             unit=MetricUnit.Count,
@@ -220,22 +238,6 @@ def log_metric(
             campaign=campaign_id,
             customer=customer_id
         )
-
-    Parameters
-    ----------
-    name : str
-        metric name, by default None
-    namespace : str
-        metric namespace (e.g. application name), by default None
-    unit : MetricUnit, by default MetricUnit.Count
-        metric unit enum value (e.g. MetricUnit.Seconds), by default None
-        API Info: https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_MetricDatum.html
-    value : float, optional
-        metric value, by default 0
-    service : str, optional
-        service name used as dimension, by default "service_undefined"
-    dimensions: dict, optional
-        keyword arguments as additional dimensions (e.g. customer=customerId)
     """
 
     warnings.warn(message="This method will be removed in GA; use Metrics instead", category=DeprecationWarning)

Diff for: python/aws_lambda_powertools/metrics/base.py

@@ -177,6 +177,7 @@ def serialize_metric_set(self, metrics: Dict = None, dimensions: Dict = None) ->
         }
         metrics_timestamp = {"Timestamp": int(datetime.datetime.now().timestamp() * 1000)}
         metric_set["_aws"] = {**metrics_timestamp, **metrics_definition}
+        metric_set.update(**dimensions)
 
         try:
             logger.debug("Validating serialized metrics against CloudWatch EMF schema", metric_set)

Diff for: python/aws_lambda_powertools/middleware_factory/__init__.py

@@ -0,0 +1,4 @@
+""" Utilities to enhance middlewares """
+from .factory import lambda_handler_decorator
+
+__all__ = ["lambda_handler_decorator"]