Merge branch 'develop' into firehose

Author: ran-isenberg

CHANGELOG.md

@@ -4,39 +4,63 @@
 <a name="unreleased"></a>
 # Unreleased
 
+## Documentation
+
+* **readme:** add lambda layer latest version badge
+
+## Maintenance
+
+* **layer:** bump to latest version 37
+
+
+<a name="v1.30.0"></a>
+## [v1.30.0] - 2022-10-05
 ## Bug Fixes
 
 * **apigateway:** update Response class to require status_code only ([#1560](https://github.com/awslabs/aws-lambda-powertools-python/issues/1560))
+* **ci:** integrate isort 5.0 with black to resolve conflicts
 * **event_sources:** implement Mapping protocol on DictWrapper for better interop with existing middlewares ([#1516](https://github.com/awslabs/aws-lambda-powertools-python/issues/1516))
 * **typing:** fix mypy error
 * **typing:** level arg in copy_config_to_registered_loggers ([#1534](https://github.com/awslabs/aws-lambda-powertools-python/issues/1534))
 
 ## Documentation
 
+* **batch:** document the new lambda context feature
+* **homepage:** introduce POWERTOOLS_DEV env var ([#1569](https://github.com/awslabs/aws-lambda-powertools-python/issues/1569))
+* **multiple:** fix highlighting after new isort/black integration
 * **parser:** add JSON string field extension example ([#1526](https://github.com/awslabs/aws-lambda-powertools-python/issues/1526))
 
 ## Features
 
+* **batch:** inject lambda_context if record handler signature accepts it ([#1561](https://github.com/awslabs/aws-lambda-powertools-python/issues/1561))
+* **event-handler:** context support to share data between routers ([#1567](https://github.com/awslabs/aws-lambda-powertools-python/issues/1567))
+* **logger:** introduce POWERTOOLS_DEBUG for internal debugging ([#1572](https://github.com/awslabs/aws-lambda-powertools-python/issues/1572))
+* **logger:** include logger name attribute when copy_config_to_registered_logger is used ([#1568](https://github.com/awslabs/aws-lambda-powertools-python/issues/1568))
 * **logger:** pretty-print JSON when POWERTOOLS_DEV is set ([#1548](https://github.com/awslabs/aws-lambda-powertools-python/issues/1548))
 
 ## Maintenance
 
-* **deps:** bump codecov/codecov-action from 3.1.0 to 3.1.1 ([#1529](https://github.com/awslabs/aws-lambda-powertools-python/issues/1529))
+* **dep:** bump pyproject to pypi sync
 * **deps:** bump fastjsonschema from 2.16.1 to 2.16.2 ([#1530](https://github.com/awslabs/aws-lambda-powertools-python/issues/1530))
 * **deps:** bump actions/setup-python from 3 to 4 ([#1528](https://github.com/awslabs/aws-lambda-powertools-python/issues/1528))
+* **deps:** bump codecov/codecov-action from 3.1.0 to 3.1.1 ([#1529](https://github.com/awslabs/aws-lambda-powertools-python/issues/1529))
+* **deps:** bump dependabot/fetch-metadata from 1.3.3 to 1.3.4 ([#1565](https://github.com/awslabs/aws-lambda-powertools-python/issues/1565))
 * **deps:** bump email-validator from 1.2.1 to 1.3.0 ([#1533](https://github.com/awslabs/aws-lambda-powertools-python/issues/1533))
+* **deps-dev:** bump mypy-boto3-secretsmanager from 1.24.54 to 1.24.83 ([#1557](https://github.com/awslabs/aws-lambda-powertools-python/issues/1557))
 * **deps-dev:** bump mkdocs-material from 8.5.3 to 8.5.4 ([#1563](https://github.com/awslabs/aws-lambda-powertools-python/issues/1563))
-* **deps-dev:** bump mypy-boto3-ssm from 1.24.80 to 1.24.81 ([#1544](https://github.com/awslabs/aws-lambda-powertools-python/issues/1544))
+* **deps-dev:** bump pytest-cov from 3.0.0 to 4.0.0 ([#1551](https://github.com/awslabs/aws-lambda-powertools-python/issues/1551))
+* **deps-dev:** bump flake8-bugbear from 22.9.11 to 22.9.23 ([#1541](https://github.com/awslabs/aws-lambda-powertools-python/issues/1541))
+* **deps-dev:** bump types-requests from 2.28.11 to 2.28.11.1 ([#1571](https://github.com/awslabs/aws-lambda-powertools-python/issues/1571))
 * **deps-dev:** bump mypy-boto3-ssm from 1.24.69 to 1.24.80 ([#1542](https://github.com/awslabs/aws-lambda-powertools-python/issues/1542))
 * **deps-dev:** bump mako from 1.2.2 to 1.2.3 ([#1537](https://github.com/awslabs/aws-lambda-powertools-python/issues/1537))
 * **deps-dev:** bump types-requests from 2.28.10 to 2.28.11 ([#1538](https://github.com/awslabs/aws-lambda-powertools-python/issues/1538))
-* **deps-dev:** bump flake8-bugbear from 22.9.11 to 22.9.23 ([#1541](https://github.com/awslabs/aws-lambda-powertools-python/issues/1541))
-* **deps-dev:** bump pytest-cov from 3.0.0 to 4.0.0 ([#1551](https://github.com/awslabs/aws-lambda-powertools-python/issues/1551))
 * **deps-dev:** bump mkdocs-material from 8.5.1 to 8.5.3 ([#1532](https://github.com/awslabs/aws-lambda-powertools-python/issues/1532))
-* **deps-dev:** bump mypy-boto3-secretsmanager from 1.24.54 to 1.24.83 ([#1557](https://github.com/awslabs/aws-lambda-powertools-python/issues/1557))
+* **deps-dev:** bump mypy-boto3-ssm from 1.24.80 to 1.24.81 ([#1544](https://github.com/awslabs/aws-lambda-powertools-python/issues/1544))
 * **deps-dev:** bump mypy-boto3-s3 from 1.24.36.post1 to 1.24.76 ([#1531](https://github.com/awslabs/aws-lambda-powertools-python/issues/1531))
 * **docs:** bump layer version to 36 (1.29.2)
 * **layers:** add dummy v2 layer automation
+* **lint:** use new isort black integration
+* **multiple:** localize powertools_dev env logic and warning ([#1570](https://github.com/awslabs/aws-lambda-powertools-python/issues/1570))
 
 
 <a name="v1.29.2"></a>
@@ -2366,7 +2390,8 @@
 * Merge pull request [#5](https://github.com/awslabs/aws-lambda-powertools-python/issues/5) from jfuss/feat/python38
 
 
-[Unreleased]: https://github.com/awslabs/aws-lambda-powertools-python/compare/v1.29.2...HEAD
+[Unreleased]: https://github.com/awslabs/aws-lambda-powertools-python/compare/v1.30.0...HEAD
+[v1.30.0]: https://github.com/awslabs/aws-lambda-powertools-python/compare/v1.29.2...v1.30.0
 [v1.29.2]: https://github.com/awslabs/aws-lambda-powertools-python/compare/v1.29.1...v1.29.2
 [v1.29.1]: https://github.com/awslabs/aws-lambda-powertools-python/compare/v1.29.0...v1.29.1
 [v1.29.0]: https://github.com/awslabs/aws-lambda-powertools-python/compare/v1.28.0...v1.29.0

README.md

@@ -1,8 +1,9 @@
+<!-- markdownlint-disable MD043 MD041 -->
 # AWS Lambda Powertools for Python
 
 [![Build](https://github.com/awslabs/aws-lambda-powertools-python/actions/workflows/python_build.yml/badge.svg)](https://github.com/awslabs/aws-lambda-powertools-python/actions/workflows/python_build.yml)
 [![codecov.io](https://codecov.io/github/awslabs/aws-lambda-powertools-python/branch/develop/graphs/badge.svg)](https://app.codecov.io/gh/awslabs/aws-lambda-powertools-python)
-![PythonSupport](https://img.shields.io/static/v1?label=python&message=3.6%20|%203.7|%203.8|%203.9&color=blue?style=flat-square&logo=python) ![PyPI version](https://badge.fury.io/py/aws-lambda-powertools.svg) ![PyPi monthly downloads](https://img.shields.io/pypi/dm/aws-lambda-powertools)
+![PythonSupport](https://img.shields.io/static/v1?label=python&message=3.6%20|%203.7|%203.8|%203.9&color=blue?style=flat-square&logo=python) ![PyPI version](https://badge.fury.io/py/aws-lambda-powertools.svg) ![PyPi monthly downloads](https://img.shields.io/pypi/dm/aws-lambda-powertools) ![Lambda Layer](https://api.globadge.com/v1/badgen/aws/lambda/layer/latest-version/eu-central-1/017000801446/AWSLambdaPowertoolsPython)
 [![Join our Discord](https://dcbadge.vercel.app/api/server/B8zZKbbyET)](https://discord.gg/B8zZKbbyET)
 
 A suite of Python utilities for AWS Lambda functions to ease adopting best practices such as tracing, structured logging, custom metrics, and more. (AWS Lambda Powertools [Java](https://github.com/awslabs/aws-lambda-powertools-java) and [Typescript](https://github.com/awslabs/aws-lambda-powertools-typescript) is also available).

aws_lambda_powertools/event_handler/api_gateway.py

@@ -10,12 +10,24 @@
 from enum import Enum
 from functools import partial
 from http import HTTPStatus
-from typing import Any, Callable, Dict, List, Match, Optional, Pattern, Set, Tuple, Type, Union
+from typing import (
+    Any,
+    Callable,
+    Dict,
+    List,
+    Match,
+    Optional,
+    Pattern,
+    Set,
+    Tuple,
+    Type,
+    Union,
+)
 
 from aws_lambda_powertools.event_handler import content_types
 from aws_lambda_powertools.event_handler.exceptions import NotFoundError, ServiceError
 from aws_lambda_powertools.shared import constants
-from aws_lambda_powertools.shared.functions import resolve_truthy_env_var_choice
+from aws_lambda_powertools.shared.functions import powertools_dev_is_set, strtobool
 from aws_lambda_powertools.shared.json_encoder import Encoder
 from aws_lambda_powertools.utilities.data_classes import (
     ALBEvent,
@@ -237,6 +249,7 @@ def build(self, event: BaseProxyEvent, cors: Optional[CORSConfig] = None) -> Dic
 class BaseRouter(ABC):
     current_event: BaseProxyEvent
     lambda_context: LambdaContext
+    context: dict
 
     @abstractmethod
     def route(
@@ -383,6 +396,14 @@ def lambda_handler(event, context):
         """
         return self.route(rule, "PATCH", cors, compress, cache_control)
 
+    def append_context(self, **additional_context):
+        """Append key=value data as routing context"""
+        self.context.update(**additional_context)
+
+    def clear_context(self):
+        """Resets routing context"""
+        self.context.clear()
+
 
 class ApiGatewayResolver(BaseRouter):
     """API Gateway and ALB proxy resolver
@@ -444,10 +465,9 @@ def __init__(
         self._cors = cors
         self._cors_enabled: bool = cors is not None
         self._cors_methods: Set[str] = {"OPTIONS"}
-        self._debug = resolve_truthy_env_var_choice(
-            env=os.getenv(constants.EVENT_HANDLER_DEBUG_ENV, "false"), choice=debug
-        )
+        self._debug = self._has_debug(debug)
         self._strip_prefixes = strip_prefixes
+        self.context: Dict = {}  # early init as customers might add context before event resolution
 
         # Allow for a custom serializer or a concise json serialization
         self._serializer = serializer or partial(json.dumps, separators=(",", ":"), cls=Encoder)
@@ -502,15 +522,37 @@ def resolve(self, event, context) -> Dict[str, Any]:
                 "You don't need to serialize event to Event Source Data Class when using Event Handler; see issue #1152"
             )
             event = event.raw_event
+
         if self._debug:
             print(self._json_dump(event), end="")
+
+        # Populate router(s) dependencies without keeping a reference to each registered router
         BaseRouter.current_event = self._to_proxy_event(event)
         BaseRouter.lambda_context = context
-        return self._resolve().build(self.current_event, self._cors)
+
+        response = self._resolve().build(self.current_event, self._cors)
+        self.clear_context()
+        return response
 
     def __call__(self, event, context) -> Any:
         return self.resolve(event, context)
 
+    @staticmethod
+    def _has_debug(debug: Optional[bool] = None) -> bool:
+        # It might have been explicitly switched off (debug=False)
+        if debug is not None:
+            return debug
+
+        # Maintenance: deprecate EVENT_HANDLER_DEBUG later in V2.
+        env_debug = os.getenv(constants.EVENT_HANDLER_DEBUG_ENV)
+        if env_debug is not None:
+            warnings.warn(
+                "POWERTOOLS_EVENT_HANDLER_DEBUG is set and will be deprecated in V2. Please use POWERTOOLS_DEV instead."
+            )
+            return strtobool(env_debug) or powertools_dev_is_set()
+
+        return powertools_dev_is_set()
+
     @staticmethod
     def _compile_regex(rule: str):
         """Precompile regex pattern
@@ -705,7 +747,7 @@ def _json_dump(self, obj: Any) -> str:
         return self._serializer(obj)
 
     def include_router(self, router: "Router", prefix: Optional[str] = None) -> None:
-        """Adds all routes defined in a router
+        """Adds all routes and context defined in a router
 
         Parameters
         ----------
@@ -718,6 +760,11 @@ def include_router(self, router: "Router", prefix: Optional[str] = None) -> None
         # Add reference to parent ApiGatewayResolver to support use cases where people subclass it to add custom logic
         router.api_resolver = self
 
+        # Merge app and router context
+        self.context.update(**router.context)
+        # use pointer to allow context clearance after event is processed e.g., resolve(evt, ctx)
+        router.context = self.context
+
         for route, func in router._routes.items():
             if prefix:
                 rule = route[0]
@@ -733,6 +780,7 @@ class Router(BaseRouter):
     def __init__(self):
         self._routes: Dict[tuple, Callable] = {}
         self.api_resolver: Optional[BaseRouter] = None
+        self.context = {}  # early init as customers might add context before event resolution
 
     def route(
         self,

aws_lambda_powertools/event_handler/appsync.py

@@ -12,6 +12,7 @@
 class BaseRouter:
     current_event: AppSyncResolverEventT  # type: ignore[valid-type]
     lambda_context: LambdaContext
+    context: dict
 
     def __init__(self):
         self._resolvers: dict = {}
@@ -34,6 +35,14 @@ def register_resolver(func):
 
         return register_resolver
 
+    def append_context(self, **additional_context):
+        """Append key=value data as routing context"""
+        self.context.update(**additional_context)
+
+    def clear_context(self):
+        """Resets routing context"""
+        self.context.clear()
+
 
 class AppSyncResolver(BaseRouter):
     """
@@ -68,6 +77,7 @@ def common_field() -> str:
 
     def __init__(self):
         super().__init__()
+        self.context = {}  # early init as customers might add context before event resolution
 
     def resolve(
         self, event: dict, context: LambdaContext, data_model: Type[AppSyncResolverEvent] = AppSyncResolverEvent
@@ -144,8 +154,12 @@ def lambda_handler(event, context):
         # Maintenance: revisit generics/overload to fix [attr-defined] in mypy usage
         BaseRouter.current_event = data_model(event)
         BaseRouter.lambda_context = context
+
         resolver = self._get_resolver(BaseRouter.current_event.type_name, BaseRouter.current_event.field_name)
-        return resolver(**BaseRouter.current_event.arguments)
+        response = resolver(**BaseRouter.current_event.arguments)
+        self.clear_context()
+
+        return response
 
     def _get_resolver(self, type_name: str, field_name: str) -> Callable:
         """Get resolver for field_name
@@ -182,9 +196,15 @@ def include_router(self, router: "Router") -> None:
         router : Router
             A router containing a dict of field resolvers
         """
+        # Merge app and router context
+        self.context.update(**router.context)
+        # use pointer to allow context clearance after event is processed e.g., resolve(evt, ctx)
+        router.context = self.context
+
         self._resolvers.update(router._resolvers)
 
 
 class Router(BaseRouter):
     def __init__(self):
         super().__init__()
+        self.context = {}  # early init as customers might add context before event resolution

aws_lambda_powertools/event_handler/lambda_function_url.py

@@ -1,7 +1,10 @@
 from typing import Callable, Dict, List, Optional
 
 from aws_lambda_powertools.event_handler import CORSConfig
-from aws_lambda_powertools.event_handler.api_gateway import ApiGatewayResolver, ProxyEventType
+from aws_lambda_powertools.event_handler.api_gateway import (
+    ApiGatewayResolver,
+    ProxyEventType,
+)
 from aws_lambda_powertools.utilities.data_classes import LambdaFunctionUrlEvent
 
 

aws_lambda_powertools/logging/formatter.py

@@ -9,7 +9,7 @@
 from typing import Any, Callable, Dict, Iterable, List, Optional, Tuple, Union
 
 from ..shared import constants
-from ..shared.functions import strtobool
+from ..shared.functions import powertools_dev_is_set
 
 RESERVED_LOG_ATTRS = (
     "name",
@@ -116,7 +116,7 @@ def __init__(
         self.json_deserializer = json_deserializer or json.loads
         self.json_default = json_default or str
         self.json_indent = (
-            constants.PRETTY_INDENT if strtobool(os.getenv("POWERTOOLS_DEV", "0")) else constants.COMPACT_INDENT
+            constants.PRETTY_INDENT if powertools_dev_is_set() else constants.COMPACT_INDENT
         )  # indented json serialization when in AWS SAM Local
         self.json_serializer = json_serializer or partial(
             json.dumps, default=self.json_default, separators=(",", ":"), indent=self.json_indent

aws_lambda_powertools/logging/logger.py

@@ -12,7 +12,11 @@
 from ..shared.functions import resolve_env_var_choice, resolve_truthy_env_var_choice
 from .exceptions import InvalidLoggerSamplingRateError
 from .filters import SuppressFilter
-from .formatter import RESERVED_FORMATTER_CUSTOM_KEYS, BasePowertoolsFormatter, LambdaPowertoolsFormatter
+from .formatter import (
+    RESERVED_FORMATTER_CUSTOM_KEYS,
+    BasePowertoolsFormatter,
+    LambdaPowertoolsFormatter,
+)
 from .lambda_context import build_lambda_context_model
 
 logger = logging.getLogger(__name__)

aws_lambda_powertools/logging/utils.py

@@ -81,6 +81,8 @@ def _configure_logger(source_logger: Logger, logger: logging.Logger, level: Unio
     logger.handlers = []
     logger.setLevel(level)
     logger.propagate = False  # ensure we don't propagate logs to existing loggers, #1073
+    source_logger.append_keys(name="%(name)s")  # include logger name, see #1267
+
     source_logger.debug(f"Logger {logger} reconfigured to use logging level {level}")
     for source_handler in source_logger.handlers:
         logger.addHandler(source_handler)

aws_lambda_powertools/package_logger.py

@@ -1,7 +1,25 @@
 import logging
 
+from aws_lambda_powertools.logging.logger import set_package_logger
+from aws_lambda_powertools.shared.functions import powertools_debug_is_set
+
+
+def set_package_logger_handler(stream=None):
+    """Sets up Lambda Powertools package logging.
+
+    By default, we discard any output to not interfere with customers logging.
+
+    When POWERTOOLS_DEBUG env var is set, we setup `aws_lambda_powertools` logger in DEBUG level.
+
+    Parameters
+    ----------
+    stream: sys.stdout
+        log stream, stdout by default
+    """
+
+    if powertools_debug_is_set():
+        return set_package_logger(stream=stream)
 
-def set_package_logger_handler():
     logger = logging.getLogger("aws_lambda_powertools")
     logger.addHandler(logging.NullHandler())
     logger.propagate = False

aws_lambda_powertools/shared/constants.py

@@ -36,3 +36,6 @@
 # JSON indentation level
 PRETTY_INDENT: int = 4
 COMPACT_INDENT = None
+
+POWERTOOLS_DEV_ENV: str = "POWERTOOLS_DEV"
+POWERTOOLS_DEBUG_ENV: str = "POWERTOOLS_DEBUG"