Skip to content

Commit 1e7388c

Browse files
roger-zhanggleandrodamascenaVandita2020Cavalcante Damascena
authored
feat(idempotency): adding redis as idempotency backend (#2567)
* (redis): initial commit * feat(redis/idempotency): creating redis connections * feat(redis/idempotency): creating redis connections * feat(redis/idempotency): fixing import * feat(redis/idempotency): adding base class * feat(redis/idempotency): adding logic to get record * feat(redis/idempotency): adding expiry timeout * feat(redis) - refactoring connection and fixing mypy errors * feat(redis) - removing wrong print * feat(redis) - removing fields and adding additional logic to validate the idempotency key * feat(redis) - adding redis as dev dependency * Update idempotency.md Signed-off-by: Vandita Patidar <[email protected]> * Update idempotency.md Signed-off-by: Vandita Patidar <[email protected]> * resolve regarding to ruben's comment * resolve regarding to ruben's comment * local test, minor fixes * add redis to extra dep * fix git issue * fix docstring * fix poetry, address some Leandro's suggestion * change redis connection * add mock redis, redis validation, sentinel support * fix test * add redis * add doc, few todos still need to address * add docs, test. Removed Connection * fix test on delete * add types-redis to pyproj * fix Literal for 3.7 * add comment for delete * add redis as dev dep * fix poetry * Simplifying DX * remove redis-config in doc,test * handle race condition * remove todo * change to use redis.set * support decode_response=False * fix mock redis * add test for no decode * fix docstring * fix coverage * add a test case to demostrate race condition * add a race condition test for empty record * add abs_lambda_path, protocol for redis client * fix typing * remove awaitable * optimize protocol * Fix Bandit issue * Refactoring integration tests to use testcontainers * Removing code smell * fix makefile, remove sentinel setup * Adding e2e tests * Testing pipeline * Removing things * Removing things * Fixing docstring and removing old code * Improving the documentation * Highlights + code removal * Removing unnecessary tests * Removing unnecessary tests * Documentation * Addressing initial Ruben's feedback * Addressing Ruben's feedback - documentation * Addressing Ruben's feedback - docstring * Addressing Ruben's feedback - SSL * Addressing Ruben's feedback - db_index * Addressing Ruben's feedback - db_index * Addressing Ruben's feedback - db_index * redis comment improvements, minor refactor. * fix example and docstring import * fix import in inegration test * Addressing Ruben's feedback - Redis name * Addressing Ruben's feedback - Comments * Minor changes in the documentation * Minor changes in the documentation * Removing Redis as builti-in dependency in our layer * Adding Redis to install when creating a new env * Adressing Ruben's feedback * Making sonar happy * e2e failing due to wrong import --------- Signed-off-by: Vandita Patidar <[email protected]> Signed-off-by: Leandro Damascena <[email protected]> Co-authored-by: Leandro Damascena <[email protected]> Co-authored-by: Vandita Patidar <[email protected]> Co-authored-by: Leandro Damascena <[email protected]> Co-authored-by: Cavalcante Damascena <[email protected]>
1 parent 0bc47c5 commit 1e7388c

37 files changed

+2658
-152
lines changed

Diff for: .github/workflows/quality_check_pydanticv2.yml

+3-1
Original file line numberDiff line numberDiff line change
@@ -58,7 +58,9 @@ jobs:
5858
python-version: ${{ matrix.python-version }}
5959
cache: "poetry"
6060
- name: Replacing Pydantic v1 with v2 > 2.0.3
61-
run: poetry add "pydantic=^2.0.3"
61+
run: |
62+
rm -rf poetry.lock
63+
poetry add "pydantic=^2.0.3"
6264
- name: Install dependencies
6365
run: make dev
6466
- name: Test with pytest

Diff for: Makefile

+2-2
Original file line numberDiff line numberDiff line change
@@ -8,13 +8,13 @@ dev:
88
pip install --upgrade pip pre-commit poetry
99
poetry config --local virtualenvs.in-project true
1010
@$(MAKE) dev-version-plugin
11-
poetry install --extras "all datamasking-aws-sdk"
11+
poetry install --extras "all datamasking-aws-sdk redis"
1212
pre-commit install
1313

1414
dev-gitpod:
1515
pip install --upgrade pip poetry
1616
@$(MAKE) dev-version-plugin
17-
poetry install --extras "all datamasking-aws-sdk"
17+
poetry install --extras "all datamasking-aws-sdk redis"
1818
pre-commit install
1919

2020
format:

Diff for: aws_lambda_powertools/shared/functions.py

+30
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@
66
import os
77
import warnings
88
from binascii import Error as BinAsciiError
9+
from pathlib import Path
910
from typing import Any, Dict, Generator, Optional, Union, overload
1011

1112
from aws_lambda_powertools.shared import constants
@@ -250,3 +251,32 @@ def dataclass_to_dict(data) -> dict:
250251
import dataclasses
251252

252253
return dataclasses.asdict(data)
254+
255+
256+
def abs_lambda_path(relative_path: str = "") -> str:
257+
"""Return the absolute path from the given relative path to lambda handler.
258+
259+
Parameters
260+
----------
261+
relative_path : str, optional
262+
The relative path to the lambda handler, by default an empty string.
263+
264+
Returns
265+
-------
266+
str
267+
The absolute path generated from the given relative path.
268+
If the environment variable LAMBDA_TASK_ROOT is set, it will use that value.
269+
Otherwise, it will use the current working directory.
270+
If the path is empty, it will return the current working directory.
271+
"""
272+
# Retrieve the LAMBDA_TASK_ROOT environment variable or default to an empty string
273+
current_working_directory = os.environ.get("LAMBDA_TASK_ROOT", "")
274+
275+
# If LAMBDA_TASK_ROOT is not set, use the current working directory
276+
if not current_working_directory:
277+
current_working_directory = str(Path.cwd())
278+
279+
# Combine the current working directory and the relative path to get the absolute path
280+
absolute_path = str(Path(current_working_directory, relative_path))
281+
282+
return absolute_path

Diff for: aws_lambda_powertools/utilities/idempotency/__init__.py

+7-1
Original file line numberDiff line numberDiff line change
@@ -11,4 +11,10 @@
1111

1212
from .idempotency import IdempotencyConfig, idempotent, idempotent_function
1313

14-
__all__ = ("DynamoDBPersistenceLayer", "BasePersistenceLayer", "idempotent", "idempotent_function", "IdempotencyConfig")
14+
__all__ = (
15+
"DynamoDBPersistenceLayer",
16+
"BasePersistenceLayer",
17+
"idempotent",
18+
"idempotent_function",
19+
"IdempotencyConfig",
20+
)

Diff for: aws_lambda_powertools/utilities/idempotency/exceptions.py

+18
Original file line numberDiff line numberDiff line change
@@ -83,3 +83,21 @@ class IdempotencyNoSerializationModelError(BaseError):
8383
"""
8484
No model was supplied to the serializer
8585
"""
86+
87+
88+
class IdempotencyPersistenceConfigError(BaseError):
89+
"""
90+
The idempotency persistency configuration was unsupported
91+
"""
92+
93+
94+
class IdempotencyPersistenceConnectionError(BaseError):
95+
"""
96+
Idempotency persistence connection error
97+
"""
98+
99+
100+
class IdempotencyPersistenceConsistencyError(BaseError):
101+
"""
102+
Idempotency persistency consistency error, needs to be removed
103+
"""

Diff for: aws_lambda_powertools/utilities/idempotency/persistence/base.py

-1
Original file line numberDiff line numberDiff line change
@@ -374,7 +374,6 @@ def save_inprogress(self, data: Dict[str, Any], remaining_time_in_millis: Option
374374
now = datetime.datetime.now()
375375
period = datetime.timedelta(milliseconds=remaining_time_in_millis)
376376
timestamp = (now + period).timestamp()
377-
378377
data_record.in_progress_expiry_timestamp = int(timestamp * 1000)
379378
else:
380379
warnings.warn(

Diff for: aws_lambda_powertools/utilities/idempotency/persistence/dynamodb.py

+2
Original file line numberDiff line numberDiff line change
@@ -66,6 +66,8 @@ def __init__(
6666
DynamoDB attribute name for status, by default "status"
6767
data_attr: str, optional
6868
DynamoDB attribute name for response data, by default "data"
69+
validation_key_attr: str, optional
70+
DynamoDB attribute name for hashed representation of the parts of the event used for validation
6971
boto_config: botocore.config.Config, optional
7072
Botocore configuration to pass during client initialization
7173
boto3_session : boto3.Session, optional

0 commit comments

Comments
 (0)