Skip to content

feat(parameters): accept boto3_client to support private endpoints and ease testing #1096

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 21 commits into from
May 19, 2022
Merged

feat(parameters): accept boto3_client to support private endpoints and ease testing #1096

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
ce851e9
feat(parameters): Allow settings boto3.client() arguments
Apr 6, 2022
b9b6753
added tests
Apr 6, 2022
159ca22
added tests
Apr 6, 2022
d33201e
chore: paying older debts on grammar
heitorlessa Apr 8, 2022
2fbdca3
added mypy type hints
Apr 10, 2022
ac5554f
merge main
Apr 18, 2022
cd426fe
cr fixes
Apr 18, 2022
cd71ee0
cr fix
Apr 19, 2022
ccc378b
merge master
Apr 19, 2022
2a77841
Merge branch 'develop' into boto
heitorlessa May 18, 2022
9b99f0b
chore: encapsulate client building logic in staticmethod
heitorlessa May 18, 2022
d2c8f30
feat(parameters): add boto3_client for dynamodb resource client
heitorlessa May 18, 2022
d854f1c
chore: shorten docstrings, fix union single typehint
heitorlessa May 18, 2022
63bdde6
chore: ignore mypy ret-type/misc caught by new typed client
heitorlessa May 18, 2022
3528b8e
docs: add note on boto3_client usefulness for searching
heitorlessa May 18, 2022
579f8e1
chore: tests for dynamodb custom boto3 client
heitorlessa May 18, 2022
a8350ae
chore: skip test coverage at runtime type_checking
heitorlessa May 18, 2022
cec2f33
chore: re-add boto config fixture custom client test
heitorlessa May 18, 2022
19ec9ae
chore: remove super leftover
heitorlessa May 19, 2022
58e2d5e
chore: final adjustments
heitorlessa May 19, 2022
98dc539
chore: correct leftover table client
heitorlessa May 19, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 10 additions & 3 deletions aws_lambda_powertools/utilities/parameters/appconfig.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,9 @@ class AppConfigProvider(BaseProvider):
config: botocore.config.Config, optional
Botocore configuration to pass during client initialization
boto3_session : boto3.session.Session, optional
Boto3 session to use for AWS API communication
Boto3 session to use for AWS API communication, will not be used if boto3_client is not None
boto3_client: boto3.client, optional
Boto3 Client to use for AWS API communication, will be used instead of boto3_session if both provided

Example
-------
Expand Down Expand Up @@ -68,14 +70,19 @@ def __init__(
application: Optional[str] = None,
config: Optional[Config] = None,
boto3_session: Optional[boto3.session.Session] = None,
boto3_client: Optional[boto3.client] = None,
):
"""
Initialize the App Config client
"""

config = config or Config()
session = boto3_session or boto3.session.Session()
self.client = session.client("appconfig", config=config)
if boto3_client is not None:
self.client = boto3_client
else:
session = boto3_session or boto3.session.Session()
self.client = session.client("appconfig", config=config)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we turn this into a builder static method in the parent class? Return type could be Any as we will be able to cast the correct one for each client built w/ mypy_boto3.

This will prevent future an accidental logic discrepancy between them and localize change. It could accept the boto3_client, boto3_session, boto3_config, and boto3_service_name to initialize.

Let me know if it doesn't make sense

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i didnt add it to the dymamo class becuase we dont use a client there, not usre you can create a table from a client class.
so if it's not the dynamodb, it shouldn't be in the parent

self.application = resolve_env_var_choice(
choice=application, env=os.getenv(constants.SERVICE_NAME_ENV, "service_undefined")
)
Expand Down
18 changes: 14 additions & 4 deletions aws_lambda_powertools/utilities/parameters/secrets.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,9 @@ class SecretsProvider(BaseProvider):
config: botocore.config.Config, optional
Botocore configuration to pass during client initialization
boto3_session : boto3.session.Session, optional
Boto3 session to use for AWS API communication
Boto3 session to use for AWS API communication, will not be used if boto3_client is not None
boto3_client: boto3.client, optional
Boto3 Client to use for AWS API communication, will be used instead of boto3_session if both provided

Example
-------
Expand Down Expand Up @@ -60,14 +62,22 @@ class SecretsProvider(BaseProvider):

client: Any = None

def __init__(self, config: Optional[Config] = None, boto3_session: Optional[boto3.session.Session] = None):
def __init__(
self,
config: Optional[Config] = None,
boto3_session: Optional[boto3.session.Session] = None,
boto3_client: Optional[boto3.client] = None,
):
"""
Initialize the Secrets Manager client
"""

config = config or Config()
session = boto3_session or boto3.session.Session()
self.client = session.client("secretsmanager", config=config)
if boto3_client is not None:
self.client = boto3_client
else:
session = boto3_session or boto3.session.Session()
self.client = session.client("secretsmanager", config=config)

super().__init__()

Expand Down
18 changes: 14 additions & 4 deletions aws_lambda_powertools/utilities/parameters/ssm.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,9 @@ class SSMProvider(BaseProvider):
config: botocore.config.Config, optional
Botocore configuration to pass during client initialization
boto3_session : boto3.session.Session, optional
Boto3 session to use for AWS API communication
Boto3 session to use for AWS API communication, will not be used if boto3_client is not None
boto3_client: boto3.client, optional
Boto3 Client to use for AWS API communication, will be used instead of boto3_session if both provided

Example
-------
Expand Down Expand Up @@ -76,14 +78,22 @@ class SSMProvider(BaseProvider):

client: Any = None

def __init__(self, config: Optional[Config] = None, boto3_session: Optional[boto3.session.Session] = None):
def __init__(
self,
config: Optional[Config] = None,
boto3_session: Optional[boto3.session.Session] = None,
boto3_client: Optional[boto3.client] = None,
):
"""
Initialize the SSM Parameter Store client
"""

config = config or Config()
session = boto3_session or boto3.session.Session()
self.client = session.client("ssm", config=config)
if boto3_client is not None:
self.client = boto3_client
else:
session = boto3_session or boto3.session.Session()
self.client = session.client("ssm", config=config)

super().__init__()

Expand Down
88 changes: 52 additions & 36 deletions docs/utilities/parameters.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,14 +24,14 @@ This utility requires additional permissions to work as expected.
???+ note
Different parameter providers require different permissions.

Provider | Function/Method | IAM Permission
------------------------------------------------- | ------------------------------------------------- | ---------------------------------------------------------------------------------
SSM Parameter Store | `get_parameter`, `SSMProvider.get` | `ssm:GetParameter`
SSM Parameter Store | `get_parameters`, `SSMProvider.get_multiple` | `ssm:GetParametersByPath`
Secrets Manager | `get_secret`, `SecretsManager.get` | `secretsmanager:GetSecretValue`
DynamoDB | `DynamoDBProvider.get` | `dynamodb:GetItem`
DynamoDB | `DynamoDBProvider.get_multiple` | `dynamodb:Query`
App Config | `AppConfigProvider.get_app_config`, `get_app_config` | `appconfig:GetConfiguration`
| Provider | Function/Method | IAM Permission |
| ------------------- | ---------------------------------------------------- | ------------------------------- |
| SSM Parameter Store | `get_parameter`, `SSMProvider.get` | `ssm:GetParameter` |
| SSM Parameter Store | `get_parameters`, `SSMProvider.get_multiple` | `ssm:GetParametersByPath` |
| Secrets Manager | `get_secret`, `SecretsManager.get` | `secretsmanager:GetSecretValue` |
| DynamoDB | `DynamoDBProvider.get` | `dynamodb:GetItem` |
| DynamoDB | `DynamoDBProvider.get_multiple` | `dynamodb:Query` |
| App Config | `AppConfigProvider.get_app_config`, `get_app_config` | `appconfig:GetConfiguration` |

### Fetching parameters

Expand Down Expand Up @@ -147,10 +147,10 @@ def handler(event, context):

The AWS Systems Manager Parameter Store provider supports two additional arguments for the `get()` and `get_multiple()` methods:

| Parameter | Default | Description |
|---------------|---------|-------------|
| **decrypt** | `False` | Will automatically decrypt the parameter.
| **recursive** | `True` | For `get_multiple()` only, will fetch all parameter values recursively based on a path prefix.
| Parameter | Default | Description |
| ------------- | ------- | ---------------------------------------------------------------------------------------------- |
| **decrypt** | `False` | Will automatically decrypt the parameter. |
| **recursive** | `True` | For `get_multiple()` only, will fetch all parameter values recursively based on a path prefix. |

```python hl_lines="6 8" title="Example with get() and get_multiple()"
from aws_lambda_powertools.utilities import parameters
Expand Down Expand Up @@ -189,9 +189,9 @@ For single parameters, you must use `id` as the [partition key](https://docs.aws

DynamoDB table with `id` partition key and `value` as attribute

| id | value |
|--------------|----------|
| my-parameter | my-value |
| id | value |
| ------------ | -------- |
| my-parameter | my-value |

With this table, `dynamodb_provider.get("my-param")` will return `my-value`.

Expand Down Expand Up @@ -223,11 +223,11 @@ You can retrieve multiple parameters sharing the same `id` by having a sort key

DynamoDB table with `id` primary key, `sk` as sort key` and `value` as attribute

| id | sk | value |
|-------------|---------|------------|
| my-hash-key | param-a | my-value-a |
| my-hash-key | param-b | my-value-b |
| my-hash-key | param-c | my-value-c |
| id | sk | value |
| ----------- | ------- | ---------- |
| my-hash-key | param-a | my-value-a |
| my-hash-key | param-b | my-value-b |
| my-hash-key | param-c | my-value-c |

With this table, `dynamodb_provider.get_multiple("my-hash-key")` will return a dictionary response in the shape of `sk:value`.

Expand Down Expand Up @@ -261,12 +261,12 @@ With this table, `dynamodb_provider.get_multiple("my-hash-key")` will return a d

DynamoDB provider can be customized at initialization to match your table structure:

| Parameter | Mandatory | Default | Description |
|----------------|-----------|---------|-------------|
| **table_name** | **Yes** | *(N/A)* | Name of the DynamoDB table containing the parameter values.
| **key_attr** | No | `id` | Hash key for the DynamoDB table.
| **sort_attr** | No | `sk` | Range key for the DynamoDB table. You don't need to set this if you don't use the `get_multiple()` method.
| **value_attr** | No | `value` | Name of the attribute containing the parameter value.
| Parameter | Mandatory | Default | Description |
| -------------- | --------- | ------- | ---------------------------------------------------------------------------------------------------------- |
| **table_name** | **Yes** | *(N/A)* | Name of the DynamoDB table containing the parameter values. |
| **key_attr** | No | `id` | Hash key for the DynamoDB table. |
| **sort_attr** | No | `sk` | Range key for the DynamoDB table. You don't need to set this if you don't use the `get_multiple()` method. |
| **value_attr** | No | `value` | Name of the attribute containing the parameter value. |

```python hl_lines="3-8" title="Customizing DynamoDBProvider to suit your table design"
from aws_lambda_powertools.utilities import parameters
Expand Down Expand Up @@ -467,22 +467,22 @@ def handler(event, context):

Here is the mapping between this utility's functions and methods and the underlying SDK:

| Provider | Function/Method | Client name | Function name |
|---------------------|---------------------------------|------------------|----------------|
| SSM Parameter Store | `get_parameter` | `ssm` | [get_parameter](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ssm.html#SSM.Client.get_parameter) |
| SSM Parameter Store | `get_parameters` | `ssm` | [get_parameters_by_path](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ssm.html#SSM.Client.get_parameters_by_path) |
| SSM Parameter Store | `SSMProvider.get` | `ssm` | [get_parameter](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ssm.html#SSM.Client.get_parameter) |
| SSM Parameter Store | `SSMProvider.get_multiple` | `ssm` | [get_parameters_by_path](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ssm.html#SSM.Client.get_parameters_by_path) |
| Provider | Function/Method | Client name | Function name |
| ------------------- | ------------------------------- | ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- |
| SSM Parameter Store | `get_parameter` | `ssm` | [get_parameter](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ssm.html#SSM.Client.get_parameter) |
| SSM Parameter Store | `get_parameters` | `ssm` | [get_parameters_by_path](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ssm.html#SSM.Client.get_parameters_by_path) |
| SSM Parameter Store | `SSMProvider.get` | `ssm` | [get_parameter](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ssm.html#SSM.Client.get_parameter) |
| SSM Parameter Store | `SSMProvider.get_multiple` | `ssm` | [get_parameters_by_path](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ssm.html#SSM.Client.get_parameters_by_path) |
| Secrets Manager | `get_secret` | `secretsmanager` | [get_secret_value](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/secretsmanager.html#SecretsManager.Client.get_secret_value) |
| Secrets Manager | `SecretsManager.get` | `secretsmanager` | [get_secret_value](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/secretsmanager.html#SecretsManager.Client.get_secret_value) |
| DynamoDB | `DynamoDBProvider.get` | `dynamodb` | ([Table resource](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/dynamodb.html#table)) | [get_item](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/dynamodb.html#DynamoDB.Table.get_item)
| DynamoDB | `DynamoDBProvider.get_multiple` | `dynamodb` | ([Table resource](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/dynamodb.html#table)) | [query](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/dynamodb.html#DynamoDB.Table.query)
| App Config | `get_app_config` | `appconfig` | [get_configuration](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/appconfig.html#AppConfig.Client.get_configuration) |
| DynamoDB | `DynamoDBProvider.get` | `dynamodb` | ([Table resource](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/dynamodb.html#table)) | [get_item](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/dynamodb.html#DynamoDB.Table.get_item) |
| DynamoDB | `DynamoDBProvider.get_multiple` | `dynamodb` | ([Table resource](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/dynamodb.html#table)) | [query](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/dynamodb.html#DynamoDB.Table.query) |
| App Config | `get_app_config` | `appconfig` | [get_configuration](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/appconfig.html#AppConfig.Client.get_configuration) |


### Customizing boto configuration

The **`config`** and **`boto3_session`** parameters enable you to pass in a custom [botocore config object](https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html) or a custom [boto3 session](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html) when constructing any of the built-in provider classes.
The **`config`** and **`boto3_session`** and **`boto3_client`** parameters enable you to pass in a custom [botocore config object](https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html) or a custom [boto3 session](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html) or a custom [boto3 client](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/boto3.html) when constructing any of the built-in provider classes.

???+ tip
You can use a custom session for retrieving parameters cross-account/region and for snapshot testing.
Expand Down Expand Up @@ -516,6 +516,22 @@ The **`config`** and **`boto3_session`** parameters enable you to pass in a cust
...
```

=== "Custom client"

```python hl_lines="2 4 5"
from aws_lambda_powertools.utilities import parameters
import boto3

boto3_client= boto3.client("ssm")
ssm_provider = parameters.SSMProvider(boto3_client=boto3_client)

def handler(event, context):
# Retrieve a single parameter
value = ssm_provider.get("/my/parameter")
...
Comment on lines +562 to +571
Copy link
Contributor

@michaelbrewer michaelbrewer Apr 19, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would update the example to still use boto session to create the client (see boto3 Sessions, and Why You Should Use Them ). And showcase what you can do when you create your own client, like customerize the endpoint url as per the original issue.

Suggested change
from aws_lambda_powertools.utilities import parameters
import boto3
boto3_client= boto3.client("ssm")
ssm_provider = parameters.SSMProvider(boto3_client=boto3_client)
def handler(event, context):
# Retrieve a single parameter
value = ssm_provider.get("/my/parameter")
...
from aws_lambda_powertools.utilities import parameters
import boto3
session = boto3.session.Session()
boto3_client= session.client(service_name="ssm", endpoint_url='custom_endpoint')
ssm_provider = parameters.SSMProvider(boto3_client=boto3_client)
def handler(event, context):
# Retrieve a single parameter
value = ssm_provider.get("/my/parameter")
...

```


## Testing your code

For unit testing your applications, you can mock the calls to the parameters utility to avoid calling AWS APIs. This
Expand Down
Loading