Feature request: utility function to get multiple parameters by name instead of path

[hooverdc]

Is your feature request related to a problem? Please describe.

I have several functions in a severless application with several SSM parameters under the same path. Each function only needs access to some parameters and most functions will share some parameters but not all. I don't want functions to have access to parameters they don't require. I also don't want to call get_parameter for each parameter in order to keep cold start times down. I would like to be able to specify the parameter names and return them in one batch.

Describe the solution you'd like

A get_parameters_by_name function that accepts a list of strings of parameter names and returns a dict similar to the get_parameters function, but with the full parameter path as the key (presumably you could call this with parameters on different paths).

https://docs.aws.amazon.com/cli/latest/reference/ssm/get-parameters.html#get-parameters

Describe alternatives you've considered

Writing my own wrapper function that calls get_parameters from boto3.

Additional context
N/A

[boring-cyborg]

Thanks for opening your first issue here! We'll come back to you as soon as we can.

[heitorlessa]

Hey @hooverdc, thanks for creating this feature request! I believe this is best handled by your own custom wrapper, or perhaps approaching this differently.

I don't want functions to have access to parameters they don't require. I also don't want to call get_parameter for each parameter in order to keep cold start times down.

Without knowing the number of parameters you have, their size, and how you create them, have you considered creating an aggregate parameter with multiple values?

Example. This allows e2e/integ scenarios to make a single call and get everything they need, while still having separate parameters that can be useful for other purposes like deployment time.

Resources:
    ResourcesMapParameter:
        Type: AWS::SSM::Parameter
        Properties:
            Name: !Sub "/${Stage}/service/trip_ingestion/test/config"
            Type: String
            Description: Service Map with common resources used for integ/e2e testing
            Value: !Sub |
                {
                    "MobileTripsBucket": "${MobileTripsBucket}",
                    "MobileTripsQueue": "${MobileTripsQueue}",
                    "TripIngestionFunction": "${TripIngestionFunction}",
                    "MobileTripsApiEndpoint": "https://${MobileTripsApi}.execute-api.${AWS::Region}.amazonaws.com/${MobileTripsApi.Stage}/trip",
                    "Stage": "${Stage}"
                }

Hope that helps!

Thanks

[hooverdc]

Hey @heitorlessa. I usually take that approach and create an aggregate SSM parameter. You can shove a lot into a single SSM parameter with base64 encoded gzip! Ultimately this isn't an issue for me. I was just hoping that Powertools supported GetParameters alongside GetParameter and GetParametersByPath out of the box.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

This issue is now closed. Please be mindful that future comments are hard for our team to see.

If you need more assistance, please either tag a team member or open a new issue that references this one.

If you wish to keep having a conversation with other community members under this issue feel free to do so.

[heitorlessa]

Hey @heitorlessa. I usually take that approach and create an aggregate SSM parameter. You can shove a lot into a single SSM parameter with base64 encoded gzip! Ultimately this isn't an issue for me. I was just hoping that Powertools supported GetParameters alongside GetParameter and GetParametersByPath out of the box.

We're in a much better position to reconsider this than we were 7 months ago - I'd love to see a POC if you have the bandwidth.

Back then we had to be more strict due to package size, but next month in V2, we'll reduce ~95-97% by making all dependencies optional and using boto3 available at Lambda runtime.

If you do have bandwidth, please reopen and I'd appreciate a draft PR to get this going

[heitorlessa]

Adding this to our next iteration (Nov 7th-20th)!

[heitorlessa]

Shifting to this current iteration (next release in two weeks), as we've managed to finish our biggest chunk of work earlier ;)

I've got a few upfront questions @hooverdc:

• Fine-grained or shared settings - how much control would you like to have?
  • If you pass parameters as List[str], we'd set the same TTL/settings for all of them.
    • Example: ["/parameter/1", "/parameter/X/Y"]
  • If you pass parameters as List[Dict], you could optionally pass per parameter info - max_age, whether to decrypt, and whether to deserialize (JSON, binary). Empty param config {} would mean using the defaults.
    • Example: [ {"/param/A": {} }, "/param/B": {"max_age": 500, "transform": "json"} ]
• Key or path. At the moment, get_parameters(path: str) expects a path and will fetch recursively. My understanding is that you'd like to pass the absolute path to the parameter, since you might have multiples in different locations. This being the main reason why get_parameters isn't sufficient as-is (and return structure of course)

This would help disambiguate the implementation, tests, and later assess what UX makes the most explicit experience.

Thank you!

[hooverdc]

@heitorlessa

Fine-grained or shared settings - I would prefer the List[Dict] approach. The big win there is being able to do a transform for some but not all parameters.
Key or path - I would use the absolute path. In my particular case I could have used get_parameters and discarded unused parameters, but I didn't want to fetch secrets that I didn't need. If you had a mix of common and app specific parameters being able to use different root absolute paths would be useful.

[heitorlessa]

Perfect, that’s why I asked ;-) I’ll shuffle the board tomorrow and start working on this tomorrow afternoon. We’ll likely use threads (futures) to fetch an arbitrary number quicker. Thanks for the swift response!

…

On Wed, 26 Oct 2022 at 18:06, David Hoover ***@***.***> wrote: @heitorlessa <https://github.com/heitorlessa> Fine-grained or shared settings - I would prefer the List[Dict] approach. The big win there is being able to do a transform for some but not all parameters. Key or path - I would use the absolute path. In my particular case I could have used get_parameters and discarded unused parameters, but I didn't want to fetch secrets that I didn't need. If you had a mix of common and app specific parameters being able to use different root absolute paths would be useful. — Reply to this email directly, view it on GitHub <#1040 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAZPQBCFSVA5CQZNWV3HQ3TWFFJJHANCNFSM5PIHNPOQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[heitorlessa]

Hey @hooverdc draft PR is ready for UX review (used a dict to improve UX): #1678

I have two questions for you before moving forward:

1. Should we keep parallel= parameter when single thread is faster* in this context?
2. What should be the error strategy?
   • (A) Treat as non-atomic by catching the error, and include each failed param in a _errors key along with the reason.
   • (B) Fail-fast by default with the option to override (raise_on_error). When overridden, it falls back to A
   • (C) Default to A by keeping raise_on_error=False. IMHO I think this is the best option.
3. How many parameters do you intend to fetch on average?

---

UX

import json
from typing import Any
from aws_lambda_powertools.utilities.parameters.ssm import get_parameters_by_name

params: dict[str, Any] = {
    "/db/proxy_arn": {"max_age": 300}, # override caching to 5 minutes
    "/cdk-bootstrap/hnb659fds/version": {},
    "/dev/service/trip_ingestion/api/keys": {"max_age": "60"},
    "/develop/service/amplify/auth/userpool/arn": {},
    "/develop/service/amplify/auth/userpool/clientId": {},
    "/develop/service/amplify/deployment/deploymentBucket": {},
    "/develop/service/booking/statemachine/processBooking": {"transform": "json"}, # final value will be dict
    "/develop/service/loadtest/csv/token": {},
    "/develop/service/loadtest/csv/user": {},
    "/develop/service/loyalty/messaging/queue": {},
}


def lambda_handler(event, context):
    ret = get_parameters_by_name(parameters=params, max_age=30) # 30 seconds in-memory caching by default
    return {
        "statusCode": 200,
        "body": json.dumps(ret),
    }

---

*Faster. I did two tests within Lambda runtime:

1. 20 params to fetch leading to a timeout with parallel=True due to SSM throttling leading to exponential backoff even w/ 30s timeout
2. 10 params to fetch

Quick results

• First trace: 20 params in single and multi-threading with 128M memory size. The errors were due to throttling leading to exponential back-off to the point the operation timed out. Successful ones were with 10 params (no throttling).

• Second trace: Same 10 params but using with 1024M memory size

Without much further digging, this is largely due to CPU slicing in Lambda (one real core only at ~1.7G), combined with LWP cost and SSM throttling.

[heitorlessa]

Pasting here from another customer on the error handling question.

I like option B I think. Fail fast should be the default assuming most of the time the customer has it correct and working, and when there is an issue with one parameter there is no point continuing with the rest. Especially as you will consume SSM API calls that are unnecessary and other systems may end up with throttle exceptions.

I do like the option of overriding it to A as some may prefer this, especially if they are in development and they have multiple access denied issues - you don’t want to deal with those one by one.

[hooverdc]

2. I like option B. Usually if one parameter fails my application isn't going to work anyways.
3. From 7 to 10.

Is there a reason for using the GetParameter operation in a loop vs the GetParameters operation? I assumed it would be the latter, but I see how that does limit the granularity of the decrypt option to per API call instead of per parameter.

[heitorlessa]

Great, this unblocks me, thank you!!

Is there a reason for using the GetParameter operation in a loop vs the GetParameters operation?

Primarily because of decrypt is either all or none, like you pointed out. Happy to test whether the latency is significant with GetParameters -- do you happen to have any SecureString?

[heitorlessa]

Quick list of Pros and Cons for using GetParameters for this feature:

• Pros: It uses a single API call decreasing the risk of hitting throttling. Slightly faster response time (0.15s vs 0.17s with 10 params)
• Cons: decrypt is either all or nothing. Max of 10 parameters (can easily slice it)

I'll give it a try on using both: GetParameters when decrypt option isn't set, and GetParameter when is set.

This would require customers to have both GetParameter and GetParameters IAM permission but doesn't seem like a big deal.

Note to self: Need to handle when customers set decrypt by default but override for certain parameters.

[heitorlessa]

Phew, this was challenging due to the flexiblity. Moved to use both GetParameters and GetParameter transparently (when you use decrypt). UX is kept.

Tested with 10 params as before, and tried with 20 to confirm slicing too - all with 128M.

Depending how much time I'll have tomorrow (admin day), I'll decide on whether to postpone for another release, or include graceful error handling in a subsequent release. Big progress nonetheless, as I had to address some tech debt and refactored a great chunk of this utility + strict typing in some areas.

[heitorlessa]

This is finally done - discovered more bugs as I had to refactor other parts to play nice.

This should be released by EOD and will auto-close when available in both PyPi and Lambda Layer/SAR.

#1678

[github-actions]

This is now released under 2.2.0 version!

[heitorlessa]

hey @hooverdc this is now available as part of v2.2.0 release (Lambda Layer v13): https://github.com/awslabs/aws-lambda-powertools-python/releases/tag/v2.2.0

Docs have been updated to show case both get_parameters and get_parameters_by_name, including the opt-in graceful error handling: https://awslabs.github.io/aws-lambda-powertools-python/2.2.0/utilities/parameters/#fetching-parameters

Let me know if that doesn't address your original ask.

[hooverdc]

Thank you @heitorlessa!