feature: Idempotency module #717

jeromevdl · 2022-01-24T20:59:50Z

Issue #, if available: #515

Description of changes:

Creation of a new module to handle idempotency of a Lambda function.
Creation of a new module to centralize json configuration, to be reused by idempotency and validation module, potentially others?

Checklist

Meet tenets criteria
[x ] Update tests
Update docs ==> will create documentation once the code itself is validated
PR title follows conventional commit semantics

Breaking change checklist

RFC issue #:

Migration process documented
Implement warnings (if it can live side by side)

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

...src/main/java/software/amazon/lambda/powertools/idempotency/internal/IdempotencyHandler.java

...y/src/main/java/software/amazon/lambda/powertools/idempotency/internal/IdempotentAspect.java

...src/main/java/software/amazon/lambda/powertools/idempotency/internal/IdempotencyHandler.java

example/HelloWorldFunction/src/main/java/helloworld/AppIdempotency.java

pankajagrawal16 · 2022-01-26T14:18:38Z

Project seems to have failing build when build on M1 Mac. Looks like the native lib does not support it ? Can be a problem

 (fat file, but missing compatible architecture (have 'i386,x86_64', need 'arm64e')), '/usr/local/lib/libsqlite4java-osx-1.0.392.dylib' (no such file), '/usr/lib/libsqlite4java-osx-1.0.392.dylib' (no such file)
        at java.base/jdk.internal.loader.NativeLibraries.load(Native Method)

pankajagrawal16 · 2022-01-26T14:21:41Z

General question. Should we use the CRT Http client by default? It's by far the fastest. It is the standard http client in the Kotlin SDK but is developer preview in the Java SDK.

Sorry I was still on UrlConnectionHttpClient (https://aws.amazon.com/blogs/developer/tuning-the-aws-java-sdk-2-x-to-reduce-startup-time/). If CRT is faster, we can use it. Any idea of the ETA of this "preview"?

I would recommend not to use the library which is in preview. It might be a an issue in adoption as well. We can definitely create a issue to upgrade to CRT once its out as GA

jeromevdl · 2022-01-26T20:50:36Z

Project seems to have failing build when build on M1 Mac. Looks like the native lib does not support it ? Can be a problem

(fat file, but missing compatible architecture (have 'i386,x86_64', need 'arm64e')), '/usr/local/lib/libsqlite4java-osx-1.0.392.dylib' (no such file), '/usr/lib/libsqlite4java-osx-1.0.392.dylib' (no such file)
at java.base/jdk.internal.loader.NativeLibraries.load(Native Method)

Yes I had the issue, I'm on M1. It works using another JDK (x86 64-bit):

java version "1.8.0_202"
Java(TM) SE Runtime Environment (build 1.8.0_202-b08)
Java HotSpot(TM) 64-Bit Server VM (build 25.202-b08, mixed mode)

.../src/main/java/software/amazon/lambda/powertools/idempotency/persistence/cache/LRUCache.java

...empotency/src/main/java/software/amazon/lambda/powertools/idempotency/IdempotencyConfig.java

example/template.yaml

pankajagrawal16 · 2022-01-28T07:59:14Z

Yes I had the issue, I'm on M1. It works using another JDK (x86 64-bit):

java version "1.8.0_202"
Java(TM) SE Runtime Environment (build 1.8.0_202-b08)
Java HotSpot(TM) 64-Bit Server VM (build 25.202-b08, mixed mode)

Hmm I didnt try this specific version, But any other way not to force developing module on a specific java version for M1s ? I dont think its ideal to force a specific version. I am using latest corretto java 8. If there is no other way of configuring things differently, then may be we should re consider how the tests are written ?

jeromevdl · 2022-01-28T10:28:52Z

Yes I had the issue, I'm on M1. It works using another JDK (x86 64-bit):

java version "1.8.0_202"
Java(TM) SE Runtime Environment (build 1.8.0_202-b08)
Java HotSpot(TM) 64-Bit Server VM (build 25.202-b08, mixed mode)

Hmm I didn't try this specific version, But any other way not to force developing module on a specific java version for M1s ? I dont think its ideal to force a specific version. I am using latest corretto java 8. If there is no other way of configuring things differently, then may be we should re consider how the tests are written ?

I will have a look at the comments on the issue, see if there are alternative solutions.

Happy to hear if you have other ideas on how to test DynamoDB locally without Dynamodb Local. Docker? I thought about that initially, but I remember having issues on Github as build are already executed in docker (docker in docker is often an issue).

pankajagrawal16

Overall UX looks solid IMO.

I have few suggestions and nit picks, but it looks nice. Have a look and see if you agree.

Also we need to bring in module under CI/CD via git actions and include new module in all the workflow files where relevant.

...ain/java/software/amazon/lambda/powertools/idempotency/persistence/BasePersistenceStore.java

...ilities/src/main/java/software/amazon/lambda/powertools/utilities/jmespath/JsonFunction.java

...tware/amazon/lambda/powertools/idempotency/exceptions/IdempotencyConfigurationException.java

...src/main/java/software/amazon/lambda/powertools/idempotency/internal/IdempotencyHandler.java

...java/software/amazon/lambda/powertools/idempotency/persistence/DynamoDBPersistenceStore.java

pankajagrawal16 · 2022-01-28T10:19:00Z

example/HelloWorldFunction/src/main/java/helloworld/AppIdempotency.java

+import java.util.Map;
+import java.util.stream.Collectors;
+
+public class AppIdempotency implements RequestHandler<APIGatewayProxyRequestEvent, APIGatewayProxyResponseEvent> {


I would avoid anything new to the examples project. see #726.

it will be great to have a working app with demo showcased here instead https://github.com/aws-samples/aws-lambda-powertools-examples

Can i just let it here and also add it to the other repo, now that it's done.

I mean if you want to leave it here for now thats fine, as long as its added in the example repo too. Eventually examples folder will be gone from the project. We are tracking this against this issue #732

Sample is created, see PR aws-samples/aws-lambda-powertools-examples#14

...src/main/java/software/amazon/lambda/powertools/idempotency/internal/IdempotencyHandler.java

example/HelloWorldFunction/src/main/java/helloworld/AppIdempotency.java

.../test/java/software/amazon/lambda/powertools/idempotency/persistence/cache/LRUCacheTest.java

.../test/java/software/amazon/lambda/powertools/idempotency/internal/IdempotencyAspectTest.java

Move ObjectMapper and JMESPath config to a new module, to be reused by other powertools modules

….10.3

jeromevdl · 2022-02-03T08:38:15Z

@msailes @pankajagrawal16 If you are ok with the code, I will create the doc, so we have everything in the same PR. I will also create (copy) the example in the other repo.

pankajagrawal16 · 2022-02-03T09:01:26Z

@msailes @pankajagrawal16 If you are ok with the code, I will create the doc, so we have everything in the same PR. I will also create (copy) the example in the other repo.

It looks good to me that we can start writing docs for sure.

Issue related to building module on M1 mac with a specific java version still remains rite ? If we dont have an alternate solution for it now, I would say document rationale, related issues and workaround in the readme for powertools-idempotency module. With step by step details for getting it work.

I can use the same steps to validate if it works on my system. Sounds good ?

example/HelloWorldFunction/build.gradle

powertools-idempotency/pom.xml

pankajagrawal16

Docs looks solid too now. Some minor suggestion

docs/utilities/idempotency.md

...empotency/src/main/java/software/amazon/lambda/powertools/idempotency/IdempotencyConfig.java

docs/utilities/utilities.md

pankajagrawal16

Looks great to me. @msailes Would you like to give a pair of 👀 to the docs as well ?

powertools-utilities/pom.xml

...empotency/src/main/java/software/amazon/lambda/powertools/idempotency/IdempotencyConfig.java

docs/utilities/idempotency.md

- add installation guide - sam template indentation - local cache disabled by default - + minors changes

msailes

This looks fantastic!

pankajagrawal16

🚀

pull-request-size bot added the size/XXL label Jan 24, 2022

pankajagrawal16 reviewed Jan 24, 2022

View reviewed changes

msailes reviewed Jan 25, 2022

View reviewed changes

example/HelloWorldFunction/src/main/java/helloworld/AppIdempotency.java Show resolved Hide resolved

msailes reviewed Jan 25, 2022

View reviewed changes

example/HelloWorldFunction/src/main/java/helloworld/AppIdempotency.java Show resolved Hide resolved

This comment has been minimized.

Sign in to view

michaelbrewer reviewed Jan 27, 2022

View reviewed changes

.../src/main/java/software/amazon/lambda/powertools/idempotency/persistence/cache/LRUCache.java Outdated Show resolved Hide resolved

msailes reviewed Jan 27, 2022

View reviewed changes

...empotency/src/main/java/software/amazon/lambda/powertools/idempotency/IdempotencyConfig.java Show resolved Hide resolved

msailes reviewed Jan 27, 2022

View reviewed changes

example/template.yaml Show resolved Hide resolved

msailes reviewed Jan 27, 2022

View reviewed changes

example/template.yaml Show resolved Hide resolved

pankajagrawal16 reviewed Jan 28, 2022

View reviewed changes

jeromevdl added a commit to jeromevdl/aws-lambda-powertools-java that referenced this pull request Feb 2, 2022

PR aws-powertools#717: document exceptions

ee5e4cb

jeromevdl added a commit to jeromevdl/aws-lambda-powertools-java that referenced this pull request Feb 2, 2022

PR aws-powertools#717: move cache to utilities

244cf1d

jeromevdl added a commit to jeromevdl/aws-lambda-powertools-java that referenced this pull request Feb 2, 2022

PR aws-powertools#717: native libs for dynamo & sqlite

265d282

jeromevdl added a commit to jeromevdl/aws-lambda-powertools-java that referenced this pull request Feb 2, 2022

PR aws-powertools#717: changes post review

b1cb8c6

jeromevdl added a commit to jeromevdl/aws-lambda-powertools-java that referenced this pull request Feb 2, 2022

PR aws-powertools#717: add modules to GitHub workflows

f85551f

jeromevdl added a commit to jeromevdl/aws-lambda-powertools-java that referenced this pull request Feb 2, 2022

PR aws-powertools#717: fix utilities dependency

cc1b628

jeromevdl added a commit to jeromevdl/aws-lambda-powertools-java that referenced this pull request Feb 2, 2022

PR aws-powertools#717: fix utilities dependency

6a6c3b3

jeromevdl added a commit to jeromevdl/aws-lambda-powertools-java that referenced this pull request Feb 2, 2022

PR aws-powertools#717: fix utilities dependency

14c6371

jeromevdl added 6 commits February 2, 2022 21:32

Move JSON config to a new utility module

4440dce

Move ObjectMapper and JMESPath config to a new module, to be reused by other powertools modules

Use new utility module for JSON config

08c575b

Idempotency module

03999db

Idempotency module example

b1d0262

Idempotency module added to lambda-powertools

2ac9f65

solve JDK 16 build

4ed575d

PR aws-powertools#717: upgrade utilities and idempotency version to 1…

1803bb9

….10.3

jeromevdl force-pushed the idempotency branch from 14c6371 to 1803bb9 Compare February 2, 2022 20:41

PR aws-powertools#717: end to end test with a handler

cd71b5f

pankajagrawal16 reviewed Feb 3, 2022

View reviewed changes

example/HelloWorldFunction/build.gradle Outdated Show resolved Hide resolved

jeromevdl added 5 commits February 3, 2022 10:22

PR aws-powertools#717: upgrade version in example

3245ba6

PR aws-powertools#717: expiration in seconds instead of ms

69521ec

PR aws-powertools#717: new test

96eadc0

adding documentation

d302f7a

removing unused dependencies

f713c3c

jeromevdl mentioned this pull request Feb 8, 2022

sample for idempotency aws-samples/aws-lambda-powertools-examples#14

Merged

pankajagrawal16 reviewed Feb 14, 2022

View reviewed changes

powertools-idempotency/pom.xml Show resolved Hide resolved

pankajagrawal16 reviewed Feb 14, 2022

View reviewed changes

docs/utilities/idempotency.md Outdated Show resolved Hide resolved

...empotency/src/main/java/software/amazon/lambda/powertools/idempotency/IdempotencyConfig.java Outdated Show resolved Hide resolved

pankajagrawal16 reviewed Feb 14, 2022

View reviewed changes

docs/utilities/utilities.md Outdated Show resolved Hide resolved

jeromevdl added 4 commits February 14, 2022 16:25

sqlite dependency for Mac M1

38522bb

update doc for sqlite dependency on M1

7401f22

change expiration to duration

c39d3a9

update doc for utilities

4855a20

pankajagrawal16 previously approved these changes Feb 14, 2022

View reviewed changes

msailes reviewed Feb 16, 2022

View reviewed changes

jeromevdl added 4 commits February 16, 2022 13:28

PR aws-powertools#717: rename utilities into serialization

81a7052

PR aws-powertools#717: move back LRU into idempotency module

cefd1f7

PR aws-powertools#717: local cache disabled by default

1550cc4

PR aws-powertools#717: update documentation

221855e

- add installation guide - sam template indentation - local cache disabled by default - + minors changes

jeromevdl dismissed pankajagrawal16’s stale review via 221855e February 16, 2022 12:34

msailes approved these changes Feb 16, 2022

View reviewed changes

pankajagrawal16 approved these changes Feb 16, 2022

View reviewed changes

pankajagrawal16 merged commit fc95eed into aws-powertools:master Feb 16, 2022

jeromevdl mentioned this pull request Feb 16, 2022

chore:Prep release 1.11.0 #753

Merged

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feature: Idempotency module #717

feature: Idempotency module #717

jeromevdl commented Jan 24, 2022

This comment has been minimized.

This comment has been minimized.

pankajagrawal16 commented Jan 26, 2022 •

edited

Loading

pankajagrawal16 commented Jan 26, 2022

jeromevdl commented Jan 26, 2022

pankajagrawal16 commented Jan 28, 2022 •

edited

Loading

jeromevdl commented Jan 28, 2022

pankajagrawal16 left a comment

pankajagrawal16 Jan 28, 2022 •

edited

Loading

jeromevdl Feb 2, 2022

pankajagrawal16 Feb 2, 2022

jeromevdl Feb 8, 2022

jeromevdl commented Feb 3, 2022

pankajagrawal16 commented Feb 3, 2022 •

edited

Loading

pankajagrawal16 left a comment

pankajagrawal16 left a comment

msailes left a comment

pankajagrawal16 left a comment

feature: Idempotency module #717

feature: Idempotency module #717

Conversation

jeromevdl commented Jan 24, 2022

Description of changes:

Breaking change checklist

This comment has been minimized.

This comment has been minimized.

pankajagrawal16 commented Jan 26, 2022 • edited Loading

pankajagrawal16 commented Jan 26, 2022

jeromevdl commented Jan 26, 2022

pankajagrawal16 commented Jan 28, 2022 • edited Loading

jeromevdl commented Jan 28, 2022

pankajagrawal16 left a comment

Choose a reason for hiding this comment

pankajagrawal16 Jan 28, 2022 • edited Loading

Choose a reason for hiding this comment

jeromevdl Feb 2, 2022

Choose a reason for hiding this comment

pankajagrawal16 Feb 2, 2022

Choose a reason for hiding this comment

jeromevdl Feb 8, 2022

Choose a reason for hiding this comment

jeromevdl commented Feb 3, 2022

pankajagrawal16 commented Feb 3, 2022 • edited Loading

pankajagrawal16 left a comment

Choose a reason for hiding this comment

pankajagrawal16 left a comment

Choose a reason for hiding this comment

msailes left a comment

Choose a reason for hiding this comment

pankajagrawal16 left a comment

Choose a reason for hiding this comment

pankajagrawal16 commented Jan 26, 2022 •

edited

Loading

pankajagrawal16 commented Jan 28, 2022 •

edited

Loading

pankajagrawal16 Jan 28, 2022 •

edited

Loading

pankajagrawal16 commented Feb 3, 2022 •

edited

Loading