Tech debt: Idempotency Thread safety E2E test is flaky

[heitorlessa]

Why is this needed?

Ensure E2E pipeline works successfully.

---

FAILED tests/e2e/idempotency/test_idempotency_dynamodb.py::test_idempotent_function_thread_safety@idempotency - assert {'3', 'l', '1', 'x', '_', '-', '8', '0', 't', '}', 'u', 'a', 'H', 'i', 'o', 'h', 'e', 'p', 'E', 'd', 'S', 'P', '6', ',', '[', '{', 'c', 'I', 'n', '7', ' ', 'D', 'N', ']', 'm', 's', 'r', 'T', 'F', '2', '5', ':', '.', '"'} == {'3', 'l', '1', 'x', '_', '-', '8', '0', 't', '}', 'u', 'a', 'H', '4', 'i', 'o', 'h', 'e', 'p', 'E', 'd', 'S', 'P', '6', ',', '[', '9', '{', 'c', 'I', 'n', '7', ' ', 'D', 'N', ']', 'm', 's', 'r', 'T', 'F', '2', '5', ':', '.', '"'}
  Extra items in the right set:
  '9'
  '4'
  Full diff:
    {
     ' ',
     '"',
     ',',
     '-',
     '.',
     '0',
     '1',
     '2',
     '3',
  -  '4',
     '5',
     '6',
     '7',
     '8',
  -  '9',
     ':',
     'D',
     'E',
     'F',
     'H',
     'I',
     'N',
     'P',
     'S',
     'T',
     '[',
     ']',
     '_',
     'a',
     'c',
     'd',
     'e',
     'h',
     'i',
     'l',
     'm',
     'n',
     'o',
     'p',
     'r',
     's',
     't',
     'u',
     'x',
     '{',
     '}',
    }
============ 1 failed, 50 passed, 10 warnings in 233.35s (0:03:53) =============

Which area does this relate to?

Tests

Suggestion

No response

Acknowledgment

• This request meets Powertools for AWS Lambda (Python) Tenets
• Should this be considered in other Powertools for AWS Lambda languages? i.e. Java, TypeScript, and .NET

[heitorlessa]

@rubenfonseca for on-call next week

[rubenfonseca]

Looking at this now

[rubenfonseca]

Note: right now tests are failing because we created too many IAM roles

 ❌  testV37-idempotency-3a6c657f-3575-46e8-848a-0703daabe1b4 failed: Error: The stack named testV37-idempotency-3a6c657f-3575-46e8-848a-0703daabe1b4 failed creation, it may need to be manually deleted from the AWS console: ROLLBACK_COMPLETE: Cannot exceed quota for RolesPerAccount: 1000 (Service: AmazonIdentityManagement; Status Code: 409; Error Code: LimitExceeded; Request ID: d0344a4f-bcb2-46cb-b669-da98fc7fd050; Proxy: null), Cannot exceed quota for RolesPerAccount: 1000 (Service: AmazonIdentityManagement; Status Code: 409; Error Code: LimitExceeded; Request ID: 429d0b45-3a96-470a-8d8c-be363b962ac4; Proxy: null)

Besides deleting the roles, I'll try to figure out why the roles are being removed at the end of each test.

[rubenfonseca]

The role APIGatewayRestCloudWatch seems to never be deleted, even when the tests succeed.

[rubenfonseca]

Found the culprit: CDK introduced a breaking change aws/aws-cdk#22020. Solution is to disable the creation of the CloudWatch role.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

This issue is now closed. Please be mindful that future comments are hard for our team to see.

If you need more assistance, please either tag a team member or open a new issue that references this one.

If you wish to keep having a conversation with other community members under this issue feel free to do so.

[github-actions]

This is now released under 2.19.0 version!