Maintenance: avoid fixed-time waiting in our E2E testings

[ijemmy]

Bug description

Currently, our E2E tests are very flaky (fail about 40-50% of the time) due to fixed waiting time. This happens in both metrics and tracers E2E tests.

For example, tracer.test.ts waits for 2 minutes before finishing the beforeAll() function.

Sometimes, the trace come back later than two minutes. The test() blocks will fail.

Expected Behavior

The tests should not be flaky.

Current Behavior

The tests are flaky. If the traces/metrics appear after the fixed wait time.

Possible Solution

1. Implement retry The tests should never wait for fixed amount of time. Instead, it should poll for trace or metrics and retry until it gets expected number of traces. This could be implemented with promise-retry library. See example code in "Reference" section below

2. Introduce uuid However, we cannot simply count the number of traces or metrics because there might be another test running concurrently (e.g. one for Node14, another for Node12) . The traces/metrics found may be from another test running. Thus, we need to introduce a uuid in metadata and filter base on that. You can pass UUID via environment variable like in this e2e test in Logger

If you need refactoring, take note of Logger's e2e tests and try to structure it similar to that.

Steps to Reproduce

Run E2E tests a few times. You can use this script:

repeat 20 {jest --group=e2e/PACKAGE; sleep 0.5}

Environment

• Powertools version used:0.7.0
• Packaging format (Layers, npm): N/A
• AWS Lambda function runtime: 12 and 14
• Debugging logs: N/A

Related issues, RFCs

N/A

Reference

Here's an example implementation on fetching log stream with promise-retry library. It keep fetching until the log group has the atLeast number of log stream. You can adapt this with traces/metrics. (but need to check uuid)

export const fetchStreamsUntil = async (logGroupName: string, atLeast:number): Promise<string[]> => {
  const retryOption = {
    retries: 6,
    factor: 1.5,
    minTimeout: 1000, 
  }
  return promiseRetry(async (retry: any, attemptNumber: number) => {
    const streams = await fetchStreams(logGroupName)
    if(streams.length < atLeast){
      console.debug(`Found ${streams.length} log streams, retry until having at least ${atLeast} stream`);
      retry();
    }
    return streams;
  }, retryOption)
};

[ijemmy]

Discussed with @AWSDB . He'll take this issue.

[dreamorosi]

@AWSDB could you leave a comment so I can assign the issue to you? (apologies for bothering you with this but GitHub doesn't let me add you if you haven't interacted with the issue first).

[ghost]

@dreamorosi That's strange. Anyway, here's my comment - happy to work on this issue!

[github-actions]

⚠️ COMMENT VISIBILITY WARNING ⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.