One of our service is failing due to out of memory

[sigs3gv]

Bug Description

We are encountering a service failure due to a memory leak.

This graph shows the gradual increase in heap usage for our service. The service eventually OOMs and then k8s restarts, which is why you see a new line coming up after one ends.

Upon investigation, we found that AWS SDK v2 is trying to register connections in a HashMap within IdleConnectionReaper and deregister it later, but heap dump shows a huge retained space not getting picked by the GC.

aws-sdk-java-v2/http-clients/apache-client/src/main/java/software/amazon/awssdk/http/apache/internal/conn/IdleConnectionReaper.java

Line 36 in b2fe89e

public final class IdleConnectionReaper {

Above is the screenshot of heapdump visualised in VisualVM. You can see 15k+ objects of PoolingHttpClientConnectionManager, causing 512MB+ of retained space, which could have been garbage collected.

My suspicion is that the deregister method is not getting called once the API call to AWS is complete.

We are using AWS SDK v2 to talk to S3 and EMR Serverless API.

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

The deregisterConnectionManager method should be called every time the connection manager the task is completed. This would ensure proper memory management by allowing garbage collection to free up memory, preventing memory leaks that ultimately lead to service failures.

Current Behavior

Currently, when the connectionManager is registered, the deregisterConnectionManager method is not called, preventing garbage collection from releasing unused memory. This results in a gradual memory buildup, eventually leading to memory failure. (Reference: [IdleConnectionReaper.java - Line 36](

aws-sdk-java-v2/http-clients/apache-client/src/main/java/software/amazon/awssdk/http/apache/internal/conn/IdleConnectionReaper.java

Line 36 in b2fe89e

public final class IdleConnectionReaper {

)).

Reproduction Steps

1. Create a function that calls registerConnectionManager.
2. After the connection manager has completed its tasks, verify if the deregisterConnectionManager method is invoked.

Possible Solution

No response

Additional Information/Context

No response

AWS SDK for Java Version

awsV2SdkVers = '2.20.38'

JDK Version

ENV JAVA_VERSION="21.0.6+7-1~20.04.1"

Operating System and Version

Ubuntu 20.04

[zoewangg]

This is likely fixed in #4087, which was released in 2.20.84. Could you try with the latest version?

[github-actions]

It looks like this issue has not been active for more than five days. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please add a comment to prevent automatic closure, or if the issue is already closed please feel free to reopen it.