StsWebIdentityCredentialsProvider memory leak

[BartXZX]

StsWebIdentityCredentialsProvider in the default credential provider chain never gets closed. This causes a created STS client to never get closed, which in turn causes the IdleConnectionReaper to keep on to connection managers.

Expected Behavior

Default clients and credential provider chains should clean up any resources it created after being closed.

Current Behavior

DefaultCredentialsProvider#close calls LazyAwsCredentialsProvider#close, which calls IoUtils.closeIfCloseable, but the provided delegate is not an AutoClosable type. It is actually a Lazy<AwsCredentialsProvider>, and Lazy<T> does not implement AutoClosable. So the chain (and consequently the STS client) in the delegate field never receives a close() call.

Screenshot of the debugger

Stack dump from IntelliJ

closeIfCloseable:89, IoUtils (software.amazon.awssdk.utils) [2]
close:50, LazyAwsCredentialsProvider (software.amazon.awssdk.auth.credentials.internal)
close:109, DefaultCredentialsProvider (software.amazon.awssdk.auth.credentials)
closeQuietly:70, IoUtils (software.amazon.awssdk.utils)
closeIfCloseable:87, IoUtils (software.amazon.awssdk.utils) [1]
lambda$close$0:86, AttributeMap (software.amazon.awssdk.utils)
accept:-1, 912038923 (software.amazon.awssdk.utils.AttributeMap$$Lambda$1914)
forEach:981, HashMap$Values (java.util)
close:86, AttributeMap (software.amazon.awssdk.utils)
close:79, SdkClientConfiguration (software.amazon.awssdk.core.client.config)
close:80, HttpClientDependencies (software.amazon.awssdk.core.internal.http)
close:73, AmazonSyncHttpClient (software.amazon.awssdk.core.internal.http)
close:118, BaseSyncClientHandler (software.amazon.awssdk.core.internal.handler)
close:12198, DefaultIamClient (software.amazon.awssdk.services.iam)
discover:37, IamServiceRoleAwsDiscovery (com.planonsoftware.cloud.library.discovery.iamrole)
provideParameters:63, PlanonEnvironmentActivitiesImpl (com.planonsoftware.cloud.pco.orchestrator.controllers.planonenvironment.activities)
invoke0:-1, NativeMethodAccessorImpl (jdk.internal.reflect)
invoke:62, NativeMethodAccessorImpl (jdk.internal.reflect)
invoke:43, DelegatingMethodAccessorImpl (jdk.internal.reflect)
invoke:567, Method (java.lang.reflect)
execute:214, POJOActivityTaskHandler$POJOActivityImplementation (com.uber.cadence.internal.sync)
handle:190, POJOActivityTaskHandler (com.uber.cadence.internal.sync)
handle:175, ActivityWorker$TaskHandlerImpl (com.uber.cadence.internal.worker)
handle:146, ActivityWorker$TaskHandlerImpl (com.uber.cadence.internal.worker)
lambda$process$0:71, PollTaskExecutor (com.uber.cadence.internal.worker)
run:-1, 640627828 (com.uber.cadence.internal.worker.PollTaskExecutor$$Lambda$1491)
runWorker:1128, ThreadPoolExecutor (java.util.concurrent)
run:628, ThreadPoolExecutor$Worker (java.util.concurrent)
run:830, Thread (java.lang)

Steps to Reproduce

I am running the following on a Kubernetes cluster with the AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE environment variables set. The memory just explodes, until eventually it goes out of memory.

while (true) {
  try (S3Client s3 = S3Client.builder().region(Region.EU_WEST_1).build()) {
    System.out.println(s3.headBucket(builder -> builder.bucket("some bucket").build()).toString());
  }
}

Possible Solution

For example, let Lazy implement AutoClosable, and propage close(), like in IoUtils.

Related Issues

I commented here, but its a closed issue and didn't get a response, so I'm opening this issue. #1679

Your Environment

• AWS Java SDK version used: 2.15.25
• JDK version used: 13
• Operating System and version:

[zoewangg]

Thank you for the detailed analysis! Sorry I didn't notice your comment in #1679. We will work on the fix

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.